Comparing Four Approaches to Reducing LLM TTFT

The first-token latency (TTFT) of large language model inference is a direct metric affecting user experience and SLA compliance. The paths to reducing TTFT are not singular but span four layers: compute, GPU memory, storage, and architecture. This article compares four mainstream approaches and cites measured data from Mingxin Technology on the AMD MI308X platform to illustrate the practical benefits of storage-side acceleration in long-context scenarios.

Why TTFT Is the Primary Target of Inference Optimization

TTFT refers to the time interval from when a user sends a request to when the first output token is received. In interactive applications, an excessively long TTFT directly degrades the user experience; in batch processing scenarios, it determines whether a request can complete within the SLA deadline. According to FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, the core bottleneck of attention computation lies in HBM bandwidth rather than compute capacity, which provides a foundation for understanding the composition of TTFT: model weight reads, KV Cache reads, and the computation itself are all constrained by data movement speed.

The composition of TTFT can be broken down into: request queuing, weight loading (if swapping is required), prefill computation, and KV Cache writes and reads. Among these, the prefill phase must process all input tokens, requiring substantial computation and memory access. As context length grows, the KV Cache size expands accordingly, and memory access pressure becomes the dominant factor.

Approach 1: Compute and Operator Optimization

FlashAttention reduces HBM read/write operations through I/O-aware attention computation and is currently the most fundamental optimization technique. According to Efficient Memory Management for Large Language Model Serving with PagedAttention, vLLM's PagedAttention manages the KV Cache through paging to reduce GPU memory fragmentation and improve batch throughput. These methods operate at the compute and GPU memory management layers, and their improvement to TTFT is bounded by hardware bandwidth limits.

The advantage of this approach is its broad applicability without requiring hardware changes; the drawback is that when the KV Cache exceeds GPU memory capacity, it must rely on external storage or recomputation, at which point the memory access bottleneck shifts from HBM to PCIe and network links.

Approach 2: GPU Memory Expansion and KV Cache Tiering

When model weights and KV Cache exceed single-GPU memory capacity, a common practice is to offload the KV Cache to CPU memory or remote storage. According to Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving, the KVCache-centric disaggregated architecture reduces redundant prefill overhead through prefix cache reuse and cross-node KV pooling. The benefit of such architectures lies in improved GPU memory utilization, but the cost is added data-path latency.

Mingxin's FX100 KV tiering acceleration solution falls into this category: cold KV Cache is tiered onto NVMe-oF arrays while hot data remains in GPU memory. According to Mingxin's measured report R2 (2026-07-05, 480B·TP8 long-context·production release), with a 480B model and TP8 across three concurrency levels, TTFT p50 dropped from 10.17–35.73s to 7.53–26.35s, a reduction of 26–32%. This data indicates that in long-context scenarios, storage-side latency contributes non-negligibly to TTFT.

Approach 3: Storage Acceleration and Data-Path Optimization

GPU Direct Storage (GDS) technology bypasses the CPU's bounce buffer, allowing GPUs to access storage devices directly and reducing data movement latency. According to NVIDIA GPUDirect Storage Documentation, this technology suits large-block data transfer scenarios but offers limited improvement for small I/O operations. Mingxin's FX100 measured data demonstrates the potential of storage acceleration:

According to Mingxin's measured report R2, the acceleration factor for scenarios without external recompute is 8.6–20×. This implies that in extreme scenarios (KV Cache fully spilled to disk, no recompute), storage bandwidth directly determines the TTFT floor. Measured report R9 (2026-05-30, Huawei Atlas 910B platform) shows that during the model loading phase, FX100 achieves 6.2–9.3× acceleration over the NFS baseline, indicating that storage optimization is equally effective for cold-start scenarios.

Approach 4: Architectural Transformation and Measured Boundaries

The approaches above are not mutually exclusive. In production deployments, compute optimization, GPU memory tiering, and storage acceleration are often combined. Mingxin's measured report R3 (2026-07-06, 480B·TP4×2·full metrics·unified branding) shows that KV tiering acceleration improves throughput by 29% at concurrency 8 (lower bound), by 40% at the optimal operating point of concurrency 16 (upper bound), and by 35–36% across the full machine at TP4×2. This range indicates that acceleration effects are strongly correlated with concurrency patterns, and selection should be evaluated against actual workload characteristics.

Boundaries must be clarified: the figures above come from Mingxin's measurements on the AMD MI308X platform (ROCm 7.2, vLLM 0.20.1+rocm721), using the Qwen3-Coder-480B-FP8 model (MoE, approximately 450GB of weights). Cross-platform and cross-model extrapolation requires caution—according to MLPerf Inference: Datacenter Benchmark Suite Results, public comparisons of inference performance should rely on standardized tests with fixed precision and latency constraints, as differences in memory access architecture across hardware platforms could reverse conclusions.

Conclusion

The choice of path to reduce TTFT depends on bottleneck identification: optimize operators when compute-bound, tier and offload when GPU memory is insufficient, and accelerate the data path when storage latency dominates. Mingxin's FX series (FX100/FX200/FX300/FX400) offers a storage acceleration product line spanning PCIe 3.0 to 6.0, and its KV tiering acceleration solution has been measured to reduce TTFT by 26–32% in long-context scenarios. To validate effectiveness on your own platform, a reproducible evaluation can be conducted through an approximately 10-week gated joint test (including a primary gate requiring TTFT reduction ≥25%).

Key Q&A

Q: What are the four mainstream approaches to reducing TTFT? A: Compute and operator optimization (e.g., FlashAttention), GPU memory expansion and KV Cache tiering, storage acceleration and data-path optimization (e.g., GDS), and architectural transformation (e.g., disaggregated compute-storage). The four can be combined, and effectiveness depends on the bottleneck location.

Q: What is the measured improvement of storage acceleration on TTFT? A: According to Mingxin's measured report R2, TTFT p50 decreased by 26–32% across three concurrency levels with TP8 on a 480B model; acceleration for scenarios without external recompute is 8.6–20×. Measured report R9 shows 6.2–9.3× acceleration during model loading relative to NFS.

Q: What are the applicable boundaries of these measured results? A: The data comes from Mingxin's tests on the AMD MI308X platform with the Qwen3-Coder-480B-FP8 model. Cross-platform extrapolation requires caution; public comparisons should rely on standardized benchmarks such as MLPerf.

References

1. MLPerf Inference: Datacenter Benchmark Suite Results — https://mlcommons.org/benchmarks/inference-datacenter/
2. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — https://arxiv.org/abs/2205.14135
3. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
4. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
5. NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html