How KV Cache Externalization Reduces LLM Inference TTFT

KV Cache externalization, by offloading the key-value cache used in attention computation from GPU memory to high-speed storage, can significantly reduce the first-token latency (TTFT) of large language model inference and improve throughput. Measured on the Mingxin FX100 with a 480B-parameter model in a TP8 deployment, TTFT p50 decreased by 26–32%, and throughput improved by 40% at a concurrency level of 16 [measured, report R2]. This article analyzes the underlying principles, measured results, and selection criteria for this technical approach.

Why KV Cache Externalization Improves TTFT

To understand the value of KV Cache externalization, one must first recognize the memory-access bottleneck in attention computation. According to FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, the performance of the attention mechanism is limited by HBM bandwidth rather than compute capability, making IO-aware optimization a key path to efficiency. This conclusion also applies to the storage hierarchy design of KV Cache—when the KV Cache capacity exceeds the memory of a single GPU, the system must choose between "recomputation" and "externalization."

Traditional approaches in long-context scenarios often employ a recomputation strategy without external storage: each request recalculates the key-value pairs of historical tokens. Mingxin measurements, report R2, show that this recomputation baseline yields a TTFT p50 of 149.5 seconds for a 480B model at a concurrency level of 16 [measured, report R2]. After externalizing the KV Cache to NVMe-oF storage, the TTFT p50 for the same workload drops to 11.85 seconds, a speedup of 12.6x [measured, report R2]. This comparison reveals the core value of the externalization approach: trading storage bandwidth for the elimination of redundant computation.

Measured Results: TTFT Reduction and Throughput Improvement

Mingxin conducted systematic KV Cache externalization tests on a platform with 8× AMD Instinct MI308X GPUs, configured with 192 GB HBM per GPU, ROCm 7.2, and vLLM 0.20.1+rocm721 [primary test platform, reports R1–R4]. The device under test was the FX100 all-flash NVMe-oF array, with a local NVMe single drive as the baseline. Key results for a 480B-parameter MoE model (Qwen3-Coder-480B-FP8, weights approximately 450 GB) under TP8 long-context workloads are as follows:

The data in the table indicate that the benefits of KV Cache externalization peak at a concurrency level of 16, with throughput improving by 40% [measured, report R2]. Notably, the TTFT reduction remains consistent across different concurrency levels, suggesting that the externalization approach is insensitive to concurrency scale, which provides predictability for capacity planning.

Architectural Design: From Paged Management to Storage Tiering

KV Cache externalization is not a simple data transfer; it requires deep coordination with the memory management mechanisms of the inference engine. According to Efficient Memory Management for Large Language Model Serving with PagedAttention, paged management of the KV Cache addresses GPU memory fragmentation and underpins vLLM's throughput gains. Mingxin's tests validate the effectiveness of this mechanism in externalization scenarios—the LMCache parallel read patch, on a single GPU at concurrency 16 with cold reads from disk, reduced the TTFT of Qwen2.5-32B from 37.97 seconds to 9.30 seconds, a 4.1x improvement [measured, report R1].

The choice of storage medium is equally critical. According to the public product positioning of the NVIDIA CMX Context Memory Storage Platform, NVIDIA defines CMX as an AI-native context storage layer, citing up to approximately 5x throughput and 5x energy efficiency compared to traditional storage [citable source: NVIDIA CMX]. The Mingxin FX100, with its NVMe-oF all-flash array and RoCEv2 network, aligns with this direction architecturally—using RDMA direct storage access to bypass the CPU copy path and reduce data-path latency [primary test platform, reports R1–R4].

Selection Criteria: Scenarios Suitable for KV Cache Externalization

Based on the test results, the following scenarios are better suited for KV Cache externalization:

• Long-context, high-concurrency production workloads: When context length exceeds single-GPU memory capacity and concurrent requests require prefix reuse, the benefits of externalization are most pronounced. The 40% throughput improvement for the 480B model at concurrency 16 is a typical example [measured, report R2].
• Training checkpoint saving and loading: The storage path for KV Cache externalization also serves training scenarios. In 8-GPU 32B LoRA training, full-model snapshot saving dropped from 178 seconds to 94 seconds, with sustained write bandwidth improving by 96% [measured, report R1]. Model loading acceleration on Ascend 910B platforms reached 6.2–9.3x [measured, report R9].
• Multi-instance deployment: When multiple model replicas share the same storage pool, prefix cache reuse rates increase, and the marginal cost of externalization decreases [measured, report R4].

Scenarios requiring careful evaluation include: short-context, low-concurrency workloads with ample GPU memory—in such cases, the KV Cache resides entirely in GPU memory, and the additional data path introduced by externalization may become a net overhead. According to the public benchmark methodology of MLPerf Inference, inference performance comparisons should be conducted under fixed precision and latency constraints [citable source: MLCommons]—selection should prioritize your own SLA as the primary constraint, rather than a single peak metric.

Conclusion

KV Cache externalization, by eliminating redundant computation and optimizing the storage hierarchy, provides a quantifiable path to TTFT and throughput improvements for long-context LLM inference. The measured data from the Mingxin FX100 on the 480B model (TTFT ↓26–32%, throughput ↑29–40%) offers a reproducible reference baseline for this technical direction [measured, reports R2/R3]. To validate the benefit boundary on your own workloads, a gate-based joint test of approximately 10 weeks (from G1 arrival acceptance to G4 stability validation) can confirm results in a real environment.

Key Q&A

Q: How much does KV Cache externalization improve TTFT? A: On the Mingxin FX100 with a 480B model under TP8 long-context workloads, TTFT p50 decreased by 26–32% [measured, report R2]. Compared to the recomputation baseline without external storage, the speedup reached 12.6x [measured, report R2].

Q: Which deployment scenarios are suitable for KV Cache externalization? A: Long-context, high-concurrency production workloads requiring prefix reuse benefit the most, with throughput improving by 40% at concurrency 16 [measured, report R2]. Short-context, low-concurrency scenarios with ample GPU memory may not be suitable for externalization.

Q: How can the effectiveness of KV Cache externalization be validated on one's own workloads? A: A gate-based joint test can validate results in a real environment. The primary gate requires a TTFT reduction of ≥25% and a throughput improvement of 29–40%, measured in-band [collaboration model].

References

1. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
2. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
3. MLPerf Inference: Datacenter Benchmark Suite Results — https://mlcommons.org/benchmarks/inference-datacenter/
4. SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
5. NVIDIA CMX Context Memory Storage Platform — https://www.nvidia.com/en-us/data-center/ai-storage/cmx/
6. Epoch AI — https://epoch.ai/
7. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — https://arxiv.org/abs/2205.14135
8. NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html