How Tiered KV Cache Storage Improves LLM Inference Throughput

In the pursuit of higher LLM inference throughput, tiered KV Cache storage has been proven through measured results to be an effective and quantifiable optimization direction. Mingxin's formal test report on a 480B-parameter model across 8 AMD MI308X GPUs shows that the tiered storage solution improves inference throughput by 29–40% and reduces time-to-first-token (TTFT) by 26–32% [measured, reports R2/R3]. This article breaks down the optimization mechanism, key benefits, and applicable boundaries of tiered KV Cache storage, based on Mingxin's measured data.

Why KV Cache Becomes a Bottleneck for Inference Throughput

To understand the value of tiered storage, we first need to clarify the role of KV Cache during inference. When an LLM generates each token, the attention mechanism must read the Key and Value vectors of all preceding tokens. The size of this cache grows with sequence length and the number of concurrent requests, placing continuous pressure on GPU memory capacity. According to Efficient Memory Management for Large Language Model Serving with PagedAttention, KV Cache memory management directly impacts the throughput of inference systems; fragmentation and insufficient capacity lead to lower memory utilization and reduced batching scale [1].

Research on FlashAttention further reveals the core issue: attention computation is limited by HBM bandwidth, not compute capacity [2]. This means that when the KV Cache cannot fully reside in GPU memory, the bandwidth and latency of reading cache from external storage directly become the ceiling for inference throughput. Mingxin's measured data on a 480B model with TP8 across three concurrency levels confirms this: without tiered storage, TTFT p50 reached 10.17–35.73 seconds, while with tiered storage it dropped to 7.53–26.35 seconds [measured, report R2].

Core Mechanism of Tiered Storage: Hot/Cold Separation and On-Demand Scheduling

The basic idea of tiered KV Cache storage is to classify cache into hot and cold tiers based on access frequency: the hot tier resides in GPU memory, serving frequently reused prefixes; the cold tier is placed on external storage and loaded block-by-block only when needed. This design aligns with the KV Cache-centric disaggregated architecture proposed in Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving, which reduces single-node memory capacity constraints through cross-node pooling and tiered placement [3].

Mingxin's FX100 measured path uses an NVMe-oF all-flash array as the cold tier, connected directly to GPUs via RoCEv2 networking. According to NVIDIA GPUDirect Storage documentation, GPU-direct storage bypasses CPU memory copies, reducing additional latency in the data path [4]. Mingxin validated the effect of parallel read patches in test R1: in a single-GPU, concurrency-16, cold-read scenario (Qwen2.5-32B), TTFT dropped from 37.97 seconds to 9.30 seconds, and bandwidth increased from 0.98 GB/s to 5.23 GB/s [measured, report R1].

Measured Benefits: Quantified Throughput Gains and Latency Reductions

Mingxin's tests under a 480B production deployment covered multiple concurrency levels and deployment configurations. The table below summarizes the core measured metrics:

Notably, the throughput improvement varies with concurrency level: concurrency level 8 is the lower bound (+29%), while concurrency level 16 reaches the upper bound (+40%). This indicates that the benefits of tiered storage are more pronounced under higher concurrency pressure—because higher concurrency means a larger total context and greater opportunities for cache reuse.

The acceleration effect is even more pronounced when compared to recomputation without external storage: the recomputation baseline TTFT p50 was 149.5 seconds (concurrency 16), versus 11.85 seconds with FX100; throughput increased from 4.1 tok/s to 74.9 tok/s, an acceleration of 8.6–20× [measured, report R2]. This data reveals a key insight: in long-context scenarios, quickly loading from external storage is preferable to recomputing from scratch on cache misses.

Applicable Boundaries and Selection Considerations

Tiered storage does not yield equal benefits in all scenarios. Three applicable conditions can be derived from Mingxin's measured data:

First, context length must be sufficiently long. The 480B model with long-context cold-restore workloads is the baseline scenario for testing. In short-context scenarios, the total KV Cache size is small and fits entirely in GPU memory, leaving limited room for tiered storage gains.

Second, concurrency patterns must offer reuse opportunities. In scenarios such as multi-turn dialogue and shared prefixes, higher prefix cache reuse rates translate to higher hit rates for cold-tier loading. According to SGLang: Efficient Execution of Structured Language Model Programs, prefix-tree reuse mechanisms significantly improve cache hit rates in multi-turn dialogue scenarios [5].

Third, storage media and network bandwidth must match. Mingxin's test platform uses an all-flash NVMe-oF array with RoCEv2 networking at 100GbE per port. If the cold tier uses HDDs or insufficient network bandwidth, load latency may offset the optimization gains.

Additionally, the value of tiered storage extends beyond inference. Mingxin's tests on the Huawei Atlas 910B platform show model inference load times reduced from 691 seconds to 112 seconds (DeepSeek-32B, 6.2×) and from 1399 seconds to 150 seconds (DeepSeek-70B, 9.3×) [measured, report R9]. In training scenarios, Checkpoint save time dropped from 178 seconds to 94 seconds, with sustained write bandwidth improving by 96% [measured, report R1]. This means the same storage architecture can serve both inference and training workloads.

Conclusion

Tiered KV Cache storage converts GPU memory capacity constraints into a manageable storage bandwidth problem through hot/cold separation and on-demand scheduling, with measured throughput improvements of 29–40% and TTFT reductions of 26–32% [measured, reports R2/R3]. Its applicability depends on the combination of long context, high concurrency, and high-speed storage media. For teams planning inference infrastructure, we recommend validating actual benefits in gate-based joint testing, using your own workload's context length and concurrency patterns as constraints. Mingxin offers a joint testing collaboration model of approximately 10 weeks, including arrival acceptance, single-node baseline, primary gate (TTFT reduction ≥25%, throughput +29–40% measured in-band), and 72-hour stability validation, with stop-loss if targets are not met. The measurement model can be reproduced in Python after NDA.

Key Q&A

Q: How much throughput improvement can tiered KV Cache storage deliver? A: Mingxin's measured results on a 480B model across 8 AMD MI308X GPUs show inference throughput improvements of 29–40%, with concurrency level 8 as the lower bound (+29%) and concurrency level 16 as the upper bound (+40%) [measured, reports R2/R3]. Time-to-first-token (TTFT) simultaneously decreases by 26–32% [measured, report R2].

Q: Which scenarios are tiered storage suitable for? A: It is primarily suitable for inference workloads with long context, high concurrency, and prefix reuse opportunities. In short-context scenarios, the total KV Cache size is small and fits in GPU memory, leaving limited benefits for tiered storage. Storage media must be all-flash NVMe-oF class, and network bandwidth must match load requirements.

Q: Does tiered storage help in training scenarios? A: Yes. Mingxin's measured results show training Checkpoint save time reduced from 178 seconds to 94 seconds (8-GPU 32B LoRA, sustained write bandwidth +96%) [measured, report R1]; model inference loading on the Huawei 910B platform accelerated by 6.2–9.3× [measured, report R9].

References

1. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
2. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — https://arxiv.org/abs/2205.14135
3. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
4. NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html
5. SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104