Why KV Cache Tiered Storage Boosts 480B LLM Inference Throughput by 29–40%: A Data-Driven Breakdown

KV Cache tiered storage technology, by retaining hot data in GPU memory and offloading cold data to an NVMe-oF flash array, achieves a throughput improvement of 29–40% (measured, report R2/R3) and a Time-to-First-Token (TTFT) reduction of 26–32% in long-context inference for a 480B-parameter MoE model. This gain does not come from computational acceleration but from breaking the GPU memory bottleneck and intelligent scheduling across storage tiers. Below is a data breakdown from three dimensions.

The GPU Memory Wall and the KV Cache Storage Contradiction

The core contradiction in current LLM inference is that GPU memory capacity is far smaller than the total demand for model weights and KV Cache. Taking Qwen3-Coder-480B-FP8 (MoE, weights ≈ 450GB) as an example, on an 8×MI308X platform (each with 192GB HBM, total 1.5TB), GPU memory must simultaneously accommodate model weights, activations, optimizer states, and KV Cache. In long-context scenarios, KV Cache grows linearly with sequence length. A typical 128K token KV Cache occupies about 50–80GB. With 16 concurrent requests, the KV Cache alone requires 800GB–1.28TB—far exceeding GPU memory capacity.

Traditional solutions rely on recomputation without external storage (i.e., recalculating all KV Cache for each request), resulting in a TTFT as high as 149.5s (conc16, measured, report R2) and a throughput of only 4.1 tok/s. Another approach uses CPU memory or NFS as extended storage, but PCIe bandwidth and network latency degrade performance. The tiered strategy of Mingxin FX100, using LMCache, treats GPU memory as L1 cache and the NVMe-oF all-flash array as L2 cache. Through tiered offloading and prefetching of KV Cache, it avoids recomputation when GPU memory is insufficient.

Cold Recovery Scenario: How Tiered Storage Achieves 29–40% Throughput Improvement

In a 480B production deployment with long-context cold recovery workload (measured, reports R2/R3), the KV Cache tiered storage of FX100 demonstrates clear throughput gains:

• Concurrency 8 (lower bound): Throughput improvement +29%. Here, GPU memory pressure is relatively low, and tiered storage primarily optimizes occasional cold data access, reducing recomputation frequency.
• Concurrency 16 (upper bound): Throughput improvement +40%. Under high concurrency, GPU memory is fully saturated. Tiered storage offloads most cold KV Cache to flash memory, while LMCache’s parallel read patch (measured, report R1) reduces single-GPU cold-read TTFT from 37.97s to 9.30s (4.1× improvement) and increases bandwidth from 0.98 GB/s to 5.23 GB/s (5.3× improvement).
• Full-system scope (TP4×2): Throughput improvement 35–36%. In cross-node scenarios, tiered storage reduces communication wait times caused by KV Cache misses between nodes.

The core mechanism is: GPU memory retains only recently active KV Cache (hot data), while historical or infrequently accessed KV Cache is compressed and written to the NVMe-oF array, then read back at speeds close to local NVMe when needed. The FX100 single-interface 100Gb (measured bandwidth ≈ 11.5 GB/s), combined with LMCache’s prefetching strategy, reduces cold data access latency from seconds (recomputation) to hundreds of milliseconds (flash readback), thereby maintaining throughput close to GPU memory hit rates under high concurrency.

TTFT Reduction of 26–32%: Latency Benefits of Tiered Storage

TTFT is a key user experience metric for long-context inference. Measured data from report R2 shows that under 480B·TP8 with three concurrency levels, FX100 reduces TTFT p50 from 10.17–35.73s to 7.53–26.35s, a reduction of 26–32%. This gain primarily comes from two aspects:

1. Avoiding recomputation without external storage: For requests that need to access historical KV Cache (e.g., conversation continuation, long document analysis), tiered storage directly returns cached KV Cache instead of recomputing. At conc16, the recomputation baseline TTFT is 149.5s, while FX100 requires only 11.85s (12.6× acceleration).
2. LMCache’s intelligent prefetching: The system prefetches KV Cache blocks likely to be accessed between GPU memory and flash memory based on the request’s context ID and position. In measured data from report R1, the single-GPU cold-read scenario shows a 4.1× TTFT improvement, indicating that the prefetching strategy effectively masks flash memory access latency.

Notably, the TTFT reduction (26–32%) is smaller than the throughput improvement (29–40%). This is because TTFT is primarily affected by the hit rate of the first KV Cache block, while throughput is more influenced by the overall cache hit rate and the cumulative effect of recomputation overhead.

Comparison with Recomputation Without External Storage: 8.6–20× Acceleration

The greatest benefit of tiered storage is evident when compared to the recomputation baseline without external storage. Measured data from report R2 shows:

• TTFT p50: Recomputation baseline 149.5s (conc16) vs. FX100 11.85s, acceleration 12.6×
• Throughput: Recomputation baseline 4.1 tok/s vs. FX100 74.9 tok/s, acceleration 18.3×
• Acceleration range: 8.6–20× (depending on concurrency and context length)

This comparison clearly demonstrates: When GPU memory is insufficient, tiered storage is a superior choice over recomputation without external storage. Although recomputation saves storage costs, the computational overhead grows quadratically with sequence length, leading to severe degradation in latency and throughput. The FX100’s NVMe-oF architecture (measured sustained write bandwidth 6.40 GB/s, 96% higher than local NVMe, measured, report R1) provides significantly higher bandwidth for saving and reading KV Cache than the memory bandwidth required for recomputation, yielding a net time benefit.

Conclusion

The core value of KV Cache tiered storage lies in trading flash memory bandwidth for GPU memory capacity, avoiding the computational explosion of recomputation in long-context, high-concurrency scenarios. Through the synergy of LMCache and the NVMe-oF all-flash array, Mingxin FX100 achieves measured improvements of 29–40% in throughput and 26–32% in TTFT on a 480B model, with performance gains expanding as concurrency and context length increase. For computing centers deploying hundred-billion-parameter models, this solution offers a cost-controllable performance optimization path beyond GPU memory expansion. For the full test report or to arrange a gated joint test (approximately 10 weeks, including G1–G4 phases), contact the Mingxin technical team to obtain a Python reproducible measurement model under NDA.

Key Q&A

Q: Why does KV Cache tiered storage improve LLM inference throughput?
A: By retaining hot data in GPU memory and offloading cold data to an NVMe-oF flash array, it avoids the computational overhead of recomputation without external storage. On a 480B model with concurrency 16, throughput improves by 40% (measured, report R2), and by 35–36% at the full-system scope (measured, report R3).

Q: How much can Time-to-First-Token (TTFT) be reduced?
A: Under 480B·TP8 with three concurrency levels, TTFT p50 is reduced by 26–32% (measured, report R2). In cold recovery scenarios, single-GPU cold-read TTFT improves by 4.1× (measured, report R1).

Q: Compared to recomputation without external storage, what is the acceleration factor of FX100?
A: TTFT acceleration is 12.6× (149.5s → 11.85s), throughput acceleration is 18.3× (4.1 → 74.9 tok/s), with an overall acceleration range of 8.6–20× (measured, report R2).