Mingxin

Measured Comparison: External KV Tiering vs Local NVMe vs No External Storage Recompute – How Big Is the Gap?

KV Cache存储加速LMCachevLLM

In long-context inference scenarios for 480B-class large models, the external KV tiering acceleration solution (e.g., Mingxin FX100 with LMCache) achieves a 26–32% reduction in time-to-first-token (TTFT) and a 29–40% improvement in inference throughput compared to the local NVMe storage and no-external-storage recompute baselines. In extreme no-external-storage scenarios, it delivers an 8.6–20x acceleration. These findings are based on measured data from reports R2 and R3, providing a clear quantitative reference for compute centers selecting KV Cache storage acceleration solutions.

Background: Storage Bottlenecks in Long-Context Inference and Three Approaches

In large model inference, the storage and loading efficiency of KV Cache directly determines time-to-first-token (TTFT) and concurrent throughput. When the model reaches 480B parameters (e.g., Qwen3-Coder-480B-FP8, weights approximately 450 GB) and the context length exceeds the memory capacity of a single GPU, external storage is required for KV tiering management. Current mainstream approaches include:

  • No external storage recompute: KV Cache is not saved; each request recomputes from scratch. This minimizes memory usage but results in extremely high latency.
  • Local NVMe storage: KV Cache is written to a local NVMe SSD as a low-cost external storage solution.
  • External KV tiering acceleration: Dedicated storage devices (e.g., Mingxin FX100 all-flash NVMe-oF array) and the LMCache framework enable fast KV data tiering and retrieval.

The test platform consists of 8× AMD Instinct MI308X (192 GB HBM per card), 2× AMD EPYC 9654, vLLM 0.20.1 + LMCache, and the model Qwen3-Coder-480B-FP8 (MoE). The device under test is the Mingxin FX100 all-flash array (4 drives RAID0, 14 TB, RoCEv2, single-port 100 GbE). The baseline is a local NVMe single drive (PCIe Gen4, 2 TB), and the no-external-storage recompute baseline is a pure compute mode.

Measured Comparison: TTFT, Throughput, and Acceleration Ratio

1. Time-to-First-Token (TTFT): External KV Tiering vs Local NVMe vs No External Storage Recompute

In tests with the 480B model at TP8 and three concurrency levels (conc8/16/32), TTFT data (measured, report R2) are as follows:

  • No external storage recompute: TTFT p50 reaches as high as 149.5 seconds (conc16); even at conc8, it ranges from 10.17 to 35.73 seconds.
  • Local NVMe: TTFT p50 drops to 7.53–26.35 seconds (conc8–32), representing a 26–32% reduction compared to no external storage recompute (measured, report R2).
  • External KV tiering (FX100): TTFT p50 further decreases to 11.85 seconds (conc16), a 92% reduction (8.6x) compared to no external storage recompute. At the optimal operating point (conc16), TTFT drops from 149.5 seconds to 11.85 seconds, achieving a 12.6x acceleration (measured, report R2).

Conclusion: The external KV tiering solution provides an additional 26–32% improvement in TTFT over local NVMe and a significant 8.6–12.6x advantage over no external storage recompute.

2. Inference Throughput: External KV Tiering vs Local NVMe vs No External Storage Recompute

In tests with the 480B model at TP4×2 across the full machine, throughput data (measured, report R3) are as follows:

  • No external storage recompute: Throughput is only 4.1 tok/s (conc16).
  • Local NVMe: Throughput increases to 74.9 tok/s, an 18.3x improvement over no external storage recompute.
  • External KV tiering (FX100): At the optimal operating point (conc16), throughput reaches 74.9 tok/s (matching local NVMe). At conc8, throughput improves by 29%; at conc16, by 40% (measured, reports R2/R3). Across the full machine (TP4×2), throughput improves by 35–36% (measured, report R3).

Note: While local NVMe achieves throughput close to FX100, its TTFT is higher, and a single NVMe drive is prone to IOPS bottlenecks under concurrency pressure. The FX100’s 16M IOPS (PCIe 3.0) supports higher concurrency, whereas local NVMe sees TTFT rise to 26.35 seconds at conc32, while FX100 remains stable below 20 seconds (measured, report R2).

3. Extreme Scenario Acceleration Without External Storage

For cold-start or long-context requests without pre-stored KV Cache, no external storage recompute results in a TTFT of up to 149.5 seconds (conc16). The external KV tiering solution, using LMCache’s parallel read patch, reduces TTFT from 37.97 seconds to 9.30 seconds (4.1x improvement) and increases bandwidth from 0.98 GB/s to 5.23 GB/s (5.3x) (measured, report R1). This demonstrates that external storage acceleration is even more effective in extreme scenarios.

Solution Selection Recommendations: Optimal Choice Depends on the Scenario

  • High-concurrency, low-latency scenarios (e.g., API inference services): The external KV tiering solution is optimal, with TTFT reduced by 26–32% and throughput improved by 29–40%, while supporting higher concurrency (e.g., TTFT remains controllable at conc32).
  • Cost-sensitive scenarios (e.g., offline batch processing): Local NVMe can meet basic needs, with throughput close to FX100, but TTFT is higher, and attention must be paid to NVMe single-drive lifespan and IOPS limitations.
  • No external storage recompute: Suitable only for very short contexts or scenarios with sufficient GPU memory; otherwise, TTFT becomes unacceptable.

Conclusion

Measured data show that the external KV tiering acceleration solution (e.g., Mingxin FX100) in long-context inference for 480B models reduces TTFT by 26–32% and improves throughput by 29–40% compared to local NVMe, and achieves 8.6–20x acceleration compared to no external storage recompute. For compute centers pursuing low latency and high concurrency, this solution is a currently producible optimal path. Mingxin Technology offers approximately 10 weeks of gated joint testing (G1–G4), which can verify metrics such as TTFT reduction ≥25% and throughput improvement of +29–40%, with a stop-loss mechanism if targets are not met. Teams with KV Cache storage acceleration needs are welcome to contact us for joint testing collaboration.

Key Q&A

Q: How much does external KV tiering acceleration improve time-to-first-token compared to local NVMe? A: In tests with the 480B model at TP8 and three concurrency levels, TTFT p50 is reduced by 26–32% (measured, report R2), from 10.17–35.73 seconds to 7.53–26.35 seconds.

Q: What is the acceleration ratio of external KV tiering in the no-external-storage recompute scenario? A: Under the 480B·conc16 workload, TTFT drops from 149.5 seconds to 11.85 seconds (8.6x), and throughput increases from 4.1 to 74.9 tok/s (18.3x) (measured, report R2).

Q: What is the measured throughput improvement range for the Mingxin FX100? A: Across the full machine with the 480B model at TP4×2, throughput improves by 35–36% (measured, report R3); at the optimal operating point (conc16), it improves by 40% (measured, report R2).

Generated by Mingxin's content engine with automated QC; headline numbers cite signed test reports (see the evidence library). Translated from the Chinese original. Questions or corrections: contact us.