KV Cache Storage Product Comparison: Methodology and Measured Benchmarks

Different brands of KV Cache storage products in vLLM inference—let's state the conclusion upfront: without a unified testing methodology, any cross-brand comparison figures lack decision-making value. The key to comparison is not "who is faster" but whether test conditions are aligned—model scale, concurrency levels, context length, cold/hot hit ratios, storage media, and network topology—each factor can significantly alter results. This article provides a reproducible comparison methodology, using Mingxin FX100 measured data on the AMD MI308X platform as a benchmark example.

Why KV Cache Storage Product Comparisons Are Easily Distorted

KV Cache is the intermediate state in LLM inference that grows with sequence length. According to Efficient Memory Management for Large Language Model Serving with PagedAttention, the motivation for paged management of KV Cache stems from GPU memory fragmentation, and vLLM's throughput gains come from efficient GPU memory utilization. When context length exceeds single-GPU memory capacity, KV Cache must spill to the storage layer, where storage read/write latency and bandwidth directly determine end-to-end inference performance.

Cross-brand comparison distortion typically arises from three dimensions:

• Misaligned model and concurrency: A 480B MoE model and a 14B dense model have completely different KV Cache access patterns; concurrency 8 versus concurrency 32 differs by an order of magnitude in storage bandwidth pressure.
• Ambiguous hit-rate definitions: Prefix cache hits (such as shared prefix reuse exploited by SGLang's RadixAttention mechanism) significantly reduce storage access frequency. Comparing storage performance without disclosing hit rates is like comparing speeds under different workloads.
• Baseline selection bias: Whether the baseline is a local NVMe single drive or NFS, the resulting speedup multiples can differ by several times.

Reproducible Comparison Test Framework

We recommend designing comparison tests in the following five steps, with each step explicitly disclosed in the report:

1. Lock the inference stack: vLLM version, LMCache version, ROCm/CUDA version, and driver version all fixed; no party may upgrade independently.
2. Fix model and weights: Same model, same precision (e.g., FP8), same weight source, to avoid quantization differences interfering.
3. Multi-level concurrency sweep: Cover at least concurrency 8/16/32, reporting TTFT p50 and throughput for each level, rather than only the optimal value.
4. Disclose hit rates: Cold reads (no prefix hits) and hot reads (high hit rates) tested separately and reported individually.
5. Unified baseline: Use "no external memory recomputation" as the theoretical lower bound and local NVMe as the practical baseline, calculating speedup multiples against each.

Mingxin FX100 Measured Benchmark Example

Using the official test report (measured, reports R2/R3) of Mingxin FX100 on an 8× AMD MI308X platform as an example to illustrate the framework's practical application. Test conditions: Qwen3-Coder-480B-FP8 (MoE, weights ~450GB), vLLM 0.20.1+rocm721, LMCache built from upstream mainline source, RoCEv2 single-port 100GbE, 4-drive RAID0 all-flash array.

First-token latency specifics: under 480B·TP8 across three concurrency levels, TTFT p50 dropped from 10.17–35.73s to 7.53–26.35s (measured, R2). Compared to the recomputation baseline (TTFT p50 149.5s), FX100 at concurrency 16 reduced TTFT to 11.85s (measured, R2).

It must be emphasized that the above figures represent only Mingxin FX100's performance on the AMD MI308X platform under the specified conditions. For cross-platform comparisons (e.g., Ascend 910B vs. NVIDIA H100), Mingxin's internal testing only includes FX100 versus NFS baseline data (measured, R9: DeepSeek-70B service loading 1399s→150s, 9.3×), which does not constitute a relative quality judgment of any third-party product.

Common Pitfalls in Comparison Testing

• Reporting only the optimal concurrency point: Vendors tend to select the concurrency level most favorable to themselves. Require full-level sweep results rather than a single point.
• Mixing cold and hot hits: In hot-read scenarios, storage latency is diluted by hit rates—what's being measured is cache hit rate, not storage performance. Require cold-read (no prefix hits) data presented separately.
• Ignoring network topology: Differences between RoCEv2 and InfiniBand, single-port bandwidth versus multi-port bonding, all change the achievable performance of storage products. Test reports must specify network configuration.
• Using throughput instead of latency: Inference SLA is typically constrained by TTFT. Comparisons reporting only throughput without latency are meaningless for online inference decision-makers.

According to MLPerf Inference: Datacenter Benchmark Suite Results, the value of publicly comparable benchmarks lies in fixed precision and latency-constrained submission criteria—this is the proper posture for cross-brand comparison: align the rules first, then discuss outcomes.

Conclusion

The effectiveness comparison of KV Cache storage products is essentially a comparison of testing methodologies. Mingxin offers an approximately 10-week gated joint testing process (G1 arrival acceptance / G2 single-node baseline / G3 main gate: TTFT reduction ≥25%, throughput +29–40% measured in-band / G4 72-hour stability), with all figures reproducible and stop-loss if targets are not met. If you wish to validate FX100 or other storage solutions on your own platform, we welcome aligned-methodology testing within the joint testing framework.

Key Q&A

Q: What matters most when comparing different brands of KV Cache storage products in vLLM inference? A: Whether test methodologies are aligned: model scale, concurrency levels, context length, cold/hot hit ratios, storage media, and network topology. Comparison figures with misaligned methodologies have no decision-making value.

Q: What is Mingxin FX100's measured performance in KV Cache scenarios? A: On an 8× AMD MI308X platform with a 480B model, KV tiered acceleration improved inference throughput by +29–40% (concurrency 8 as lower bound, concurrency 16 as upper bound), with TTFT reduced by 26–32% (measured, R2/R3).

Q: What are common pitfalls in cross-brand comparisons? A: Reporting only the optimal concurrency point, mixing cold and hot hits, ignoring network topology, and using throughput instead of latency. Require full-level sweep data, cold-read data, complete network configuration, and use TTFT as the SLA constraint metric.

References

1. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
2. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
3. MLPerf Inference: Datacenter Benchmark Suite Results — https://mlcommons.org/benchmarks/inference-datacenter/
4. SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
5. NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html
6. Epoch AI — https://epoch.ai/
7. SNIA — Storage Networking Industry Association — https://www.snia.org/
8. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — https://arxiv.org/abs/2205.14135