NVMe-oF Performance Bottlenecks in KV Cache Scenarios and the Mingxin FX Series Solution

What Real Performance Bottlenecks Does NVMe-oF Face in KV Cache Scenarios?

NVMe-oF (NVMe over Fabrics) is widely regarded as a critical path for addressing KV Cache external memory expansion, but in actual LLM inference deployments, it does not automatically deliver its theoretical bandwidth advantages. The core contradiction lies in the compounding of three factors: protocol stack overhead, IO path misalignment, and cold-read amplification effects, which together keep end-to-end TTFT (Time to First Token) high—particularly in long-context cold-restore scenarios for 480B-class MoE models.

As measured in report R2, under a TP8 configuration, the cold-read TTFT p50 for Qwen3-Coder-480B-FP8 reached 10.17–35.73s. Meanwhile, the baseline local NVMe single disk, though bandwidth-limited, delivered more predictable first-token responses under low concurrency due to zero network hops and an extremely low-latency path. This reveals a key fact: the value of NVMe-oF is not determined by peak bandwidth, but by actual service latency and throughput stability under KV Cache's random small-block, high-frequency, low-latency-sensitive workloads—the "third quadrant" that traditional storage arrays and general-purpose RDMA networks struggle to address simultaneously.

Further analysis of R1 measured data shows that when the LMCache parallel-read patch is enabled, the cold-read TTFT for Qwen2.5-32B at single-card concurrency 16 dropped from 37.97s to 9.30s (a 4.1× improvement), with corresponding disk-side bandwidth rising from 0.98 GB/s to 5.23 GB/s (↑5.3×). This jump did not come from a network upgrade, but from KV tiered scheduling strategies that redirect and aggregate NVMe-oF channels—indicating that the bottleneck lies not in the physical link, but in the software-defined data-path orchestration capability.

Why Does the Standard NVMe-oF Architecture Struggle to Meet KV Cache Access Patterns?

KV Cache access patterns exhibit three atypical characteristics: extremely fine granularity (KB-level KV pairs), strong temporal dependencies (decoder step-by-step serial triggering), and cross-node prefix reuse requirements (multi-turn shared prefixes). These fundamentally misalign with traditional block storage optimization goals designed for large-file sequential reads and writes.

The SGLang paper points out that RadixAttention significantly improves multi-turn hit rates through prefix-tree reuse, but this mechanism relies on low-latency, high-concurrency KV retrieval responses. Standard NVMe-oF implementations, however, often merge multiple small IOs into large IOs to boost throughput, which introduces additional queuing and scheduling latency that disrupts the decoder's stepwise rhythm. The Mooncake architecture similarly emphasizes that KV-centric disaggregated compute-storage requires a reconstructed data plane rather than simply reusing a general-purpose storage protocol stack.

Mingxin's R2 test platform (8×MI308X + RoCEv2) confirmed through measurement: without KV tiered acceleration enabled, the FX100 array's cold-read bandwidth reached only 0.98 GB/s (R1 measured), far below its nominal 100GbE (≈12.5 GB/s) theoretical ceiling. The reason is that vLLM's default KV loading strategy does not adapt to the RDMA semantic boundaries of NVMe-oF—for example, the RDMA protocol defined in RFC 5040 requires explicit memory registration and zero-copy paths, whereas the traditional POSIX file interface routes through the Linux kernel block layer, introducing bounce buffers and multiple context switches that negate RDMA's advantages.

How Does the Mingxin FX Series Achieve NVMe-oF and KV Cache Co-Optimization?

The Mingxin FX series is not merely high-bandwidth NVMe-oF hardware; it forms a closed-loop optimization system through a hardware-aware KV tiered scheduling engine + ROCm-native RDMA driver stack + vLLM custom patches. Its core breakthroughs are threefold:

1. KV Tiered Acceleration Engine: An embedded lightweight KV metadata index and prefetch decision unit within the FX100 controller maps logical KV requests by heat/temporal patterns across different media tiers (HBM cache → NVMe SSD → remote NVMe-oF targets), avoiding full external memory loading. R2/R3 measurements show that in 480B production deployments, this mechanism improves inference throughput by +29–40% (concurrency 8: +29%; optimal operating point concurrency 16: +40%).

2. RoCEv2 Native Driver Stack: Bypasses the Linux kernel block layer by building a user-space IO path directly on the ROCm HIP-RDMA API, eliminating bounce buffers and kernel scheduling latency. In R1 measurements, sustained checkpoint write bandwidth improved from 3.26 GB/s to 6.40 GB/s (+96%), confirming the path's effectiveness for sustained write workloads. For KV Cache small-block random reads, TTFT p50 decreased by 26–32% (R2 measured).

3. vLLM-LMCache Joint Tuning: Deep integration with upstream vLLM 0.20.1+rocm721 and the LMCache mainline (2026-06-29), supporting cross-node KV shard pooling and RoCE-aware batch scheduling. R1 measurements show a 4.1× TTFT improvement for Qwen2.5-32B cold reads, with bandwidth utilization reaching 5.23 GB/s.

The table below summarizes FX100's measured gains in key KV Cache scenarios:

Conclusion: From Protocol Adaptation to Semantic Co-Optimization

NVMe-oF is not a "silver bullet" for KV Cache acceleration; it is a technology carrier that must be deeply coupled with model execution semantics. The value of the Mingxin FX series lies in placing storage hardware, the RDMA protocol stack, and the LLM inference framework on the same optimization plane, using reproducible measured TTFT and throughput metrics to validate the engineering feasibility of KV-centric disaggregated compute-storage. Mingxin has now opened a G1–G4 four-stage gated joint-testing process (including a Python-reproducible model after NDA), and welcomes compute centers and large-model teams to pursue customized adaptation collaborations.

Key Q&A

Q: How much does the FX100 reduce TTFT in KV Cache scenarios as measured?
A: Under 480B·TP8 across three concurrency levels, TTFT p50 decreased by 26–32%, with a range of 7.53–26.35s (R2 measured).

Q: What is the acceleration factor of the FX100 for scenarios without external memory recomputation?
A: TTFT p50 dropped from 149.5s (conc16) to 11.85s, an acceleration of 8.6–20× (R2 measured).

Q: What is the model-loading speedup of the FX100 on the Huawei Ascend platform compared to NFS?
A: DeepSeek-70B load time was reduced from 1399s to 150s, a 9.3× speedup (R9 measured).

References

1. SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
2. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
3. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
4. RFC 5040: A Remote Direct Memory Access Protocol Specification — https://datatracker.ietf.org/doc/html/rfc5040
5. NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html
6. The Linux Kernel documentation — https://www.kernel.org/doc/html/latest/