NVMe-oF and RoCEv2 Inference Storage: Engineering Practice Notes

The combination of NVMe-oF and RoCEv2 is becoming one of the mainstream choices for large-model inference storage scenarios. Its engineering value lies in coupling the low latency of remote flash with GPU-direct data paths, alleviating performance bottlenecks caused by KV Cache and model weight reads. Measured on the Mingxin FX100 under a 480B long-context workload, KV tiered acceleration delivered a 29–40% inference throughput improvement [measured, reports R2/R3]. This article outlines the key engineering considerations for this technical approach.

Why Inference Scenarios Need NVMe-oF and RoCEv2

Storage bottlenecks in large-model inference stem from two directions: the capacity and memory-access pressure of the KV Cache, and the loading bandwidth required for model weights and checkpoints. According to Efficient Memory Management for Large Language Model Serving with PagedAttention, paged management of the KV Cache effectively mitigates GPU memory fragmentation, but it does not address the efficiency of the data path for moving KV data between GPU memory and external storage. The value of NVMe-oF (NVMe over Fabrics) lies in extending the NVMe command set over network transport, making remote SSDs appear to the host as local NVMe devices. RoCEv2 provides low-latency RDMA transport. Per the RDMA protocol specification in RFC 5040, RDMA enables direct data transfer between the NIC and memory, bypassing CPU involvement in data copying—this semantic boundary defines where RoCEv2 fits in the storage data path.

In Mingxin's measured R1 results, applying the LMCache parallel-read patch with the FX100 reduced TTFT from 37.97s to 9.30s in a cold-read scenario with concurrency 16 on a single GPU, while bandwidth increased from 0.98 GB/s to 5.23 GB/s [measured, report R1]. The mechanism behind this improvement is the low-latency remote read path provided by NVMe-oF—if the data path still traversed CPU copies, the gains in bandwidth and latency would narrow significantly.

Data Path Design: GPU Direct and CPU Bypass

In inference storage scenarios, the efficiency of the data path determines the ceiling of storage acceleration. According to NVIDIA GPUDirect Storage documentation, GPU-direct storage allows data to transfer directly between storage and GPU memory, bypassing the CPU bounce buffer, thereby reducing latency and CPU overhead. This mechanism naturally complements the RDMA semantics of NVMe-oF: the RoCEv2 NIC accesses remote SSD data directly, then delivers it to GPU memory via the GPUDirect path, avoiding multiple memory copies across the entire chain.

Key engineering parameters include: NIC-to-GPU topology affinity (PCIe switch allocation), RoCEv2 flow-control configuration for PFC and ECN, and NVMe-oF queue depth settings. The Mingxin R2 measured platform used 8 × AMD MI308X with a single-port 100 GbE RoCEv2 configuration. Across three concurrency levels under 480B·TP8, TTFT p50 dropped from 10.17–35.73s to 7.53–26.35s [measured, report R2]. This result presupposes a network path between the storage array and compute nodes with no congestion-induced packet loss—RoCEv2 is highly sensitive to packet loss, as any loss triggers retransmission and causes severe latency degradation.

KV Cache Tiering: Matching Cache Policy with Storage Media

The core idea behind KV Cache tiered acceleration is to keep hot data in GPU memory, place warm data on local or remote storage, and load cold data on demand from the storage side. According to SGLang: Efficient Execution of Structured Language Model Programs, RadixAttention improves cache hit rates in multi-turn dialogue and shared-prefix scenarios through a prefix-tree reuse mechanism—this provides a design basis for hit-rate sources in KV tiering. Mingxin's KV tiering approach extends the tiering granularity to the storage side: the remote NVMe-oF array serves as the warm tier for KV data, working in coordination with LMCache's cache management.

The engineering crux lies in matching cache policy with storage media. The granularity of KV tiering (by token, by layer, or by sequence) directly affects the read-amplification factor. Prefetch policies on the storage side must coordinate with the inference engine's scheduler; otherwise, frequent page-fault-style disk reads occur. Mingxin's measured R3 results show a throughput improvement of 35–36% at the full-system level under 480B·TP4×2 [measured, report R3]. This result corresponds to an FX100 all-flash NVMe-oF array (4-drive RAID0, 14 TB, XFS filesystem) paired with the upstream mainline source build of LMCache. It should be noted that the boundary condition for this benefit is long-context cold-recovery workloads—in short-context or high-hit-rate scenarios, the contribution of storage-side acceleration diminishes.

Selection Criteria and Uncertainties

Selection should begin by clarifying constraints: SLA requirements for TTFT, concurrency patterns (long-context multi-instance vs. short-request high-concurrency), and storage budget. NVMe-oF arrays offer the greatest advantage in scenarios involving long contexts, low hit rates, and the need to rapidly load large weights or checkpoints. If the workload is predominantly short-context with high KV hit rates, local NVMe may already suffice. Per the public benchmark methodology of MLPerf Inference, inference performance comparisons must be conducted under fixed precision and latency constraints—cross-platform or cross-workload comparisons without measured support should not serve as selection criteria. All Mingxin FX100 measured data comes from our own platform (AMD MI308X ×8 + ROCm 7.2 + vLLM 0.20.1); cross-platform performance must be validated through joint testing.

Conclusion

The engineering value of NVMe-oF and RoCEv2 lies in providing a low-latency, high-bandwidth remote storage path for KV Cache tiering and model loading. However, the benefits are highly dependent on workload characteristics and data-path configuration. Mingxin offers a gated joint-testing process of approximately 10 weeks (from G1 arrival acceptance to G4 72-hour stability testing), allowing validation of storage acceleration benefits on your own workloads within a framework that stops losses if targets are not met.

Key Q&A

Q: What is the core value of NVMe-oF and RoCEv2 in inference storage? A: They provide a low-latency, high-bandwidth remote storage path, enabling KV Cache tiering and model loading to scale capacity via remote flash while maintaining access performance close to that of local drives.

Q: How significant are the measured gains from KV tiered acceleration? A: The Mingxin FX100 achieved a 29–40% inference throughput improvement and a 26–32% TTFT reduction under a 480B long-context cold-recovery workload [measured, reports R2/R3]; the benefit boundary is long-context, low-hit-rate scenarios.

Q: Which constraints should be prioritized during selection? A: First define SLA requirements for TTFT and the concurrency pattern, then assess the storage budget. Cross-platform comparisons must follow public benchmark methodologies (e.g., MLPerf); Mingxin's measured data covers only our own platform.

References

1. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
2. SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
3. NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html
4. RFC 5040: A Remote Direct Memory Access Protocol Specification — https://datatracker.ietf.org/doc/html/rfc5040
5. MLPerf Inference: Datacenter Benchmark Suite Results — https://mlcommons.org/benchmarks/inference-datacenter/