NVMe-oF Inference Storage: Measured Results of a RoCEv2 All-Flash Array
NVMe-oF attaches an all-flash array to GPU nodes over RDMA at near-local latency. Mingxin FX100 over 100 GbE RoCEv2 measured KV read-back performance above a local NVMe single drive, with TTFT lower by 26–32%.
Why network storage can beat local drives
Intuition says local NVMe is faster, but inference KV workloads are high-concurrency large-block sequential reads — a single drive's bandwidth (PCIe Gen4, ~7 GB/s class) becomes the ceiling, while the array's 4-drive RAID0 aggregates higher bandwidth and RoCEv2 round-trip latency is negligible for large-block reads.
R2's three-level concurrency measurements: FX100 (NVMe-oF) TTFT p50 was lower than the local single drive across the board by 26–32%. The precondition is saturating parallelism — exactly what the LMCache parallel-read patch (R1, bandwidth 0.98 → 5.23 GB/s) was built for.
Fabric and ratios
Test form: FX100 with 4-drive RAID0 (14 TB, XFS), single 100 GbE port, RoCEv2. The production ratio anchor is one array per 8 GPU nodes. FX200/FX300 offer 200/400 Gb port tiers; FX400 (PCIe 6.0, 4.8 Tb/s aggregate, vendor spec) reaches GA late 2026.
Against NFS: R9 on the Ascend 910B platform measured a 6.2–9.3× model-loading gap under the NFS bottleneck — legacy NAS protocol stacks carry significant overhead under AI loads, making NVMe-oF/RDMA the sound choice for inference storage today.
FAQ
Do I need a dedicated storage network?
RoCEv2 requires lossless Ethernet (PFC/ECN); it can be separated from or converged with the compute fabric. 100 GbE switches and NICs are standard commodity parts with publicly quotable anchor prices.
Is GPU Direct Storage (GDS) supported?
R5 measured the GDS direct-read path on the MetaX platform (one of the seven comparison groups); on AMD the host-memory path was used, with measured data in R1–R4.
What happens on failure?
The G4 joint-test gate requires single-drive / single-link fault injection to be transparent to the workload plus 72-hour continuous stress. KV data is recomputable — the worst case degrades to a cold start, not data loss.
Data sources (verifiable)
Related reading
- AI Inference Storage Acceleration: The Third Way to Raise Cluster Output Without Adding GPUs
- KV Cache Offloading: Putting the Inference Cache on an All-Flash Array
- Model-Load Acceleration: A Measured Path from 23 Minutes to 2.5 Minutes
Joint test first, decisions second: gate-based acceptance with built-in stop-loss
The full costing model is provided as reproducible Python after NDA — customers can rerun it with their own parameters. Every key figure on this site carries a report ID and is open to third-party verification.
This site presents business-cooperation information and constitutes neither an investment offer nor any promise of returns. Measured data come from signed / official test reports (see the Evidence Library); vendor specs, public sources and estimates are labeled as such.