How NVMe-oF Storage Accelerates KV Cache Inference Performance

Key Findings

An NVMe-oF-based tiered acceleration storage for KV Cache delivers 29–40% throughput gains and 26–32% reductions in time-to-first-token (TTFT) under long-context inference workloads for a 480B-parameter large model (measured, reports R2/R3). The performance benefit stems from tiering KV Cache off GPU HBM onto an NVMe-oF all-flash array, while prefix-tree reuse and parallel read optimization keep external storage access latency within acceptable bounds. For real-time database query scenarios—particularly mixed workloads with multi-turn sessions, shared prefixes, and long contexts—this architecture offers a viable scaling path under HBM capacity constraints.

KV Cache Capacity Bottlenecks and the Case for Tiered Storage

KV Cache in large-model inference grows linearly with sequence length, while GPU HBM remains limited. As described in Efficient Memory Management for Large Language Model Serving with PagedAttention, the motivation for paged management of KV Cache stems from HBM fragmentation and waste. When context lengths reach tens of thousands of tokens, KV Cache can occupy tens to hundreds of GB of HBM, forcing inference systems to trade off batch size against context length.

The tiered storage approach offloads less frequently accessed portions of KV Cache to external storage, reading them back to the GPU only when needed. According to Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving, a KVCache-centric disaggregated architecture alleviates single-node HBM pressure through prefix cache reuse and cross-node KV pooling. Mingxin FX100 follows a similar measured path but shifts the offload target from remote memory to an NVMe-oF all-flash array, achieving a larger KV capacity pool at lower cost.

NVMe-oF (NVMe over Fabrics) extends NVMe commands across the network via RDMA protocols (semantic boundaries defined in RFC 5040: A Remote Direct Memory Access Protocol Specification), enabling GPU-side access to remote flash with latency approaching local NVMe. Per NVIDIA GPUDirect Storage documentation, a GPU-direct data path can bypass CPU bounce buffers, reducing data movement overhead—a prerequisite for the NVMe-oF approach to be practical.

Measured Data: Quantified Benefits of Tiered Acceleration

Mingxin FX100 measurements on an 8× AMD MI308X platform (192 GB HBM per GPU) are as follows (measured, reports R2/R3):

It is important to note that these improvements do not arise from the storage device's own compute capability but from the synergy between tiering strategy and access-pattern optimization. The key workload in R2 testing was a long-context cold-recovery scenario for a 480B-parameter MoE model (Qwen3-Coder-480B-FP8, weights ~450 GB)—where all or most of the KV Cache must be reloaded from external storage. In this scenario, the parallel read capability of the NVMe-oF array directly determines recovery latency.

R1 measurements further decomposed the mechanism contributions: in a cold-read scenario with Qwen2.5-32B on a single GPU at 16-way concurrency, the LMCache parallel-read patch reduced TTFT from 37.97s to 9.30s (4.1× improvement) and increased bandwidth from 0.98 GB/s to 5.23 GB/s (5.3×). This indicates that storage-side bandwidth itself is not the bottleneck; rather, the degree to which the software stack exploits parallel reads is what matters.

Applicability Analysis for Real-Time Database Query Scenarios

Mapping the above results to real-time database query scenarios requires distinguishing two workload patterns:

Pattern 1: Multi-turn sessions and shared-prefix queries. According to SGLang: Efficient Execution of Structured Language Model Programs, RadixAttention's prefix-tree reuse mechanism significantly improves hit rates in multi-turn dialogue and shared-prefix scenarios. When multiple queries share system prompts, tool definitions, or historical context, prefix KV Cache can be reused without recomputation. Here, the value of NVMe-oF tiered storage lies in the larger KV pool: more prefixes can be retained rather than evicted due to HBM exhaustion.

Pattern 2: Long-context cold-start queries. When queries involve very long documents or require recovery from checkpoints, KV Cache must be loaded from external storage. The 8.6–20× speedups measured in R2 (versus the no-external-recompute baseline) map directly to this scenario. However, these speedup factors assume a baseline of "no external recompute"—i.e., every query computes KV from scratch—whereas production systems typically keep some KV resident in HBM.

For real-time database queries, the more relevant metric is the number of concurrent queries a system can support while meeting SLA constraints (e.g., TTFT below a threshold). The +40% throughput gain at 16-way concurrency measured in R2 implies that, with the same hardware budget, the system can handle more concurrent requests within the same SLA envelope, or allocate longer contexts per request. This is what "throughput improvement" means in a cost model—it does not directly equate to cost savings, but it does equate to expanded service capacity under equivalent resources.

Selection Guidance: Applicability Boundaries of the NVMe-oF Approach

NVMe-oF KV Cache acceleration is not a universal optimum; its applicability boundaries must be made explicit:

Suitable conditions:

• Workloads dominated by long contexts, multi-turn sessions, and shared prefixes, where KV Cache capacity demands far exceed single-GPU HBM
• Query concurrency exhibits clear hot/cold tiering, with most KV accesses concentrated on a small set of active prefixes
• Existing RoCEv2 or InfiniBand network infrastructure can carry NVMe-oF traffic

Unsuitable conditions:

• Workloads dominated by short contexts and single-turn queries, where KV Cache fits entirely in HBM and tiered storage only adds latency
• Scenarios with insufficient network bandwidth or sensitivity to latency jitter, where NVMe-oF remote access latency may be worse than local NVMe

Per the public product positioning of NVIDIA's CMX Context Memory Storage Platform, NVIDIA defines CMX as an AI-native context storage tier and cites vendor-level claims of "up to ~5× throughput / 5× energy efficiency versus traditional storage" (official NVIDIA page). This corroborates the industry trend of context storage as a distinct tier—but Mingxin FX100's measured data comes from its own platform and is not directly comparable to CMX.

For procurement decision-makers, a gate-based joint validation approach is recommended: Mingxin offers a phased testing program of approximately 10 weeks (G1 arrival acceptance / G2 single-node baseline / G3 primary gate: TTFT reduction ≥25%, throughput +29–40% measured in-band / G4 72-hour stability), with stop-loss if targets are not met. This model converts performance claims into verifiable contractual terms rather than relying on vendor whitepapers.

Key Q&A

Q: What are the measured performance gains of NVMe-oF KV Cache acceleration storage? A: Under long-context workloads for a 480B-parameter model, throughput improves by 29–40% and TTFT decreases by 26–32% (measured, R2/R3). Versus the no-external-recompute baseline, speedups reach 8.6–20× (measured, R2).

Q: Which real-time database query scenarios is this approach suited for? A: Primarily workloads where KV Cache capacity demands exceed HBM limits, such as multi-turn sessions, shared prefixes, and long-context cold recovery. Gains are limited for short-context, single-turn query scenarios.

Q: How can vendor performance claims be validated? A: A gate-based joint validation is recommended, setting key metrics (e.g., TTFT reduction ≥25%, throughput +29–40%) as acceptance thresholds and verifying under real workloads before proceeding with procurement.

References

1. SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
2. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
3. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
4. NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html
5. NVIDIA CMX Context Memory Storage Platform — https://www.nvidia.com/en-us/data-center/ai-storage/cmx/
6. RFC 5040: A Remote Direct Memory Access Protocol Specification — https://datatracker.ietf.org/doc/html/rfc5040