NVIDIA Dynamo with Third-Party Storage: KV Cache Practice

NVIDIA Dynamo, as an infrastructure orchestration framework for AI factories, integrates KV Cache management with third-party storage solutions, which is becoming a critical path for optimizing large language model (LLM) inference performance. This article analyzes the practical considerations and performance gains of offloading KV Cache to third-party storage under the Dynamo framework, based on measured data from Mingxin's FX100 on a 480B model, providing a reference for technical decision-makers at compute centers.

Dynamo's KV Cache Management Mechanism and Storage Offload Requirements

NVIDIA Dynamo is an infrastructure orchestration framework for large-scale AI inference and training. One of its core designs is to treat KV Cache as a schedulable system resource rather than mere in-memory data. In long-context inference scenarios, the size of KV Cache grows linearly with sequence length. A single long-context session on a 480B model can have a KV Cache reaching tens of gigabytes, far exceeding the memory capacity of a single GPU. According to Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP '23), paged management of KV Cache can effectively mitigate memory fragmentation issues. However, when the working set exceeds memory limits, offloading cold data to external storage becomes a necessary choice.

Dynamo's disaggregated compute-storage architecture aligns with the KVCache-centric pooled design proposed in Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving (arXiv:2407.00079): decoupling KV Cache from GPU memory and interacting with the storage layer via high-speed networks, enabling independent scaling of inference instances and cache resources. Under this architecture, the performance of third-party storage directly determines the access latency and throughput after KV Cache offload, making it a key variable in system bottlenecks.

Measured Path and Performance of Third-Party Storage with Dynamo

Mingxin completed compatibility testing of the FX100 all-flash NVMe-oF array with the Dynamo framework on an 8×AMD MI308X platform (192GB HBM per GPU, ROCm 7.2, vLLM 0.20.1+rocm721), using the Qwen3-Coder-480B-FP8 model (MoE, approximately 450GB weights). The tests covered multiple concurrency configurations for KV-tiered accelerated inference, with core results as follows:

Table 1: Measured performance of FX100 under 480B·TP8 long-context cold-recovery load (Source: Measured, R2/R3)

The tests used RoCEv2 networking with a single 100GbE port, and a 4-drive RAID0 configuration (14TB, XFS filesystem). The data shows that under the Dynamo framework, tiered offload of KV Cache to the FX100 array improves throughput by 29–40% and reduces time-to-first-token (TTFT) by 26–32%. Concurrency 16 is the optimal operating point, with throughput improvement of 35–36% at the full-machine level with TP4×2 (measured, R3).

Notably, in the no-external-storage recompute scenario comparison: baseline TTFT p50 was 149.5s (concurrency 16), while the FX100 solution reduced it to 11.85s, a 12.6× speedup; throughput increased from 4.1 tok/s to 74.9 tok/s, a 18.3× improvement (measured, R2). This data indicates that when KV Cache fully resides on high-speed third-party storage, the inference system can avoid the external storage recompute path triggered by cache misses, achieving order-of-magnitude performance gains.

Key Criteria for Storage Selection and Architecture Adaptation

Based on the above measurements, storage selection for third-party storage under the Dynamo framework should focus on three dimensions:

Bandwidth and Latency Matching. The KV Cache offload path is highly sensitive to storage latency. In Mingxin's LMCache parallel read patch test (single GPU, concurrency 16, cold read from disk, Qwen2.5-32B), TTFT dropped from 37.97s to 9.30s (4.1× improvement), and bandwidth increased from 0.98 GB/s to 5.23 GB/s (↑5.3×) (measured, R1). This shows that linear increases in storage read bandwidth translate directly into near-linear TTFT improvements. Selection should prioritize measured bandwidth over nominal IOPS as the primary criterion.

Coordination with Dynamo Scheduler. Dynamo's KV Cache scheduler needs to be aware of storage-layer performance characteristics (e.g., partition latency, concurrent read capability) to make optimal offload decisions. According to NVIDIA GPUDirect Storage documentation, the GPU-direct storage data path can bypass the CPU bounce buffer, reducing data copy overhead. In the Dynamo architecture, this mechanism can further reduce CPU overhead on the KV Cache offload path, but it requires confirming whether the third-party storage supports direct protocols such as RDMA or GPUDirect.

Capacity and Cost Balance. The FX100 full-configuration reference price is approximately ¥371,200 (about ¥2,014/TB), FX200 approximately ¥331,200 (about ¥1,797/TB), and FX300 approximately ¥924,000 (about ¥5,014/TB) (vendor quotation basis). For production deployment, trade-offs must be made among capacity, bandwidth, and unit cost based on KV Cache working set size, concurrency, and SLA requirements. For the 480B model long-context scenario, if concurrency 16 is the typical load, FX100's bandwidth performance already meets the measured throughput ceiling of +40%; for higher concurrency, evaluation is needed to determine whether FX300's PCIe 5.0×400Gb interface can deliver additional benefits.

Practical Recommendations and Boundary Conditions

When introducing third-party storage under the Dynamo framework, the following steps are recommended:

1. Test Before Selecting: Use representative workloads to measure TTFT and throughput on the target model and concurrency configuration, rather than relying on nominal specifications. Mingxin offers approximately 10-week gate-based joint testing (G1 arrival acceptance / G2 single-node baseline / G3 main gate: TTFT reduction ≥25%, throughput +29–40% measured in-band / G4 72-hour stability), allowing performance targets to be locked before procurement.

2. Focus on Cold Start and Long Tail: The benefits of KV-tiered acceleration are most pronounced under cold-recovery loads. Test data shows concurrency 8 is the lower bound (+29%) and concurrency 16 is the upper bound (+40%); actual gains depend on the cache hit rate distribution of the workload. For scenarios such as multi-turn dialogue and shared prefixes, reference can be made to the RadixAttention prefix-tree reuse mechanism in SGLang: Efficient Execution of Structured Language Model Programs (arXiv:2312.07104), using Dynamo's scheduler to optimize prefix reuse and improve hit rates.

3. Define Applicability Boundaries: This test was based on the AMD MI308X platform with ROCm 7.2. If the target platform is NVIDIA GPUs or Ascend, re-validation is required. Mingxin's testing on the Huawei Atlas 910B platform showed model inference loading acceleration (vs NFS) of 6.2–9.3× (DeepSeek-32B 691s→112s, DeepSeek-70B 1399s→150s, measured, R9), but KV Cache offload performance migration across platforms requires additional validation.

Conclusion

NVIDIA Dynamo's KV Cache management capabilities provide a clear integration point for third-party storage, and storage-layer performance directly determines post-offload inference quality. Measured data from Mingxin's FX100 on the 480B model shows that through reasonable tiered offload, throughput can improve by 29–40% and TTFT can decrease by 26–32%. Technical decision-makers are advised to base selection on measured data, combined with their own workload characteristics and SLA constraints, and validate the synergy between storage solutions and the Dynamo framework in a joint test environment. Mingxin offers a gate-based joint testing mechanism, enabling evaluation of performance and cost boundaries with reproducible Python-based measurement models before procurement.

Key Q&A

Q: Under the NVIDIA Dynamo framework, what is the primary value of third-party storage for KV Cache management? A: Dynamo treats KV Cache as a schedulable resource; when the working set exceeds memory, offload to external storage is required. The bandwidth and latency of third-party storage directly determine post-offload TTFT and throughput. Measured data shows high-speed arrays can deliver 29–40% throughput improvement and 26–32% TTFT reduction (measured, R2/R3).

Q: What is the most critical performance metric when selecting third-party storage for Dynamo? A: Read bandwidth, not nominal IOPS. Mingxin's measurements show that when bandwidth increased from 0.98 GB/s to 5.23 GB/s, TTFT improved by 4.1× (measured, R1); linear bandwidth increases translate directly into near-linear TTFT improvements.

Q: What is the measured performance of Mingxin's FX100 on the 480B model? A: On the 8×AMD MI308X platform, the FX100 all-flash array compared to local NVMe single drive delivers 29–40% throughput improvement (concurrency 8 is the lower bound at +29%, concurrency 16 is the upper bound at +40%), and TTFT p50 drops from 10.17–35.73s to 7.53–26.35s (measured, R2/R3).

References

1. SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
2. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
3. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
4. SNIA — Storage Networking Industry Association — https://www.snia.org/
5. NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html