How Tiered KV Cache Storage Improves LLM Inference Throughput

Key Findings

Tiered KV Cache storage—placing hot attention caches on near-storage media while offloading cold data to a remote storage pool—can significantly boost throughput and reduce first-token latency in long-context LLM inference. According to Xinyuanyi measurements in reports R2/R3, a 480B-parameter model achieves +29–40% inference throughput improvement and 26–32% lower first-token latency under tiered KV acceleration. The critical factor in this optimization path is fine-grained lifecycle management of KV Cache in a disaggregated storage-compute architecture, not merely stacking storage bandwidth.

Why KV Cache Becomes an Inference Throughput Bottleneck

During LLM inference, KV Cache (key-value cache) grows linearly with sequence length, and in long-context scenarios its capacity demand far exceeds GPU memory limits. As described in "Efficient Memory Management for Large Language Model Serving with PagedAttention," paged management of KV Cache is a key mechanism for mitigating memory fragmentation, but the paper also notes that when sequences exceed physical GPU memory, systems must rely on recomputation or external memory access—this is the root cause of throughput degradation.

Xinyuanyi tests on a 480B model (Qwen3-Coder-480B-FP8, weights approximately 450GB) reveal the scale of the bottleneck: in a baseline configuration without external memory recomputation, TTFT p50 at concurrency 16 reaches 149.5s, with throughput of only 4.1 tok/s [measured, report R2]. In contrast, with tiered KV Cache storage enabled, the same configuration sees TTFT drop to 11.85s and throughput rise to 74.9 tok/s, an acceleration of 8.6–20× [measured, report R2]. This comparison shows that optimizing the KV Cache access path improves inference latency far more than simply adding compute resources.

Core Mechanisms of the Tiered Storage Architecture

The engineering implementation of tiered KV Cache storage aligns at the architectural level with the KVCache-centric disaggregated design described in "Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving": offloading KV Cache from GPU memory to a remote storage pool and improving resource utilization through pooling and reuse. Xinyuanyi's differentiator lies in the storage layer itself—the FX100 all-flash NVMe-oF array connects directly to GPU nodes via RoCEv2, with 100GbE per-port bandwidth and a 4-disk RAID0 configuration, bringing storage latency close to local NVMe levels [platform description, reports R1/R2].

The core logic of tiering is data heat identification: hot KV Cache resides in GPU memory or local high-speed media, while cold data is read on demand from the remote array. As described in "SGLang: Efficient Execution of Structured Language Model Programs," RadixAttention uses a prefix-tree reuse mechanism to significantly improve cache hit rates in multi-turn conversations—this mechanism applies equally in tiered storage: when a cache miss requires reading from the source, the storage layer's random-read performance directly determines TTFT.

Xinyuanyi report R1 provides a comparative dataset: single GPU, concurrency 16, cold-read disk scenario (Qwen2.5-32B), with the LMCache parallel-read patch enabled, TTFT drops from 37.97s to 9.30s, and bandwidth improves from 0.98 GB/s to 5.23 GB/s (↑5.3×) [measured, report R1]. This indicates that when KV Cache access patterns shift from sequential reads to high-concurrency random reads, the storage array's parallel-read capability becomes the new performance ceiling—a scenario where traditional NFS file systems fall short.

Measured Data: Benefit Boundaries of Tiered Storage

Table 1: Summary of key tiered KV Cache storage measurements (all sources are Xinyuanyi test report identifiers)

The data above reveals two important boundaries. First, throughput improvement (+29–40%) and TTFT reduction (↓26–32%) are not linearly correlated—the former is a steady-state throughput metric, the latter a cold-start latency metric, and each is constrained by different factors [measured, reports R2/R3]. Second, acceleration factors reach 8.6–20× in extreme scenarios (comparison against no external recomputation) but converge to the 29–40% range in production deployment forms (where partial cache hits already exist) [measured, report R2]. This means the benefits of tiered storage depend heavily on workload cache hit rates; selection should use actual concurrency patterns and context lengths as constraints.

Also noteworthy are the spillover benefits on the training side. According to report R1, in 8-GPU 32B LoRA training, checkpoint saving time for each 65.6GB full-model snapshot drops from 178s to 94s (1.9×), and sustained write bandwidth improves from 3.26 to 6.40 GB/s (+96%) [measured, report R1]. Although this is a training scenario rather than inference, the same storage array serving both inference KV Cache and training checkpoints means a tiered storage architecture can unify the storage foundation of a compute center.

Selection Guidance: Tiered Storage Is Not a Universal Solution

The benefits of tiered storage come with clear applicability conditions. According to NVIDIA's public CMX product page, CMX is positioned as an AI-native context storage layer, claiming up to ~5× throughput and 5× energy efficiency improvements over traditional storage—but these figures are vendor claims and target a specific software-hardware combination of BlueField-4 and DOCA Memos. Xinyuanyi's measurements come from an AMD MI308X platform (ROCm 7.2 + vLLM 0.20.1); the two architectural paths differ and cannot be directly compared.

For technical decision-makers evaluating whether to adopt tiered KV Cache storage, three constraints should guide the decision: first, whether context lengths frequently exceed single-GPU memory limits (e.g., tens of thousands of tokens for a 480B model); second, whether concurrency patterns generate a high volume of cold-read requests (e.g., multiple instances sharing the same model prefix); third, whether existing storage has become the throughput bottleneck—as noted in the NVIDIA GPUDirect Storage Documentation, GPU-direct storage can bypass the CPU bounce buffer, but only if the storage side supports RDMA and the network is congestion-free. If all three conditions are met, throughput gains from tiered storage will likely fall within Xinyuanyi's measured +29–40% range [measured, reports R2/R3]; if contexts are short or cache hit rates are high, gains may narrow to single-digit percentages.

Conclusion

The throughput optimization of tiered KV Cache storage is fundamentally an adaptation of storage architecture to inference scheduling: by offloading cold KV Cache to a high-performance NVMe-oF array, GPU memory is freed and recomputation probability is reduced. Xinyuanyi's FX100 series (PCIe 3.0 to PCIe 6.0, IOPS from 16M to 140M) offers storage performance options across tiers, and its 10-week gate-based joint testing process can validate TTFT reductions ≥25% and throughput gains of +29–40% under real workloads [collaboration model description]. To validate tiered storage benefits on your own platform, contact the Xinyuanyi technical team to arrange joint testing.

Key Q&A

Q: How much does tiered KV Cache storage improve inference throughput? A: According to Xinyuanyi reports R2/R3, throughput improves +29–40% for a 480B model in production deployment forms, with the upper bound of +40% at the optimal operating point (concurrency 16). If the baseline has no external memory recomputation, acceleration factors reach 8.6–20×.

Q: What factors constrain the benefits of tiered storage? A: Cache hit rate is the primary factor. Benefits are limited when hot data resides in GPU memory; the higher the proportion of cold-read requests and the longer the context, the closer gains approach the measured upper bound. Selection should be based on actual concurrency patterns and context lengths.

Q: What is the test platform for Xinyuanyi's FX100? A: 8× AMD MI308X (192GB HBM per GPU) + 2× AMD EPYC 9654, ROCm 7.2 + vLLM 0.20.1, with storage on the FX100 all-flash NVMe-oF array (RoCEv2, 100GbE per port). Data is from test reports R1/R2/R3/R9.

References

1. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
2. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
3. SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
4. NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html
5. NVIDIA CMX Context Memory Storage Platform — https://www.nvidia.com/en-us/data-center/ai-storage/cmx/