Implementation Paths for External KV Cache Storage in LLM Inference

In long-context inference scenarios, KV Cache memory consumption grows rapidly, making external storage a viable direction for balancing cost and performance. Current technical implementations mainly follow three paths: disaggregated storage-compute architectures that offload KV Cache to remote memory pools, paging and prefix reuse mechanisms that improve cache hit rates, and high-speed networking with direct-attached storage protocols that reduce external access latency. These three paths are not mutually exclusive—production deployments often combine them, and their effectiveness is strongly correlated with model scale, concurrency patterns, and hardware platforms.

Path 1: Remote KV Pooling under Disaggregated Storage-Compute Architecture

Disaggregated storage-compute is the mainstream architectural approach for external KV Cache today. According to Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving, this architecture centers on KV Cache, splitting prefill and decode phases across different compute nodes, and achieving elastic sharing of cache resources through cross-node KV pooling. The core motivation: the prefill phase is compute-intensive while the decode phase is memory-access-intensive—these two workload types have significantly different resource demands, and physical isolation allows each to scale independently based on its own bottleneck.

The key implementation point of this path lies in cache pool scheduling strategy. The remote KV pool must handle data return latency on cache misses, so it is typically paired with prefix cache reuse mechanisms that persist repeated system prompts, few-shot examples, and other common prefixes remotely, reducing redundant computation. Mingxin observed in production deployment at 480B scale that KV tiered acceleration delivers throughput gains of +29–40% for long-context cold-recovery workloads (concurrency 8 as lower bound at +29%, optimal operating point at concurrency 16 as upper bound at +40%, TP4×2 full-node basis at +35–36%), where cold recovery is precisely the scenario where remote pooling yields the greatest benefit [measured, reports R2/R3].

Path 2: Paging Management and Prefix Tree Reuse to Improve Hit Rates

External storage access latency is far higher than HBM, so improving cache hit rates is key to reducing external access frequency. According to Efficient Memory Management for Large Language Model Serving with PagedAttention, KV Cache paging management borrows the paging concept from operating system virtual memory, mapping contiguous logical KV blocks to non-contiguous physical memory pages, thereby eliminating memory fragmentation and improving batch throughput. According to SGLang: Efficient Execution of Structured Language Model Programs, the RadixAttention mechanism further organizes caches using a prefix tree structure, significantly improving cache reuse rates in multi-turn dialogue and shared-prefix scenarios.

These two works provide an important prerequisite for external storage solutions: only when cache management granularity is fine enough and reuse algorithms are efficient enough can the capacity advantage of external storage translate into actual hit benefits. Mingxin, in a single-GPU concurrency-16 cold disk-read scenario (Qwen2.5-32B), measured TTFT dropping from 37.97s to 9.30s with the LMCache parallel read patch—a 4.1× improvement—and bandwidth rising from 0.98 GB/s to 5.23 GB/s [measured, report R1]. This result demonstrates that, with cache management algorithms properly optimized, external storage read bandwidth can approach practical usability.

Path 3: High-Speed Networking and Direct-Attached Storage Protocols to Reduce Access Latency

The physical location of external storage sets the lower bound on access latency, making data path design the third critical path. According to NVIDIA GPUDirect Storage Documentation, GPU direct-attached storage technology allows data to bypass the CPU and host memory bounce buffer, establishing a direct data path between GPU and storage devices, thereby reducing copy counts and latency. This mechanism suits scenarios requiring frequent loading of large data blocks from storage into GPU memory, aligning closely with the access patterns of external KV Cache.

At the protocol level, NVMe-oF (NVMe over Fabrics) with RoCEv2 is the current mainstream combination for low-latency network storage. Mingxin's FX100 all-flash NVMe-oF array adopts RoCEv2 with single-port 100 GbE configuration. On the Huawei Atlas 910B platform, compared against an NFS baseline, measured model inference loading acceleration was 6.2–9.3× (DeepSeek-32B service loading 691s → 112s; DeepSeek-70B 1399s → 150s) [measured, report R9]. This result reflects the advantages of direct-attached storage protocols over traditional network file systems in both metadata overhead and data path efficiency.

Applicability Boundaries and Combination Strategies for the Three Paths

The benefit boundaries of the three paths must be assessed against specific workloads. Disaggregated storage-compute yields clear gains in high-concurrency scenarios, but under single-concurrency low load, remote pooling scheduling overhead may offset the benefits; paging and prefix reuse work well for dialogue scenarios with high shared-prefix ratios, but offer limited help for workloads like long-document generation where prefix reuse opportunities are scarce; direct-attached storage protocols significantly improve cold-start loading scenarios, but show diminishing marginal returns when KV Cache hit rates are already high. According to FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, attention computation itself is limited by HBM bandwidth rather than compute capacity, suggesting that the ultimate ceiling on external storage solution gains remains constrained by the physical bandwidth of data movement.

Mingxin's measurements in the external KV Cache direction cover combined forms of all three paths. The FX100 all-flash NVMe-oF array, tested on a 480B model with TP8 across three concurrency levels, showed TTFT p50 reduced from 10.17–35.73s to 7.53–26.35s (a 26–32% reduction) [measured, report R2]; acceleration over the no-external-storage recompute baseline reached 8.6–20× (recompute baseline TTFT p50 149.5s versus FX100's 11.85s; throughput 4.1 versus 74.9 tok/s) [measured, report R2]. These data indicate that combined deployment of the three paths delivers verifiable performance gains under long-context production workloads.

Key Q&A

Q: What are the three technical paths for external KV Cache storage? A: Remote KV pooling under disaggregated storage-compute architecture, paging management with prefix tree reuse to improve hit rates, and high-speed data paths such as GPU direct-attached storage and NVMe-oF. The three can be deployed in combination, with gains strongly correlated to workload patterns.

Q: What is the measured performance of external storage solutions? A: Mingxin FX100 on 480B model long-context cold-recovery workloads measured throughput gains of +29–40% and TTFT reductions of 26–32%; acceleration over the no-external-storage recompute baseline was 8.6–20× [measured, reports R2/R3].

Q: What should be prioritized when selecting an external storage solution? A: First clarify SLA constraints and concurrency patterns. High-concurrency long-context scenarios suit disaggregated storage-compute; dialogue scenarios with high shared-prefix ratios should prioritize cache reuse optimization; frequent cold starts require attention to data path efficiency in direct-attached storage protocols.

References

1. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
2. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
3. SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
4. NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html
5. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — https://arxiv.org/abs/2205.14135