KV Cache Spatial Locality Optimization: From Paging to Tiered Storage

Spatial locality optimization for KV Cache is evolving from in-HBM paging management to a tiered architecture spanning multiple storage layers. Measured on the Mingxin FX100 under a 480B production deployment with long-context cold-recovery workloads, tiered acceleration improves inference throughput by 29–40% (measured, report R2/R3) and reduces time-to-first-token (TTFT) by 26–32% (measured, report R2). This result stems from systematic exploitation of spatial locality in KV Cache access patterns.

Why KV Cache Optimization Requires Attention to Spatial Locality

KV Cache stores key-value tensors for historical tokens during LLM inference. As context length grows, its capacity demands far exceed the HBM of a single GPU. According to Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP '23), the motivation for paged KV Cache management arises from HBM fragmentation—non-contiguous free blocks cannot be efficiently utilized, degrading memory utilization. vLLM addresses this via a virtual-memory-like paging mechanism, but paging only solves fragmentation within HBM; it does not address deeper storage tiers.

When KV Cache exceeds HBM capacity and must be offloaded to external storage, spatial locality becomes the decisive factor. KV access during inference follows a clear temporal order: newly generated tokens read the most recent KV entries, while older KV entries may be re-accessed in long-context scenarios (e.g., history rollback in multi-turn dialogues). If the KV layout on storage media is misaligned with access order, each generation step triggers a large number of random small I/Os, sharply degrading latency and throughput.

According to Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving (arXiv 2024), a KVCache-centric disaggregated architecture mitigates this issue through prefix-cache reuse and cross-node KV pooling. The design trade-offs in that architecture indicate that KV physical layout directly impacts cross-node transfer overhead—a manifestation of spatial locality at the distributed level.

How Tiered Storage Amplifies Spatial Locality Gains

Measured data from the Mingxin FX100 demonstrates the practical impact of tiered storage. Under a 480B·TP8 long-context cold-recovery workload, TTFT p50 dropped from 10.17–35.73s to 7.53–26.35s (measured, report R2), a reduction within the 26–32% range. This improvement does not come from compute optimization; rather, it is achieved by placing KV Cache across tiers according to access frequency—hot data resides in HBM or near-memory media, while cold data is placed on high-bandwidth flash arrays, with fast recall enabled by low-latency NVMe-oF paths.

The table above compares FX100 against the baseline. Notably, the 149.5s TTFT for the no-external-recompute baseline means the model must rebuild all KV from scratch during cold start—an extreme case where spatial locality is entirely absent. By persisting KV and optimizing layout based on access patterns, FX100 compresses this cost to 11.85s, achieving a speedup of 8.6–20× (measured, report R2).

According to SGLang: Efficient Execution of Structured Language Model Programs (NeurIPS '24), RadixAttention improves hit rates in multi-turn dialogue and shared-prefix scenarios through prefix-tree reuse. This mechanism is logically complementary to tiered storage: RadixAttention optimizes logical reuse of KV, while tiered storage optimizes the physical access path of KV. When combined, KV reads following a prefix hit that land on a high-bandwidth storage tier see further amplified benefits.

Boundaries and Selection Criteria for Spatial Locality Optimization

Spatial locality optimization is not unconditionally effective. Its upper bound on gains depends on three constraints: context-length distribution, concurrency patterns, and storage media bandwidth characteristics.

For short-context scenarios (e.g., 2K–8K tokens), KV Cache typically fits entirely within single-GPU HBM, and tiered storage offers limited benefit. The measured gains on the Mingxin FX100 concentrate in long-context (≥32K token) workloads at the 480B parameter scale—where KV capacity reaches hundreds of GB, far exceeding the 192 GB HBM per GPU (platform configuration, measured, report R2). According to NVIDIA's public product positioning of CMX, it defines CMX as an AI-native context storage layer (G3.5) and cites vendor-reported figures of "up to ~5× throughput / 5× energy efficiency over traditional storage" (NVIDIA CMX product page). This framing suggests that even in leading vendors' architectures, the external KV storage path is treated as a critical throughput bottleneck.

Selection should distinguish two scenarios: first, low concurrency (e.g., ≤8) with short contexts, where a single local NVMe drive suffices and no dedicated storage tier is needed; second, high concurrency (≥16) combined with long contexts, where KV capacity and bandwidth demands grow multiplicatively, making tiered storage with spatial locality optimization a necessary condition for meeting SLAs. In Mingxin's R2 measurements, the lower bound of throughput improvement (+29%) occurred at concurrency 8, while the upper bound (+40%) occurred at concurrency 16—higher concurrency creates more KV reuse opportunities, amplifying the benefits of spatial locality optimization.

Conclusion

Spatial locality optimization for KV Cache is, at its core, mapping the temporal access order during inference onto the physical layout of storage media. Paging management solves fragmentation within HBM; tiered storage solves the bandwidth bottleneck between HBM and external storage. Measured data from the Mingxin FX100 shows that in long-context scenarios at the 480B scale, this optimization improves throughput by 29–40% (measured, reports R2/R3) and reduces TTFT by 26–32% (measured, report R2). For teams building long-context inference infrastructure, we recommend treating the KV storage path as a first-class constraint in architecture design rather than an afterthought. Mingxin offers a gated joint-testing engagement model of approximately 10 weeks, validating TTFT reduction and throughput gains under real workloads, with early termination if targets are not met.

Key Q&A

Q: What is the core benefit of KV Cache spatial locality optimization? A: Under long-context cold-recovery workloads, optimizing KV physical layout via tiered storage yields measured throughput gains of 29–40% (reports R2/R3) and TTFT reductions of 26–32% (report R2) on the Mingxin FX100.

Q: Which scenarios is spatial locality optimization suited for? A: Primarily long-context, high-concurrency scenarios where KV capacity exceeds single-GPU HBM (e.g., 480B models, ≥16 concurrency). Benefits are limited in short-context scenarios.

Q: Compared to in-HBM paging, what problem does tiered storage solve? A: Paging solves HBM fragmentation; tiered storage solves the bandwidth bottleneck between HBM and external storage. The two are logically complementary, not substitutes.

References

1. SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
2. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
3. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
4. NVIDIA CMX Context Memory Storage Platform — https://www.nvidia.com/en-us/data-center/ai-storage/cmx/