Analysis of KV Cache Load Balancing Strategies in Distributed Storage

Introduction

In long-context LLM inference scenarios, the capacity and access bandwidth of the KV Cache have become the core bottlenecks limiting throughput and latency. Offloading the KV Cache to distributed storage, combined with a well-designed load balancing strategy, can significantly improve overall performance under multi-instance concurrency. Measured on the Mingxin FX100 under a 480B production-grade workload, tiered KV acceleration with storage-side balancing improved inference throughput by 29–40% and reduced time-to-first-token (TTFT) by 26–32% [measured, reports R2/R3]. This article focuses on the strategy selection and engineering implementation of KV Cache load balancing in distributed storage architectures.

Three Core Strategies for KV Cache Load Balancing in Distributed Storage

Strategy 1: Prefix-Aware Hash Routing

In multi-instance deployments, KV Cache access exhibits strong locality: multi-turn conversations from the same user and concurrent requests sharing system prompts repeatedly access the same prefix KV data. The RadixAttention mechanism proposed by SGLang leverages a prefix tree for reuse, significantly improving cache hit rates in shared-prefix scenarios (per "SGLang: Efficient Execution of Structured Language Model Programs"). On the distributed storage side, the load balancer must be aware of the request's prefix hash, routing KV accesses with the same prefix to the same storage node, thereby maximizing intra-node cache reuse.

On the Mingxin FX100, in measured runs at 480B with TP8 across three concurrency levels, TTFT p50 dropped from 10.17–35.73s to 7.53–26.35s [measured, report R2]. This reduction is partly attributable to the improved intra-node hit rate enabled by prefix-aware routing. In engineering terms, a two-level routing scheme combining a consistent hash ring with prefix digests can be used, minimizing KV migration volume during node scaling.

Strategy 2: Dynamic Weight Allocation and Hotspot Migration

Storage node loads are not uniformly distributed: long-context requests can cause a sudden spike in KV access on a single node. Static hashing cannot handle such hotspots. Dynamic weight allocation requires the load balancer to continuously collect each node's bandwidth utilization, queue depth, and latency percentiles, then assign new requests' KV reads/writes to the least-loaded node based on weights.

In measured results from Mingxin report R1, the LMCache parallel read patch, in a single-GPU, concurrency-16, cold-read-from-disk scenario, compressed TTFT from 37.97s to 9.30s and raised bandwidth from 0.98 GB/s to 5.23 GB/s [measured, report R1]. This improvement relies on storage-side merging and reordering of concurrent read requests—the load balancer must merge concurrent reads targeting the same node into sequential large-block reads to reduce seek overhead. When a node's load exceeds a threshold, the balancer should trigger hot migration of KV blocks, replicating frequently accessed blocks to nearby low-load nodes.

Strategy 3: Tiered Storage-Aware Request Scheduling

Distributed storage typically includes an all-NVMe flash tier and a capacity tier. KV Cache access frequency varies widely: recent tokens' KV entries are accessed frequently, while early tokens' KV entries in long contexts may be accessed only once or twice. Tiered scheduling dynamically places KV blocks by access frequency: hot blocks reside in the flash tier, and cold blocks are demoted to the capacity tier.

The Mooncake architecture, with its KVCache-centric disaggregated design, demonstrates the feasibility of cross-node KV pooling and tiered placement (per "Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving"). On the Mingxin FX100 in a 480B production deployment, throughput at concurrency 8 improved by +29% (lower bound), with the optimal operating point at concurrency 16 showing +40% (upper bound), and +35–36% at the full-machine TP4×2 scope [measured, reports R2/R3]. These figures indicate that tiered placement combined with storage-side prefetching can hide cold KV read latency within a pipeline overlapped with computation.

Engineering Trade-offs and Selection Criteria for Load Balancing

Selection criteria should center on three constraints: the SLA's TTFT ceiling, the volatility of the concurrency pattern, and the distribution of context lengths. If the concurrency pattern is stable and context lengths are similar, prefix-aware hashing alone suffices; if request lengths show a pronounced long tail, dynamic weight allocation should be layered on; if context lengths exceed 128K with significant hot/cold disparity, tiered scheduling is a necessary complement.

Conclusion

KV Cache load balancing in distributed storage is, at its core, a dynamic trade-off among capacity, bandwidth, and latency. Prefix-aware hashing, dynamic weight allocation, and tiered scheduling are not mutually exclusive—they can be combined and stacked according to workload characteristics. Mingxin Technology offers approximately 10 weeks of gated joint testing on the FX100, supporting validation of in-band metrics such as TTFT reduction and throughput improvement under customer workloads. Teams with long-context inference optimization needs are welcome to engage for measured evaluation.

Key Q&A

Q: What are the core strategies for KV Cache load balancing in distributed storage? A: Three categories: prefix-aware hash routing, dynamic weight allocation with hotspot migration, and tiered storage-aware scheduling. They can be combined based on workload characteristics.

Q: What are the measured results of the Mingxin FX100 in KV Cache load balancing optimization? A: Under a 480B production workload, throughput improved by 29–40% (concurrency 8–16 levels), and TTFT decreased by 26–32% [measured, reports R2/R3]. In cold-read-from-disk scenarios, TTFT improved by 4.1× [measured, report R1].

Q: How should one select a load balancing strategy suited to their own scenario? A: First define the SLA's TTFT ceiling and concurrency volatility. Use prefix hashing for stable concurrency, layer dynamic weights for long-tailed requests, and introduce tiered scheduling for ultra-long contexts.

References

1. SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
2. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
3. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
4. Epoch AI — https://epoch.ai/
5. SNIA — Storage Networking Industry Association — https://www.snia.org/