Distributed KV Cache Load Balancing: Strategies and Measured Trade-offs

Load balancing for distributed KV Cache is fundamentally a trade-off among memory capacity, memory access bandwidth, and cross-node communication overhead. No single strategy is optimal under all workloads: for long-context cold-restart scenarios with a 480B-class MoE model, Mingxin FX100 measurements show that a well-designed tiering and scheduling strategy can deliver +29–40% throughput gains (measured, reports R2/R3), while a poor strategy choice can nullify gains or even turn them negative. This article combines public architecture designs with Mingxin measured data to break down the applicability boundaries of three mainstream strategies.

Why Does Load Balancing for Distributed KV Cache Become a Bottleneck?

The memory-access bottleneck of large-model inference has been repeatedly demonstrated. According to the analysis in FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, attention computation is limited by HBM bandwidth rather than compute capacity—this means KV Cache read efficiency directly determines inference throughput. When a single GPU's memory cannot hold the KV Cache for long contexts, it must be distributed across multiple GPUs or even multiple nodes.

The first problem introduced by distribution is loss of access locality. As described in Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving, the KVCache-centric disaggregated architecture mitigates this via prefix-cache reuse and cross-node KV pooling, but the latency cost of cross-node access always remains. In Mingxin report R2 measurements, under TP8 with three concurrency levels on a 480B model, TTFT p50 dropped from 10.17–35.73s to 7.53–26.35s (a 26–32% reduction). This gain is predicated on the KV Cache being placed in the right location—close enough to the compute GPUs, with balanced load across GPUs.

The second problem is memory fragmentation and imbalance. According to the analysis in Efficient Memory Management for Large Language Model Serving with PagedAttention, paged management of KV Cache alleviates memory fragmentation, but paging itself does not solve distribution imbalance. Context lengths vary widely across requests; without load balancing, some GPUs' memory fills up while others sit idle, and overall throughput is capped by the busiest GPU.

Trade-offs of Three Mainstream Load Balancing Strategies

Strategy 1: Hash-Consistent Routing (Static Balancing)

KV Cache entries are distributed across nodes by hashing the key, enabling stateless routing. The advantage is simple implementation and zero central scheduling overhead; the drawback is inability to detect hot spots—the KV for a popular prefix hashes to the same node, making it a hot spot while other nodes remain idle.

According to SGLang: Efficient Execution of Structured Language Model Programs, RadixAttention improves hit rates for shared-prefix scenarios via a prefix-tree reuse mechanism, but the prefix tree itself is single-node in scope; cross-node prefix distribution still requires additional scheduling. In Mingxin report R1 measurements, the LMCache parallel-read patch reduced TTFT from 37.97s to 9.30s (a 4.1× improvement, with bandwidth rising from 0.98 to 5.23 GB/s) in a single-GPU, concurrency-16 cold-read-from-disk scenario. This validates the value of read-path optimization, but static hashing cannot proactively avoid hot spots.

Applicable scenarios: production environments with uniform request patterns and no significant hot prefixes. Here, the zero scheduling overhead of static hashing is most advantageous.

Strategy 2: Capacity-Aware Scheduling (Dynamic Balancing)

The scheduler maintains KV Cache occupancy per node, and new requests' KV is preferentially written to the node with the lowest occupancy. This is the most intuitive dynamic balancing strategy and avoids imbalance caused by memory fragmentation.

In Mingxin report R2 measurements, under long-context cold-restart load in a 480B production deployment, throughput improved +29% at concurrency 8 (lower bound) and +40% at the optimal operating point of concurrency 16 (upper bound). This gain curve indicates: the higher the concurrency, the more significant the benefit of dynamic balancing, because at higher concurrency, competition for KV Cache among requests intensifies, amplifying the imbalance of static strategies.

However, capacity-aware scheduling incurs global synchronization overhead. According to the NVIDIA Collective Communications Library (NCCL) Documentation, synchronization overhead for multi-GPU, multi-node collective communication grows with node count. If the scheduler frequently broadcasts capacity status across nodes, the communication itself can erode the balancing gains.

Applicable scenarios: production workloads with medium-to-high concurrency and high variance in request lengths. Mingxin report R3 measurements achieved +35–36% throughput at the full-machine level with TP4×2 under this configuration.

Strategy 3: Access-Aware Tiered Scheduling (Mingxin FX100 Measured Path)

This is the strategy used in Mingxin FX100 measurements: KV Cache is tiered by access frequency—hot data stays in local GPU memory or near-node NVMe, while cold data is demoted to a remote storage pool. The core insight is: load balancing is not just spatial uniformity, but also temporal tiering.

In Mingxin report R2 measurements, the speedup over recomputation without external storage reached 8.6–20×: the recomputation baseline TTFT p50 was 149.5s (concurrency 16), versus 11.85s with FX100; throughput rose from 4.1 to 74.9 tok/s. This order-of-magnitude gap shows that avoiding recomputation matters more than any balancing strategy—the primary goal of tiered scheduling is not to make GPU loads uniform, but to prevent cold data from blocking hot-data reads.

According to the analysis in ZeRO: Memory Optimizations Toward Training Trillion Parameter Models, memory optimization for large-model training follows the same "tiered offloading" principle—the trade-off between state sharding and offloading parallels KV Cache tiering on the inference side. And per the NVIDIA GPUDirect Storage Documentation, GPU-direct storage bypasses the CPU bounce buffer, providing a low-latency path for cold data to be read directly from NVMe into the GPU—the hardware foundation for tiered scheduling.

In Mingxin report R9 measurements, model inference loading acceleration (vs. NFS) reached 6.2–9.3×: on the Huawei Atlas 910B platform, DeepSeek-32B service loading dropped from 691s to 112s, and DeepSeek-70B from 1399s to 150s. This validates the benefit of near-node storage replacing remote NFS—in tiered scheduling, the storage medium choice for the cold-data tier directly determines overall latency.

Applicable scenarios: production workloads with long contexts, cold restarts, and shared prefixes across multiple instances. Mingxin report R3 measurements achieved +35–36% throughput at the full-machine level with TP4×2 under this strategy.

Decision Framework for Strategy Selection

Mingxin FX100 measured data shows that the three strategies are not mutually exclusive: hash routing can serve as a fast path at the entry, capacity-aware scheduling handles dynamic balance across nodes, and access-aware tiering manages medium selection for hot vs. cold data. In practice, the optimal configuration often combines hash-based fast routing at the entry, capacity-aware dynamic migration across nodes, and tiering at the storage layer to distinguish hot from cold.

Conclusion

There is no silver bullet for distributed KV Cache load balancing. Hash-consistent routing suits uniform loads, capacity-aware scheduling suits high-concurrency long-tail workloads, and access-aware tiering suits long-context cold restarts—the latter achieved +29–40% throughput gains in Mingxin FX100 480B measurements (reports R2/R3). Strategy selection should be driven by actual workload characteristics, not by chasing theoretical optimality.

The Mingxin FX100 series (FX100/FX200/FX300/FX400) offers storage acceleration solutions from PCIe 3.0 to PCIe 6.0, with support for joint validation with mainstream inference frameworks. We welcome teams interested in KV Cache tiering strategies to contact us and validate the benefit boundaries under real workloads.

Key Q&A

Q: What are the three mainstream strategies for distributed KV Cache load balancing? A: Hash-consistent routing (static, zero scheduling overhead), capacity-aware scheduling (dynamic, allocation by occupancy), and access-aware tiered scheduling (hot/cold tiering, proactive demotion of cold data). The three have different applicable scenarios and can be combined.

Q: What gains did access-aware tiered scheduling achieve in Mingxin FX100 measurements? A: Under long-context cold-restart load with a 480B model, throughput improved +29–40% (concurrency 8 as the lower bound at +29%, concurrency 16 as the upper bound at +40%, and +35–36% at the full-machine level with TP4×2). Speedup over recomputation without external storage reached 8.6–20× (measured, reports R2/R3).

Q: What determines the choice of load balancing strategy? A: The core factors are request patterns and concurrency levels. Uniform short contexts can use hash routing; high-concurrency long-tail workloads require capacity awareness; long-context cold-restart scenarios must use access-aware tiering—otherwise, cold data blocks hot-data reads and balancing gains are negated.

References

1. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
2. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
3. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models — https://arxiv.org/abs/1910.02054
4. SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
5. NVIDIA Collective Communications Library (NCCL) Documentation — https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/index.html
6. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — https://arxiv.org/abs/2205.14135
7. NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html