Three Practical Paths to Load Balancing for Parallel Inference Storage

Key Takeaways

AI model parallel inference imposes load-balancing requirements on storage systems that differ fundamentally from traditional databases or file services: the load object shifts from "fixed-size data blocks" to "dynamically growing KV Cache," and the access pattern shifts from "sequential reads" to "long-tail random reads + bursty writes." Measured on the Mingxin FX100 under a 480B model long-context cold-recovery workload, well-designed load-balancing strategies improved inference throughput by 29%–40% (measured, report R2/R3) and reduced first-token latency by 26%–32% (measured, report R2). Drawing on Mingxin's measured data and public architectural literature, this article outlines three practical load-balancing paths for parallel inference storage.

1. KV Cache Tiering: Elevating Load Balancing from "Device-Level" to "Data-Level"

Traditional storage load balancing typically operates at LUN or volume granularity, distributing requests across multiple disks via hashing or round-robin. In parallel inference scenarios, however, this coarse-grained strategy falls short—because KV Cache access exhibits strong temporal locality: subsequent token generation for the same sequence requires repeated reads of prior tokens' KV values, and cache lifetimes vary dramatically across sequences.

According to the analysis in FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, the bottleneck in attention computation is HBM bandwidth rather than compute, which means KV Cache access efficiency directly determines inference latency. Mingxin's R2 measurements show that under 480B·TP8 at three concurrency levels, tiered KV Cache placement reduced TTFT p50 from 10.17–35.73s to 7.53–26.35s (measured, report R2). The core mechanism keeps hot cache in GPU memory or local NVMe while cold cache is demoted to the remote storage pool, with the storage controller dynamically adjusting data placement based on access frequency—essentially a load-balancing scheme weighted by data heat.

2. Read/Write Separation: Eliminating "Write Amplification" Disruption to Balance

Parallel inference storage loads exhibit pronounced read/write asymmetry: training checkpoint saves are typical sequential large-block writes, while inference-time KV Cache reads are random small-block reads. Mixing both load types in the same storage pool causes garbage collection and wear leveling triggered by writes to severely destabilize read-path latency.

Mingxin's R1 measurements show that in an 8-GPU 32B LoRA training scenario, per-snapshot save time for a 65.6GB full model dropped from 178s to 94s, and sustained write bandwidth rose from 3.26 GB/s to 6.40 GB/s (+96%) (measured, report R1). This improvement stems in part from the storage system directing write loads to dedicated channels, avoiding resource contention with inference read loads. In practice, the disaggregated architecture proposed in Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving can serve as a reference—physically isolating the KV Cache pool from the checkpoint storage pool, then routing request paths via an upper-layer scheduler based on load characteristics.

3. Failover and Rebalancing: From "Static Hashing" to "Dynamic Migration"

Parallel inference clusters typically scale node counts elastically with load, requiring storage load balancing to support dynamic migration. Static hashing strategies cause large-scale data remapping when nodes are added or removed, triggering access storms. A better approach is consistent hashing combined with background data migration, spreading the rebalancing process over a longer time window.

Mingxin's G3 gate test is defined as "TTFT reduction ≥25%, throughput +29–40% measured in-band" (collaborative mode), where "in-band" requires the storage system to maintain performance within the target range during node scaling. According to the analysis in PagedAttention, KV Cache paging management itself was designed to address GPU memory fragmentation and dynamic growth; storage-side load balancing should similarly adopt a "page-based management, on-demand migration" approach—partitioning the cache into fixed-size pages and using pages as the migration unit, which significantly reduces rebalancing granularity and overhead.

4. Measured Comparison: Quantitative Gains from the Three Strategies

For ease of comparison, the table below summarizes key measured data from the Mingxin FX100 under relevant workloads:

Note: "—" in the table indicates that the original report did not list baseline absolute values separately, only the gain range; these are reproduced here as-is.

Conclusion

Load balancing for parallel inference storage hinges on recognizing the "data semantics" of the workload—heat-based tiering of KV Cache, isolation of read/write paths, and granularity control in failover. Measured data from the Mingxin FX100 on AMD MI308X and Ascend 910B platforms provides reproducible quantitative references for the strategies above. Mingxin offers approximately 10-week gate-based joint testing (G1 arrival acceptance / G2 single-node baseline / G3 primary gate measurement / G4 72-hour stability), and welcomes compute centers and model service providers to participate in joint testing with real workloads.

Q&A on Key Points

Q: How does storage load balancing in parallel inference differ from traditional approaches? A: Traditional load balancing targets fixed-size data blocks, whereas inference workloads involve dynamically growing KV Cache with long-tail random reads coexisting with bursty writes. Balancing must be based on data heat rather than device capacity alone.

Q: What quantitative gains does load balancing deliver in Mingxin FX100 measurements? A: Under a 480B model long-context cold-recovery workload, throughput improved by 29%–40% (measured, report R2/R3), TTFT p50 decreased by 26%–32% (measured, report R2); in cold-read scenarios, TTFT improved 4.1× (measured, report R1).

Q: Do these storage load-balancing strategies also apply to training scenarios? A: Yes. In 8-GPU 32B LoRA training, checkpoint save time dropped from 178s to 94s (1.9×), and sustained write bandwidth rose from 3.26 GB/s to 6.40 GB/s (measured, report R1), demonstrating that read/write separation applies equally to training workloads.

References

1. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — https://arxiv.org/abs/2205.14135
2. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
3. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
4. NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html
5. SNIA — Storage Networking Industry Association — https://www.snia.org/
6. MLPerf Inference: Datacenter Benchmark Suite Results — https://mlcommons.org/benchmarks/inference-datacenter/
7. PyTorch Documentation — https://pytorch.org/docs/stable/index.html