GPU Clusters Busy Wait: Storage's Impact on Efficiency

In inference scenarios within GPU clusters, insufficient storage bandwidth is a core bottleneck causing "busy wait" (GPU idling while waiting for data). By leveraging tiered KV Cache acceleration and high-performance NVMe-oF storage, inference throughput can be improved by 29–40%, and first-token latency reduced by 26–32%. This "busy wait" phenomenon is particularly pronounced in long-context inference and multi-instance concurrency: when the GPU must load historical KV Cache or model weights from external storage, if the storage bandwidth is far lower than the GPU's compute throughput, the GPU frequently enters idle states, reducing compute efficiency. Measured results on the Mingxin FX100 all-flash array with the AMD MI308X platform show that reducing storage latency from seconds to milliseconds significantly alleviates this bottleneck, bringing GPU utilization closer to theoretical peaks.

How Insufficient Storage Bandwidth Causes GPU Compute Idling

When storage bandwidth cannot match the GPU's compute demands, the GPU enters a waiting state during each data load. This "busy wait" manifests in inference as decreased GPU utilization and increased end-to-end latency.

In the PagedAttention mechanism based on the vLLM paper (arXiv:2309.06180), on-demand loading of KV Cache is a critical step in inference. When model size exceeds GPU memory (e.g., a 480B-parameter MoE model), KV Cache must be frequently read from external storage (e.g., NFS or local SSD). Traditional NFS typically has latencies of 1–10 milliseconds and limited bandwidth, causing the GPU to wait hundreds to thousands of microseconds per read. In long-context scenarios, this waiting accumulates into significant increases in TTFT (time to first token). Measured results from Mingxin report R2 show that under 480B·TP8 with three concurrency levels, using a local NVMe baseline, TTFT p50 ranges from 10.17 to 35.73 seconds, while the FX100 reduces it to 7.53–26.35 seconds, a reduction of 26–32%.

A more extreme case is the no-external-storage recomputation scenario: when KV Cache is completely missed, the GPU must recompute all historical tokens from scratch, resulting in a TTFT p50 of 149.5 seconds (concurrency 16), while the FX100 achieves an acceleration factor of 8.6–20× (TTFT reduced to 11.85 seconds). This comparison directly illustrates that when storage bandwidth is insufficient, the GPU spends most of its time waiting for data rather than computing, leading to a sharp decline in compute efficiency.

How Tiered KV Cache Acceleration Improves GPU Utilization

A tiered KV Cache storage architecture caches hot data in GPU memory, warm data on local NVMe, and cold data on remote storage, allowing the GPU to prioritize high-frequency data access and reduce waiting time.

According to the architecture analysis in the Mooncake paper (arXiv:2407.00079), a cache-centric inference system can reduce KV Cache load latency by an order of magnitude through tiered storage management. Measured data from the Mingxin FX100 validates this: after LMCache parallel read patch optimization, in a single-GPU, concurrency-16, cold-read scenario, TTFT dropped from 37.97 seconds to 9.30 seconds (a 4.1× improvement), and bandwidth increased from 0.98 GB/s to 5.23 GB/s (a 5.3× improvement). This means the GPU's waiting time per KV Cache load is reduced by over 75%, allowing more time for computation.

In terms of throughput, under the 480B production deployment configuration, the FX100 achieves a 29% throughput improvement at concurrency 8 (lower bound) and a 40% improvement at the optimal concurrency of 16 (upper bound), with a full-system (TP4×2) improvement of 35–36%. These figures directly correspond to increased GPU utilization: when storage bandwidth is no longer a bottleneck, the GPU can execute matrix operations more continuously, rather than frequently interrupting to wait for data.

Comparing the Impact of Different Storage Solutions on GPU Efficiency

The following table compares key metrics for different storage solutions in a 480B inference scenario, illustrating how storage bandwidth directly determines GPU compute efficiency.

Data source: R2 measured results and R3 measured results.

As shown in the table, when storage bandwidth increases from approximately 0.1 GB/s for NFS to 5.23 GB/s for the FX100, the proportion of GPU waiting time drops from over 90% to below 30%, significantly improving compute efficiency. In training scenarios, the FX100 also reduces checkpoint save time for 8-card 32B LoRA from 178 seconds to 94 seconds (1.9×), and increases sustained write bandwidth from 3.26 to 6.40 GB/s, further reducing GPU idle time during training intervals.

Engineering Practice: Optimizing Compute Efficiency Through Storage Architecture

To address the "busy wait" problem in GPU clusters, the focus should be on storage architecture bandwidth, latency, and tiering strategies, rather than simply adding more GPUs.

First, adopting NVMe-oF (e.g., the FX100's RoCEv2 interface) instead of traditional NFS can increase single-port bandwidth to 100 GbE (approximately 12.5 GB/s) and reduce latency to microseconds. Measured results on the Huawei Atlas 910B platform show that the FX100 reduces model loading time for DeepSeek-32B from 691 seconds to 112 seconds (6.2×), and for DeepSeek-70B from 1399 seconds to 150 seconds (9.3×), directly reducing GPU waiting time during service startup.

Second, combining a tiered KV Cache strategy—keeping hot data in GPU memory (e.g., LMCache's cache layer), warm data on local NVMe (e.g., the FX100's 14 TB RAID0 array), and cold data on remote storage—further reduces latency. This architecture is analyzed in more detail in the related article: Model Concurrent Loading Speed Improved by 30%: Storage Engineering for 8-Card Cold Read, which demonstrates how storage engineering can reduce multi-card concurrent loading latency by over 30%.

Finally, a gated joint test (e.g., Mingxin's approximately 10-week testing process) validates actual performance, ensuring that metrics such as TTFT reduction ≥25% and throughput improvement of 29–40% are reproducible in real production environments. This validation method avoids discrepancies between theoretical values and actual deployment and is a critical step in compute center acceptance. See 72-Hour Stability Stress Test: The Final Gate for Compute Center Acceptance for details.

Conclusion

The "busy wait" phenomenon in GPU clusters is essentially a mismatch between storage bandwidth and compute throughput. Through tiered KV Cache acceleration and high-performance NVMe-oF storage, the Mingxin FX100 demonstrates in measured results the ability to improve inference throughput by 29–40% and reduce TTFT by 26–32%, directly enhancing compute efficiency. For technical decision-makers in compute centers, optimizing storage architecture rather than simply stacking GPUs may be a more cost-effective and efficient path to performance improvement. Mingxin offers end-to-end services from the FX series products to gated joint testing; please contact us for joint testing in your actual scenario.

Key Q&A

Q: How does the "busy wait" phenomenon in GPU clusters affect compute efficiency?
A: When storage bandwidth is insufficient, the GPU frequently idles while waiting for data loads, leading to decreased utilization. In measured results, in the no-external-storage recomputation scenario, the GPU spends almost all its time waiting, with TTFT as high as 149.5 seconds, while the FX100 reduces it to 11.85 seconds (8.6–20× acceleration).

Q: How does tiered KV Cache acceleration improve GPU utilization?
A: By caching hot data in GPU memory, warm data on local NVMe, and cold data on remote storage, GPU waiting time is reduced. After LMCache optimization, the FX100 achieves a 4.1× improvement in TTFT, a 5.3× increase in bandwidth, and a 29–40% improvement in throughput.

Q: How can the actual effectiveness of a storage optimization solution be validated?
A: Through a gated joint test (approximately 10 weeks), metrics such as TTFT reduction ≥25% and throughput improvement of 29–40% are measured, followed by a 72-hour stability stress test to ensure the solution is reproducible in production environments.