GPU Cluster "Busy Waiting": How Storage Bandwidth Shapes Compute Efficiency

In GPU cluster inference scenarios, insufficient storage bandwidth is a core bottleneck causing compute "busy waiting" (GPU idling while waiting for data). Tiered KV Cache acceleration combined with high-performance NVMe-oF storage can boost inference throughput by 29–40% and reduce time-to-first-token (TTFT) by 26–32%. This "busy waiting" is especially pronounced during long-context inference and multi-instance concurrency: when the GPU must load historical KV Cache or model weights from external storage, and storage bandwidth falls far below GPU compute throughput, the GPU frequently enters idle cycles, reducing compute efficiency. Measured on the AMD MI308X platform, Mingxin Technology’s FX100 all-flash array demonstrates that reducing storage latency from seconds to milliseconds significantly alleviates this bottleneck, bringing GPU utilization closer to theoretical peaks.

How Insufficient Storage Bandwidth Causes GPU Idle Time

When storage bandwidth cannot match GPU compute demand, the GPU enters a waiting state during each data load. In inference, this "busy waiting" manifests as lower GPU utilization and higher end-to-end latency.

In the PagedAttention mechanism (based on the vLLM paper, arXiv:2309.06180), on-demand loading of KV Cache is a critical step. When model size exceeds GPU memory (e.g., a 480B-parameter MoE model), KV Cache must be frequently read from external storage (e.g., NFS or local SSD). Traditional NFS typically has 1–10 ms latency with limited bandwidth, forcing the GPU to wait hundreds to thousands of microseconds per read. In long-context scenarios, this wait accumulates into significant TTFT increases. Measured in report R2, under 480B·TP8 with three concurrency levels, the local NVMe baseline showed TTFT p50 of 10.17–35.73 seconds, while the FX100 reduced it to 7.53–26.35 seconds—a 26–32% reduction.

A more extreme case is recomputation without external storage: when KV Cache is completely missed, the GPU must recompute all historical tokens from scratch, resulting in a TTFT p50 of 149.5 seconds (concurrency 16). The FX100 achieves an 8.6–20× speedup (TTFT down to 11.85 seconds). This comparison directly shows that when storage bandwidth is insufficient, the GPU spends most of its time waiting for data rather than computing, causing a sharp drop in compute efficiency.

How Tiered KV Cache Acceleration Improves GPU Utilization

A tiered KV Cache storage architecture caches hot data in GPU memory, warm data on local NVMe, and cold data on remote storage, allowing the GPU to prioritize high-frequency data access and reduce wait time.

According to the architecture analysis in the Mooncake paper (arXiv:2407.00079), a cache-centric inference system using tiered storage management can reduce KV Cache load latency by an order of magnitude. Measured data from the Mingxin FX100 confirms this: after LMCache parallel read patch optimization, in a single-GPU, concurrency-16, cold-read scenario, TTFT dropped from 37.97 seconds to 9.30 seconds (a 4.1× improvement), and bandwidth increased from 0.98 GB/s to 5.23 GB/s (a 5.3× improvement). This means the GPU waits over 75% less time per KV Cache load, freeing up more time for computation.

In terms of throughput, in a 480B production deployment, the FX100 delivers a 29% throughput improvement at concurrency 8 (lower bound) and a 40% improvement at the optimal concurrency 16 (upper bound). At the full-node level (TP4×2), the improvement is 35–36%. These numbers directly correspond to higher GPU utilization: when storage bandwidth is no longer the bottleneck, the GPU can execute matrix operations more continuously, without frequent interruptions to wait for data.

Comparing Storage Solutions’ Impact on GPU Efficiency

The following table compares key metrics across different storage solutions in a 480B inference scenario, showing how storage bandwidth directly determines GPU compute efficiency.

Data sources: measured in report R2 and measured in report R3.

As the table shows, when storage bandwidth increases from ~0.1 GB/s (NFS) to 5.23 GB/s (FX100), the GPU’s waiting time drops from over 90% to under 30%, significantly improving compute efficiency. In training scenarios, the FX100 also reduces checkpoint save time for 8-card 32B LoRA from 178 seconds to 94 seconds (1.9×), and increases sustained write bandwidth from 3.26 to 6.40 GB/s, further reducing GPU idle time between training cycles.

Engineering Practice: Optimizing Compute Efficiency from the Storage Architecture

To solve the "busy waiting" problem in GPU clusters, the focus should be on storage architecture—bandwidth, latency, and tiering strategies—rather than simply adding more GPUs.

First, replacing traditional NFS with NVMe-oF (e.g., the FX100’s RoCEv2 interface) can boost single-port bandwidth to 100 GbE (~12.5 GB/s) and reduce latency to microseconds. Measured on the Huawei Atlas 910B platform, the FX100 reduced model loading time for DeepSeek-32B from 691 seconds to 112 seconds (6.2×), and for DeepSeek-70B from 1,399 seconds to 150 seconds (9.3×), directly cutting GPU wait time during service startup.

Second, combine this with a tiered KV Cache strategy: keep hot data in GPU memory (e.g., LMCache’s cache layer), warm data on local NVMe (e.g., the FX100’s 14 TB RAID0 array), and only access cold data from remote storage. This architecture is further analyzed in the related article: "30% Faster Concurrent Model Loading: Storage Engineering for 8-Card Cold Read", which shows how storage engineering reduces multi-card concurrent load latency by over 30%.

Finally, use gated joint validation (e.g., Mingxin’s ~10-week test process) to verify real-world results, ensuring that metrics such as TTFT reduction ≥25% and throughput improvement of 29–40% are reproducible in actual production environments. This validation approach avoids discrepancies between theoretical values and real-world deployment and is a critical step in AI datacenter acceptance. See the 72-hour stability stress test: the final gate for AI datacenter acceptance for more details.

Conclusion

The "busy waiting" phenomenon in GPU clusters is fundamentally a mismatch between storage bandwidth and compute throughput. Through tiered KV Cache acceleration and high-performance NVMe-oF storage, the Mingxin FX100 has demonstrated the ability to increase inference throughput by 29–40% and reduce TTFT by 26–32% in real-world tests, directly improving compute efficiency. For AI datacenter technical decision-makers, optimizing the storage architecture—rather than simply stacking more GPUs—may be a more cost-effective and efficient path to performance gains. Mingxin offers end-to-end services, from the FX series products to gated joint validation. Contact us to conduct joint testing in your actual scenario.

Key Takeaways (Q&A)

Q: How does the "busy waiting" phenomenon in GPU clusters affect compute efficiency?
A: When storage bandwidth is insufficient, the GPU frequently idles while waiting for data loads, leading to lower utilization. In the recomputation-without-external-storage scenario, the GPU spends almost all its time waiting, with TTFT as high as 149.5 seconds. The FX100 reduces this to 11.85 seconds (an 8.6–20× speedup).

Q: How does tiered KV Cache acceleration improve GPU utilization?
A: By caching hot data in GPU memory, warm data on local NVMe, and cold data on remote storage, it reduces GPU wait time. After LMCache optimization, the FX100 achieves a 4.1× TTFT improvement and a 5.3× bandwidth increase, resulting in a 29–40% throughput improvement.

Q: How can the real-world effectiveness of a storage optimization solution be verified?
A: Through gated joint validation (approximately 10 weeks), metrics such as TTFT reduction ≥25% and throughput improvement of 29–40% are measured in real-world scenarios, followed by a 72-hour stability stress test to ensure the solution is reproducible in production environments.