Bandwidth Bottlenecks in AI Inference Storage Acceleration and Optimization Paths

The core challenge in storage acceleration for AI inference is not capacity but bandwidth: attention computation is limited by HBM bandwidth rather than compute, and the swapping of KV Cache to external storage makes PCIe and network bandwidth the new bottleneck. Measured data from Mingxin FX100 shows that for long-context cold-restore workloads, KV tiered acceleration improves inference throughput by 29–40% and reduces time-to-first-token (TTFT) by 26–32% [measured, reports R2/R3]. This article analyzes the sources of bandwidth demand from a memory-access perspective and compares the applicability boundaries of two optimization strategies.

Why KV Cache External Storage Access Becomes a Bandwidth Bottleneck

The attention mechanism in large-model inference exhibits a distinctly memory-access-intensive profile. According to FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, attention computation is limited by HBM bandwidth rather than compute, and the gains from IO-aware optimization stem precisely from this. As sequence length grows, KV Cache size expands; once it exceeds the memory capacity of a single GPU, it must be swapped to external storage. At that point, the bandwidth of the external storage link directly determines the latency of swap-in and swap-out.

Take the Mingxin R2 test platform as an example: 8× AMD MI308X GPUs, each with 192 GB HBM, running Qwen3-Coder-480B-FP8 (MoE, weights approximately 450 GB). Across three concurrency levels under TP8, TTFT p50 dropped from a baseline of 10.17–35.73 s to 7.53–26.35 s [measured, report R2]. This improvement comes from migrating KV Cache from a single local NVMe drive to the FX100 all-flash NVMe-oF array (4-drive RAID0, RoCEv2, 100 GbE per port), which multiplies external storage read bandwidth.

It is worth emphasizing that bandwidth demand is not uniformly distributed. In cold-start or long-context restore scenarios, KV Cache must be read in bulk at once, and instantaneous bandwidth demand far exceeds that of steady-state inference. According to Efficient Memory Management for Large Language Model Serving with PagedAttention, KV Cache paging addresses GPU memory fragmentation and dynamic allocation, but paging itself does not change the physical ceiling of external storage bandwidth. Therefore, storage acceleration solutions must be designed around peak bandwidth, not average bandwidth.

Tiered Acceleration vs. Direct Storage: Measured Comparison of Two Optimization Paths

For the bandwidth bottleneck described above, current mainstream optimization paths fall into two categories: first, KV Cache tiered acceleration, which reduces the number of external storage accesses through cache tiering and prefetching strategies; second, GPU Direct Storage (GDS), which shortens the data path by bypassing CPU memory copies. The two paths can be combined, but their sources of benefit differ.

The benefit of tiered acceleration is quantified in Mingxin R2/R3 tests. Under a 480B production-deployment long-context cold-restore workload, throughput improves by 29% at concurrency 8 (lower bound) and by 40% at the optimal operating point of concurrency 16 (upper bound); at the full-machine level with TP4×2, throughput improves by 35–36% [measured, reports R2/R3]. The mechanism: hot KV blocks are retained in GPU memory or near-storage media, while cold blocks are prefetched on demand, spreading external storage bandwidth pressure from peak across the time axis.

The qualitative value of GPU Direct Storage lies in eliminating the data-copy overhead of the CPU bounce buffer. According to the NVIDIA GPUDirect Storage Documentation, this technology provides GPUs with a direct path to storage devices, avoiding data relay through CPU memory. In the Mingxin R1 test, the LMCache parallel-read patch combined with FX100, under a single-GPU concurrency-16 cold-read scenario, reduced TTFT from 37.97 s to 9.30 s and increased bandwidth from 0.98 GB/s to 5.23 GB/s [measured, report R1]. This result includes both tiered prefetching and direct-read optimizations, but the 5.3× bandwidth improvement indicates substantial headroom in the external storage path itself.

Upper Bound of Bandwidth Demand in No-External-Storage Recompute Scenarios

When KV Cache cannot reside in GPU memory at all, inference degrades to the recompute baseline, and bandwidth demand reaches its upper bound. In the R2 test, the recompute baseline TTFT p50 was 149.5 s (concurrency 16), while the FX100 solution achieved 11.85 s, a speedup of 8.6–20× [measured, report R2]. This comparison reveals that the value ceiling of a storage acceleration solution depends on how severely the unoptimized baseline degrades.

The implication for storage systems: bandwidth planning should reference peaks, not averages. For the 480B model as an example, if KV Cache swap-in must complete within seconds, the required bandwidth reaches the tens of GB/s scale. According to Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving, a KVCache-centric disaggregated compute-storage architecture improves resource utilization through cross-node KV pooling, but pooling itself depends on high-speed networking. The Mingxin FX100's single-interface 100 Gb (PCIe 3.0) specification is sufficient for single-GPU scenarios, but for multi-GPU concurrency, the array's aggregate bandwidth must be evaluated against the inference cluster's concurrency scale.

It should be noted that the bandwidth figures above come from measured results on the Mingxin FX100 with the AMD MI308X platform (reports R1–R4), with the test environment running ROCm 7.2, vLLM 0.20.1+rocm721, and LMCache compiled from upstream mainline source. Absolute values will differ across GPU platforms and framework versions, but the qualitative conclusion about the bandwidth bottleneck and the optimization direction are generally applicable.

Conclusion

The bandwidth demand in AI inference storage acceleration is fundamentally determined by the memory-access-intensive nature of attention computation and the external storage swap of KV Cache. Tiered acceleration and GPU Direct Storage alleviate the bottleneck from two dimensions—reducing access count and shortening the data path, respectively. Measured data from the Mingxin FX100 (throughput +29–40%, TTFT ↓26–32%) provides a reproducible quantitative reference. For compute-center builders, we recommend planning around peak bandwidth rather than average bandwidth during solution selection, and validating actual gains through gate-based joint testing (e.g., TTFT reduction ≥25%). Mingxin offers a gate-based joint testing collaboration model of approximately 10 weeks, with a Python-reproducible measurement model available under NDA. Teams with long-context inference optimization needs are welcome to reach out.

Key Q&A

Q: Why does the bandwidth bottleneck in AI inference storage acceleration concentrate on KV Cache external storage access? A: Attention computation is limited by HBM bandwidth (qualitative conclusion per the FlashAttention paper). As sequence length grows, KV Cache exceeds GPU memory and must be swapped out; the external storage link bandwidth directly determines swap-in latency. Mingxin R2 measurements show that after optimizing the external storage path, TTFT p50 dropped from 10.17–35.73 s to 7.53–26.35 s [measured, report R2].

Q: What are the measured benefits of KV tiered acceleration versus GPU Direct Storage? A: Tiered acceleration improves throughput by 29–40% under 480B cold-restore workloads [measured, reports R2/R3]; GDS eliminates CPU relay overhead (qualitative conclusion per NVIDIA GDS documentation), and combined with the LMCache parallel-read patch, single-GPU cold-read TTFT dropped from 37.97 s to 9.30 s [measured, report R1].

Q: Should bandwidth planning be based on average or peak values? A: Planning should be based on peak bandwidth. During cold start or long-context restore, KV Cache must be read in bulk at once, and instantaneous bandwidth demand far exceeds steady state. In the R2 test, the no-external-storage recompute baseline TTFT reached 149.5 s, compared to 11.85 s with FX100, a speedup of 8.6–20× [measured, report R2].

References

1. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — https://arxiv.org/abs/2205.14135
2. NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html
3. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
4. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079