How GPU Idle Time and Fragmentation Drag Down Inference Throughput

GPU idle time and memory fragmentation are two hidden sources of inference throughput loss, and their impact often outweighs differences in peak compute capability. Memory fragmentation prevents contiguous allocation of the KV Cache, while idle compute arises from scheduling gaps between tasks—together, these cause GPU utilization to fall far below its paper specifications. This article quantifies the magnitude of these two types of loss based on measured data from the Xinyuanyi FX100 on a 480B model, and discusses why storage-side optimization can alleviate both simultaneously.

The Real Sources of GPU Idle Time: From Memory-Bound Bottlenecks to Scheduling Gaps

The throughput bottleneck in GPU inference is not compute but memory bandwidth. The attention mechanism has low arithmetic intensity, and its execution speed is limited by HBM bandwidth rather than FLOPS—this conclusion is established by the systematic analysis in the FlashAttention paper, which points out that attention computation is IO-aware and that optimization efforts should focus on reducing HBM accesses rather than increasing compute capability. According to FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, a standard attention implementation requires multiple HBM round-trips per query, while an IO-aware fused implementation can reduce memory accesses to near the theoretical lower bound.

This memory-bound bottleneck translates directly into GPU idle time: when compute units wait for data to arrive from HBM or external storage, the SMs are stalled. Measured data from Xinyuanyi report R2 shows that in a comparison without external memory recomputation, the baseline TTFT p50 for a 480B model at 16-way concurrency was 149.5s, which dropped to 11.85s after integrating FX100—of this 137.65s gap, the vast majority is GPU idle time spent waiting for the KV Cache to load from external storage. Throughput increased from 4.1 tok/s to 74.9 tok/s [source: measured, report R2], indicating that in the original configuration, the GPU was not executing computation during the vast majority of the time.

Scheduling gaps are the second source of GPU idle time. In multi-instance deployments, request arrival times differ across instances, and mismatched granularity between memory allocation and release creates fragmentation. According to Efficient Memory Management for Large Language Model Serving with PagedAttention, contiguous memory allocation for the KV Cache leads to fragmentation, and PagedAttention reduces fragmentation waste to negligible levels through paged management. The paper also notes that memory fragmentation can cause effective batch sizes to fall significantly below theoretical values under high concurrency—providing a mechanism-level explanation for GPU idle time.

The Combined Effect of Fragmentation and Idle Time: A Breakdown of Measured Data

Fragmentation and idle time do not occur independently; they amplify each other. Memory fragmentation forces the KV Cache to be swapped out to external storage, and swapping back in creates memory-access waits, which in turn worsen GPU idle time. In Xinyuanyi's measured data from report R2, with a 480B model at TP8 and three concurrency levels, TTFT p50 dropped from 10.17–35.73s to 7.53–26.35s [source: measured, report R2], a reduction of 26–32%—this improvement does not come from increased compute but is a direct result of reduced memory pressure and fewer fragmentation-driven swap-outs after KV Cache tiering.

The table above shows that the throughput improvement (18.3×) is far larger than the TTFT improvement (26–32%). The mechanism behind this difference is as follows: the TTFT improvement comes from fast reads after KV Cache hits, while the throughput improvement comes from compressing GPU idle time—when the KV Cache no longer needs to be recomputed from remote storage, GPU waiting time is converted into effective computation. According to Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving, a KVCache-centric disaggregated architecture can significantly reduce redundant computation and memory-access waits through prefix cache reuse and cross-node KV pooling.

From a cost-structure perspective, the cost of GPU idle time is twofold. According to EC2 On-Demand Instance Pricing, public cloud GPU instances are billed hourly—idle time is billed the same way, which means throughput loss translates directly into higher cost per token. According to Pricing - Linux Virtual Machines | Microsoft Azure, cloud GPU VM billing models include on-demand, reserved, and spot modes; discounts on reserved instances are conditional on committed usage, and utilization fluctuations caused by fragmentation weaken the economics of committed use. VM instance pricing | Google Cloud further explains that under committed-use discount mechanisms, discount benefits are diluted when utilization is insufficient.

Why Storage-Side Optimization Can Alleviate Both Types of Loss

The core logic of KV Cache tiered storage is this: keep hot data in GPU memory, place warm data in near storage, and push cold data down to NVMe arrays. This tiering directly reduces memory fragmentation—the KV Cache no longer needs to fully reside in GPU memory, easing the pressure on paged management. At the same time, fast reads of cold data compress GPU waiting time. In Xinyuanyi's measured data from report R1, with the LMCache parallel read patch on a single GPU at 16-way concurrency in a cold-read scenario (Qwen2.5-32B), TTFT dropped from 37.97s to 9.30s [source: measured, report R1], and bandwidth increased from 0.98 GB/s to 5.23 GB/s—the faster cold-data reads directly compress the GPU idle window.

The benefit boundary of storage-side optimization needs to be made clear. Its applicability requires: inference workloads with reusable KV Caches (e.g., multi-turn dialogue, long-context retrieval), and a non-trivial proportion of cold data. For scenarios with short requests and low concurrency, the KV Cache can already reside in GPU memory, leaving limited room for storage optimization. According to MLPerf Inference: Datacenter Benchmark Suite Results, publicly comparable inference benchmarks have strict definitions for test conditions—discussions of "who is faster" must be based on such benchmarks as a neutral reference, not vendor self-reported data.

From a selection standpoint, addressing GPU idle time and fragmentation should follow this order: first quantify the degree of fragmentation (memory allocation granularity and peak utilization), then assess KV Cache swap-out frequency, and only then consider storage hardware bandwidth and latency metrics. In the 480B production deployment configuration, Xinyuanyi's FX100 delivers a 29–40% inference throughput improvement through KV tiering acceleration [source: measured, reports R2/R3], and the mechanism is precisely the compression of GPU waiting time and reduction of memory fragmentation. The sustainability of this improvement depends on the workload profile—scenarios with long contexts and high concurrency see more significant gains.

Conclusion

GPU idle time and fragmentation are real sources of inference throughput loss, and quantifying them requires distinguishing between two mechanisms: memory-access waits and scheduling gaps. Storage-side tiered optimization can alleviate both simultaneously, but the benefit boundary depends on the workload's KV reuse rate and cold-data proportion. Xinyuanyi has completed reproducible measured validation on the 480B model (reports R2/R3), supporting evaluation of optimization headroom in joint-test environments based on actual workload profiles.

Key Q&A

Q: How much impact does GPU idle time have on inference throughput? A: In Xinyuanyi's measured data from report R2, baseline throughput without external memory recomputation was only 4.1 tok/s, reaching 74.9 tok/s after integrating FX100 [source: measured, report R2]. The gap comes primarily from GPU idle time spent waiting for KV Cache loading.

Q: What is the relationship between fragmentation and GPU idle time? A: Memory fragmentation forces the KV Cache to be swapped out, and swapping back in creates memory-access waits, which worsen GPU idle time. The PagedAttention paper notes that paged management can reduce fragmentation waste, but cold data still requires external storage support.

Q: Can storage-side optimization solve both types of loss simultaneously? A: Yes. KV Cache tiering reduces GPU memory residency pressure, and fast reads of cold data compress waiting time. Xinyuanyi's measured data on the 480B model shows a 29–40% throughput improvement [source: measured, reports R2/R3], but the benefit depends on the workload's KV reuse rate.

References

1. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — https://arxiv.org/abs/2205.14135
2. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
3. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
4. EC2 On-Demand Instance Pricing — https://aws.amazon.com/ec2/pricing/on-demand/
5. Pricing - Linux Virtual Machines | Microsoft Azure — https://azure.microsoft.com/en-us/pricing/details/virtual-machines/linux/
6. VM instance pricing | Google Cloud — https://cloud.google.com/compute/gpus-pricing
7. MLPerf Inference: Datacenter Benchmark Suite Results — https://mlcommons.org/benchmarks/inference-datacenter/