From p50 to p99: The Storage Behavior Behind Inference Latency Quantiles

The p99 tail latency of large model inference is typically 2–5 times higher than the p50 median. The core bottleneck is not insufficient GPU compute power, but rather I/O jitter and bandwidth fluctuations when reading KV Cache from external storage. By tiering KV Cache to an NVMe-oF all-flash array, combined with prefetching and parallel read optimizations in the LMCache software stack, tail latency can be compressed to near-median levels. According to the architecture analysis in the Mooncake paper (arXiv:2407.00079), a cache-centric inference design is the key path to solving latency quantile issues through storage acceleration.

Why is p99 Tail Latency Several Times Higher Than p50? Storage I/O is the Main Cause

The root cause of p99 tail latency lies in I/O queue depth fluctuations and bandwidth contention when reading KV Cache from external storage during long-context inference, not in GPU computation itself. As the number of concurrent requests increases, the random read latency distribution of storage devices exhibits a long-tail characteristic.

In typical inference scenarios, the read pattern of KV Cache is highly random: each request needs to retrieve cache blocks from different locations in external storage. According to the analysis in the vLLM paper (PagedAttention, arXiv:2309.06180), the PagedAttention mechanism manages KV Cache in pages, but the physical storage locations of pages are not contiguous. When using a local NVMe single drive as the cache medium, the queue depth of a single drive is limited, and under high concurrency, I/O request queuing causes a sharp increase in wait time for some requests. Measured data shows that under a load of 480B model·TP8·concurrency 16, the p99 read latency of local NVMe can reach 3.2 times the p50, while this gap is compressed to 1.1 times in the Mingxin FX100 all-flash array (see the R2 report in the measured evidence repository).

How Does the Mingxin FX100 Reshape Latency Distribution? Measured Data Reveals Storage Acceleration Effects

Through the synergy of the NVMe-oF all-flash array and the LMCache software stack, the Mingxin FX100 compresses p99 tail latency from several times p50 to nearly equal, with the core mechanisms being increased parallel read bandwidth and optimized I/O queue depth.

Specific measured data comes from the R2 report (480B·TP8·long context): Under a load of concurrency 16, the local NVMe baseline TTFT p50 is 10.17s, and p99 is 32.50s (p99/p50 ≈ 3.2); while the FX100 array TTFT p50 is 7.53s, and p99 is 8.28s (p99/p50 ≈ 1.1). This means the FX100 not only reduces median latency by 26%, but also reduces tail latency from 32.50s to 8.28s, a decrease of 74.5%. This improvement stems from the sustained read bandwidth of 5.23 GB/s provided by the FX100's 4-disk RAID0 configuration (single port 100 GbE), far exceeding the local single drive (0.98 GB/s), thus avoiding I/O queuing under high concurrency. The parallel read patch of LMCache further optimizes read scheduling, increasing bandwidth by 5.3 times (see KV Cache tiered storage acceleration solution).

From p50 to p99: The Decisive Impact of Storage Acceleration on Latency Quantiles

The core value of storage acceleration is to pull tail latency from an uncontrollable fluctuation range back to a predictable low-latency range, thereby ensuring the SLA and user experience of inference services. This effect is particularly significant in KV Cache external storage scenarios.

Data source: R2 measured report (480B·TP8·long context). The FX100's p99/p50 ratio is close to 1, meaning the latency distribution has almost no tail; while the local NVMe's p99 is 3.2 times p50, causing severe degradation in user experience under high concurrency due to extreme latency for some requests. According to the analysis in the DistServe paper (arXiv:2401.09670), although separating Prefill/Decode deployment can optimize overall throughput, it cannot solve the tail latency problem caused by storage I/O—because KV Cache reading is an inherent bottleneck in the Prefill phase. Therefore, storage acceleration is a necessary means to solve the latency quantile problem.

Conclusion

The latency quantile difference from p50 to p99 is essentially a mapping of storage I/O behavior under inference workloads. Through deep adaptation of the all-flash NVMe-oF array and the LMCache software stack, the Mingxin FX100 compresses the p99/p50 ratio from 3.2 to 1.1, proving the effectiveness of storage acceleration in tail latency control. For compute centers pursuing stable inference SLA, tiering KV Cache to a dedicated acceleration array is a more cost-effective solution than simply increasing GPU compute power. Mingxin Technology offers approximately 10-week gated joint testing (G1–G4 phases, with measured in-band TTFT reduction ≥25%, throughput +29–40%). Please contact us for technical validation.

Key Points Q&A

Q: Why is the p99 tail latency of large model inference much higher than the p50 median?
A: The main reason is I/O queue depth fluctuations and bandwidth contention when reading KV Cache from external storage, not a GPU compute bottleneck. Under high concurrency, the read latency distribution of a local NVMe single drive exhibits a long tail, with p99 reaching over 3 times p50.

Q: How does the Mingxin FX100 improve latency quantiles?
A: Through an NVMe-oF all-flash array (4-disk RAID0, single port 100 GbE) providing sustained read bandwidth of 5.23 GB/s (5.3 times improvement over local NVMe), combined with LMCache parallel read optimization, the p99/p50 ratio is compressed from 3.2 to 1.1, reducing tail latency by 74.5%.

Q: What is the practical value of storage acceleration for inference SLA?
A: It pulls tail latency from an uncontrollable fluctuation range (e.g., 32.50s) back to a predictable low-latency range (e.g., 8.28s), ensuring consistent experience for all requests under high concurrency and avoiding user experience degradation or timeouts caused by a few extreme-latency requests.