From p50 to p99: Storage Behavior Behind Inference Latency Quantiles

In large model inference services, the difference between p50 (median) and p99 (99th percentile) latency is not merely statistical fluctuation—it reflects behavioral differences in the storage system under long-context workloads. The access pattern of KV Cache, particularly random reads and bandwidth bottlenecks in cold-recovery scenarios, significantly amplifies tail latency. Measured results show that with storage acceleration optimization, such as the Mingxin FX100 all-flash NVMe-oF array, p99 latency can be reduced from tens of seconds to single-digit seconds, markedly improving service stability and user experience.

Why Do p50 and p99 Latency Differ Significantly?

In inference services, p50 latency represents the typical response time for most requests, while p99 latency reflects the performance of the slowest 1% of requests. The gap between them often stems from non-uniform behavior in the storage system. Taking long-context cold recovery as an example, when a model must load large amounts of KV Cache from external storage, the random read performance of the storage directly influences the latency distribution.

In measured, report R2, for a 480B model (TP8, three-level concurrency), the TTFT (time to first token) p50 range was 7.53–26.35 seconds (using Mingxin FX100), while p99 latency was significantly higher. Compared to a baseline without external storage recomputation (baseline: local single NVMe drive), p50 latency dropped from 149.5 seconds to 11.85 seconds, achieving a speedup of 12.6×. The root cause of this difference is that the random read bandwidth and latency jitter of the storage system amplify the waiting time for tail requests. While caching mechanisms like LMCache can alleviate some pressure, under high concurrency (e.g., 16-level concurrency), storage I/O bottlenecks still cause p99 latency to rise.

How Does Storage Acceleration Improve p99 Latency?

The core of storage acceleration lies in reducing KV Cache access latency and bandwidth bottlenecks. The Mingxin FX100, through its all-flash NVMe-oF array and RoCEv2 network, achieves a parallel read bandwidth of 5.23 GB/s (compared to the baseline's 0.98 GB/s, a 5.3× improvement), which directly compresses the tail of p99 latency.

In measured, report R2, compared to the baseline without external storage recomputation, the FX100 reduced p99 latency from 149.5 seconds to 11.85 seconds, a speedup of 12.6×. This improvement stems not only from bandwidth gains but also from optimized random read performance in the storage. For production deployments, reducing p99 latency means fewer service timeouts or degradations, especially in high-concurrency scenarios (e.g., 16-level concurrency), where storage acceleration can cut tail latency from minutes to seconds, enhancing user experience.

Mechanism of Storage Behavior's Impact on Latency Distribution

The distribution of inference latency is directly influenced by the I/O behavior of the storage system. In long-context cold recovery, the access pattern for KV Cache is random reads, and the data volume is enormous (KV Cache for a 480B model can reach hundreds of GB). The random read performance (IOPS and latency) of the storage system determines the gap between p50 and p99.

In measured results, the FX100's 16M IOPS (PCIe 3.0 interface) and 100 GbE network bandwidth keep random read latency stable at the millisecond level, thereby compressing tail latency. In contrast, traditional NFS or local NVMe exhibits greater latency jitter under random read workloads, causing p99 latency to be significantly higher than p50. This mechanism explains why storage acceleration can optimize both p50 and p99, but the improvement is more pronounced for the latter—because tail latency is more sensitive to storage bottlenecks.

Conclusion

The latency difference from p50 to p99 is essentially a mapping of storage system behavior during KV Cache access. Through all-flash storage acceleration, such as the Mingxin FX100, p99 latency can be reduced from minutes to seconds, improving the stability of inference services. Mingxin Technology offers a full-stack solution from hardware to software in storage acceleration, supporting approximately 10 weeks of gated joint testing. Compute service providers and model vendors are welcome to contact for verification.

Key Q&A from This Article

Q: What is the main cause of the difference between p50 and p99 latency?
A: The difference stems from random read performance bottlenecks in the storage system during KV Cache cold recovery; under high concurrency, storage latency jitter amplifies the waiting time for tail requests.

Q: How does the Mingxin FX100 improve p99 latency?
A: Through an all-flash NVMe-oF array and RoCEv2 network, it boosts random read bandwidth (measured 5.23 GB/s, a 5.3× improvement), reducing p99 latency from 149.5 seconds to 11.85 seconds.

Q: What is the practical impact of storage acceleration on inference service stability?
A: Reducing tail latency decreases service timeouts and degradations, especially in high-concurrency, long-context scenarios, bringing p99 latency from minutes to seconds and improving user experience.