How vLLM Prefix Caching Reduces Inference Latency

Latency Gains from Prefix Caching

Prefix caching reuses computed KV states to significantly reduce time-to-first-token (TTFT) for large-model inference and improve throughput. Measured on Mingxin's platform with a 480B-parameter model across 8 AMD MI308X GPUs, layered acceleration on the storage side delivered throughput gains in the +29–40% range (measured, report R2/R3) and TTFT reductions of 26–32% (measured, report R2). Note that these figures reflect Mingxin's own test platform and specific workload patterns; applicability to your production environment depends on context-length distribution, concurrency profiles, and cache hit rates.

The core mechanism of prefix caching is straightforward: when multiple requests share the same prefix—such as system prompts, few-shot examples, or multi-turn conversation history—the system computes the KV tensors for that portion only once, and subsequent requests reuse them directly. According to Efficient Memory Management for Large Language Model Serving with PagedAttention, paged management of the KV cache resolves GPU memory fragmentation, enabling cache reuse. Per SGLang: Efficient Execution of Structured Language Model Programs, RadixAttention organizes caches via a prefix tree, further improving hit rates in shared-prefix scenarios.

How Cache Hit Rates Affect Latency Gains

The benefits of prefix caching depend heavily on hit rates. In ideal scenarios—multi-turn conversations, batch processing with shared system prompts, code completion—hit rates can reach high levels, and latency gains approach the upper bound observed in Mingxin's tests. In workloads with random queries or highly divergent prefixes, hit rates drop, and gains narrow significantly.

Mingxin's R2 measured workload used a 480B model, TP8 deployment, and cold recovery with long contexts—a scenario that stresses the cache heavily. Even so, TTFT p50 dropped from 10.17–35.73s to 7.53–26.35s (measured, report R2). Throughput improved +29% (lower bound) at concurrency 8, with concurrency 16 as the optimal operating point at +40% (upper bound) (measured, reports R2/R3).

A notable comparison: under a baseline with no external-memory recomputation (i.e., KV computed from scratch for every request), TTFT p50 reached 149.5s (concurrency 16), while with FX100 acceleration it was 11.85s—an acceleration factor of 8.6–20× (measured, report R2). This shows that when the cache misses entirely, layered storage-side acceleration can substantially compress recomputation costs—a second layer of benefit beyond prefix caching.

Why Storage Performance Becomes the Bottleneck on Cache Misses

Prefix caching is not a universal solution. When a cache miss occurs, or KV states must be swapped in and out of external storage, storage bandwidth directly dictates latency. According to FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, attention computation is limited by HBM bandwidth rather than compute capacity—a conclusion that extends to external storage: the read bandwidth for KV determines the time for recomputation or swap-in.

In Mingxin's R1 measurements, on a single GPU with concurrency 16 and cold disk reads (Qwen2.5-32B), the LMCache parallel-read patch reduced TTFT from 37.97s to 9.30s, with bandwidth rising from 0.98 GB/s to 5.23 GB/s (↑5.3×). This indicates that storage-side parallel-read optimizations and caching mechanisms are complementary: caching improves hit rates, while storage acceleration reduces the cost of misses.

Per the NVIDIA GPUDirect Storage Documentation, GPU-direct storage bypasses the CPU bounce buffer to shorten the data path. This mechanism aligns with the architecture of Mingxin's FX100 NVMe-oF all-flash array, but Mingxin's measured data comes from its own platform; cross-platform comparisons are outside the scope of this article.

Storage Acceleration Also Benefits Training Scenarios

Prefix caching primarily serves inference, but storage acceleration is equally effective for training. Mingxin's R1 measurements show that in 8-GPU 32B LoRA training, saving each 65.6GB full-model snapshot dropped from 178s to 94s (1.9×), with sustained write bandwidth improving from 3.26 GB/s to 6.40 GB/s (+96%). Faster checkpoint saving directly reduces training interruption time, which has practical significance for scheduling efficiency in large-scale training jobs.

For model loading on Ascend platforms, Mingxin's R9 measurements show DeepSeek-32B service loading reduced from 691s to 112s (6.2×), and DeepSeek-70B from 1399s to 150s (9.3×). Model loading is the dominant cost in inference service cold starts, making this acceleration especially critical for elastic scaling scenarios.

How to Evaluate Whether Your Scenario Justifies Adoption

To determine whether prefix caching and storage acceleration fit your scenario, consider the following steps:

1. Analyze request prefix characteristics: Measure system prompt length, multi-turn conversation depth, and the proportion of shared prefixes. If shared prefixes are rare, caching gains will be limited.
2. Measure current TTFT and throughput: Establish a baseline. If TTFT already meets your SLA, optimization can be deprioritized.
3. Assess storage bottlenecks: If latency spikes sharply on cache misses, storage bandwidth is the constraint, and an NVMe-oF-class solution is worth considering.
4. Define verifiable acceptance criteria: Mingxin's engagement model involves approximately 10 weeks of gate-based joint testing, with core gates of TTFT reduction ≥25% and throughput within the +29–40% measured band (G3); if targets are not met, the engagement stops. This model suits buyers with hard performance requirements.

Note that inference cost metrics should be normalized per million tokens, per concurrency, or per QPS. Define SLA, context length, and concurrency constraints first, then discuss optimization approaches. According to public research from Epoch AI, AI compute scale and costs show a sustained upward trend, and the value of storage and caching optimization grows with longer context lengths—but this is third-party research; refer to that institution's current publications for specific figures.

Key Q&A

Q: How much inference latency can prefix caching reduce? A: On Mingxin's platform with a 480B model and 8 MI308X GPUs, TTFT decreased by 26–32% and throughput improved by 29–40% (measured, reports R2/R3). Actual gains depend on cache hit rates and concurrency patterns.

Q: What happens on a cache miss? A: Storage bandwidth becomes the bottleneck. In Mingxin's measurements, the LMCache parallel-read patch reduced cold-read TTFT from 37.97s to 9.30s (measured, report R1). Storage-side acceleration and caching mechanisms are complementary.

Q: How can I verify whether the solution fits my scenario? A: Start by analyzing request prefix characteristics and measuring current TTFT baselines, then validate via gate-based joint testing—Mingxin offers approximately 10 weeks of testing, with core gates of TTFT reduction ≥25% and throughput within the +29–40% measured band; if targets are not met, the engagement stops.

References

1. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
2. SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
3. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — https://arxiv.org/abs/2205.14135
4. NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html
5. Epoch AI — https://epoch.ai/