Comparing KV Cache Hit-Rate Optimization Strategies in vLLM Inference

KV Cache hit rate directly determines the latency and throughput of large-model inference. Optimization approaches fall into three main categories: vLLM's paged management, prefix-tree reuse represented by RadixAttention, and disaggregated storage-compute architectures represented by Mooncake. Measured data from Mingxin's FX100 shows that under long-context cold-restore workloads, tiered storage can improve throughput by 29–40% and reduce time-to-first-token (TTFT) by 26–32% [measured, reports R2/R3]. This article compares these three routes across mechanism, applicable scenarios, and measured results.

Why KV Cache Hit Rate Is the Gatekeeper of Inference Performance

KV Cache stores key-value tensors during autoregressive decoding, with memory usage growing linearly with sequence length. According to "Efficient Memory Management for Large Language Model Serving with PagedAttention," traditional memory management suffers from severe fragmentation. PagedAttention divides KV Cache into fixed-size blocks allocated on demand, improving memory utilization. However, paging only solves the "can it fit" problem, not the "does it need recomputation" problem—when a request's prefix does not match the cache, recomputation is still required.

According to "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness," attention computation is limited by HBM bandwidth rather than compute capacity. This means that recomputation caused by KV Cache misses wastes not only compute but also repeatedly consumes memory-access bandwidth. In long-context scenarios, this cost is amplified: the longer the context, the more tokens must be recomputed per miss.

Mechanisms and Trade-offs of the Three Main Technical Routes

Route 1: Paged Management (PagedAttention)

vLLM's core contribution is paged KV Cache management. Per "Efficient Memory Management for Large Language Model Serving with PagedAttention," this mechanism pushes memory utilization close to zero fragmentation, supporting higher concurrency. Its hit-rate optimization logic is "let more requests reside in memory simultaneously"—the higher the concurrency, the greater the chance of shared prefixes across requests.

Applicable scenarios: high-concurrency online inference with short-to-medium contexts and limited prefix sharing.

Route 2: Prefix-Tree Reuse (RadixAttention)

RadixAttention, proposed by SGLang, organizes KV Cache as a prefix tree and automatically reuses shared prefixes. According to "SGLang: Efficient Execution of Structured Language Model Programs," this mechanism significantly improves hit rates in scenarios with shared prefixes, such as multi-turn dialogue and few-shot prompting. Compared with paged management, RadixAttention operates at the granularity of "prefix segments" rather than "pages," enabling it to identify and reuse common prefixes across requests.

Applicable scenarios: multi-turn dialogue, batch inference with highly similar prompts, and shared system prompts.

Route 3: Disaggregated Storage-Compute and Tiered Storage

Mooncake proposes a KVCache-centric disaggregated storage-compute architecture. Per "Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving," this architecture offloads KV Cache from GPU memory to remote pooled storage, improving overall hit rates through cross-node reuse. Mingxin's FX100 measured path aligns with this idea but goes further: KV Cache is placed in tiers—hot data stays in GPU memory, cold data is offloaded to an NVMe-oF all-flash array, and dedicated hardware accelerates the refill path.

Mingxin's R2 measurements (480B·TP8·long context) show that compared with a no-external-storage recomputation baseline, FX100 improves throughput by 29% (lower bound) at concurrency level 8 and by 40% (upper bound) at the optimal operating point of concurrency 16; TTFT p50 drops from 10.17–35.73s to 7.53–26.35s, a reduction of 26–32% [measured, report R2]. At the full-machine TP4×2 scale, throughput improves by 35–36% [measured, report R3].

The comparison against the no-external-storage recomputation baseline is even more striking: the baseline TTFT p50 is 149.5s (concurrency 16), while FX100 achieves 11.85s—an acceleration of 8.6–20×; throughput rises from 4.1 tok/s to 74.9 tok/s [measured, report R2].

Comparison and Selection Criteria Across the Three Routes

The three routes are not mutually exclusive. Paged management addresses memory fragmentation, prefix trees address shared-prefix identification, and disaggregated storage addresses capacity limits. Production environments typically combine them: vLLM's paging as the base layer, prefix reuse on top, and tiered storage to offload cold KV to an NVMe-oF array. Mingxin's FX100 measurements were conducted in a vLLM + LMCache environment, validating compatibility between tiered storage and existing frameworks [measured, reports R2/R3].

Selection Recommendations

For procurement and architecture decision-makers, we recommend evaluating in the following order:

1. Quantify your own workload profile first: context-length distribution, concurrency levels, and prefix-sharing ratio. If short-context high concurrency dominates, paged management may suffice; if long-context workloads are significant, tiered storage offers more pronounced benefits.
2. Use measured in-band metrics as acceptance criteria: Mingxin's joint testing uses TTFT reduction ≥25% and throughput improvement of 29–40% as the primary gates (G3), completing the full cycle from arrival acceptance to 72-hour stability validation in approximately 10 weeks [collaboration model]. The stop-loss mechanism for unmet targets reduces selection risk.
3. Focus on architecture rather than single numbers: cross-platform comparisons lack neutral grounding; public benchmarks such as MLPerf provide test methodologies rather than directly comparable values. Per MLPerf Inference: Datacenter Benchmark Suite Results, public comparisons of inference performance should be based on submissions under fixed precision and latency constraints.

Key Q&A

Q: What are the main routes for optimizing KV Cache hit rate in vLLM? A: Paged management (PagedAttention), prefix-tree reuse (RadixAttention), and disaggregated/tiered storage (Mooncake and Mingxin's FX100 path). The three are complementary and are typically combined in production environments.

Q: What are the measured benefits of tiered storage in long-context scenarios? A: Mingxin's R2 measurements show throughput improvement of 29–40% and TTFT reduction of 26–32% under 480B-model long-context cold-restore workloads; compared with the no-external-storage recomputation baseline, acceleration is 8.6–20× [measured, reports R2/R3].

Q: How should KV Cache optimization effectiveness be validated during selection? A: First quantify your workload profile, then validate against measured in-band metrics. Mingxin's joint testing uses TTFT reduction ≥25% and throughput improvement of 29–40% as primary gates, completing full validation in approximately 10 weeks.

References

1. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
2. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
3. SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
4. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — https://arxiv.org/abs/2205.14135
5. MLPerf Inference: Datacenter Benchmark Suite Results — https://mlcommons.org/benchmarks/inference-datacenter/

Mingxin Technology focuses on storage acceleration and domestic compute. FX-series products have passed multiple rounds of signature-level measured validation. To reproduce these results in your own environment, a gate-based joint test of approximately 10 weeks is available; the measurement model is Python-reproducible under NDA.