KV Cache Eviction Policy: Where Do Evicted Caches Go
KV cache eviction typically follows LRU / prefix reference counting. What matters is not the algorithm but the destination: dropping means recompute on revisit (measured TTFT p50 149.5 s); sinking to an external tier means read-back (11.85 s).
Eviction is inevitable; dropping is not
GPU memory is fixed, so once sessions pile up KV must be evicted — vLLM's block manager reclaims KV blocks by reference count and LRU, and the open-source layer does this well. The real architectural fork comes after eviction: drop (recompute the whole prefill on the next visit) or sink (write to RAM / external flash and read back on revisit).
The cost gap between the two paths is measured: on the 480B cold-recovery load at concurrency 16, the drop-and-recompute baseline posted TTFT p50 149.5 s and 4.1 tok/s; sink-and-read-back (FX100 external tier) posted 11.85 s and 74.9 tok/s — an 8.6–20× gap (measured, R2).
Engineering tiered eviction
LMCache turns eviction into inter-tier migration: HBM overflow sinks to host memory, memory overflow sinks to external all-flash; KV blocks are content-addressed by prefix hash and naturally idempotent. The external tier holds hundreds of TB (FX100 fully populated: 184.3 TB raw), stretching the eviction window to days.
Sink-path writes do not block the inference hot path; read-back performance depends on parallelism — LMCache's default serial reads cannot saturate the array, which is why Mingxin's parallel-read patch raised single-GPU cold-read bandwidth from 0.98 GB/s to 5.23 GB/s (measured, R1; patch in the R8 export package).
FAQ
Does the eviction policy need business-side tuning?
Default LRU suffices in most cases; adjust the retention window if your revisit distribution is unusual (e.g. fixed-period batch jobs). We recommend validating with real traces in a joint test — the G3 gate is TTFT reduction ≥25%.
Do sink writes slow down inference?
KV sinking is asynchronous large-block sequential writing overlapped with decoding; the R2/R3 throughput numbers (+29–40%) already include sink overhead — they are end-to-end figures.
What if the external tier fills up too?
The external tier also evicts the coldest KV by LRU — only then is data truly dropped. The difference is the window: minutes for HBM versus days for the external tier, so the vast majority of revisits land inside it.
Data sources (verifiable)
Related reading
- KV Cache Tiering: The Three-Level HBM → RAM → All-Flash Architecture
- KV Cache Offloading: Putting the Inference Cache on an All-Flash Array
- Long-Context Cold Recovery: The Main Performance Battleground of the Agent Era
Joint test first, decisions second: gate-based acceptance with built-in stop-loss
The full costing model is provided as reproducible Python after NDA — customers can rerun it with their own parameters. Every key figure on this site carries a report ID and is open to third-party verification.
This site presents business-cooperation information and constitutes neither an investment offer nor any promise of returns. Measured data come from signed / official test reports (see the Evidence Library); vendor specs, public sources and estimates are labeled as such.