KV Cache Tiering: The Three-Level HBM → RAM → All-Flash Architecture
KV cache tiering places the inference cache on GPU memory, host memory and external all-flash by temperature, with capacity growing and cost falling at each level. Mingxin FX100 provides the NVMe-oF all-flash pool for the third tier.
What each tier does
The GPU-memory tier (HBM) holds KV for currently active sessions — fastest, smallest. Host memory takes recent sessions; the external all-flash tier absorbs the full session history plus cross-instance sharing, with capacity in the hundreds of TB. Open-source frameworks such as LMCache manage blocks by token-prefix hash and refill tier by tier on a hit.
The value of tiering is not replacing HBM but turning "evicted means recompute" into "evicted but recoverable". A single long-context session on a 480B model produces KV measured in GB — only an external tier can keep the hit rate meaningful.
Measured anchors
R2/R3 (Qwen3-Coder-480B-FP8, 8× MI308X): tiering lifts throughput +29–40%, cuts TTFT p50 by 26–32%, and beats no-external-storage re-compute by 8.6–20×.
R5 (a 14B model, seven-way comparison on the MetaX N260 platform) establishes the HBM-efficiency methodology: an external KV tier can substitute for ~128 GB-class HBM residency. The methodology is platform-portable; the numbers belong to that platform and are not mixed with MI308X (labeled as such).
Sizing capacity and ratios
The production ratio anchor is one FX100 serving 8 GPU nodes (100 GbE sustains 8-node KV read-back, measured). Actual capacity depends on session-length distribution and the retention window. Mingxin provides a reproducible Python sizing model — customers can rerun it with their own parameters after NDA. The on-site ROI calculator gives a quick first estimate.
FAQ
How does tiering relate to vLLM's built-in prefix caching?
Prefix caching is the first tier, inside GPU memory. Tiering extends it to host memory and external flash, multiplying capacity by orders of magnitude and adding cross-instance sharing (verified in R3).
What hit rate can I expect?
It depends on your session-revisit distribution — there is no universal number. Our recommendation: run a joint test with your own traces, with gate-based acceptance (TTFT reduction ≥25% to pass) and stop-loss if the gate is missed.
How is consistency kept across the three tiers?
KV blocks are content-addressed by prefix hash and thus naturally idempotent; LMCache handles inter-tier migration and invalidation while FX100 provides persistence and sharing at the block-storage layer.
Data sources (verifiable)
Related reading
- KV Cache Offloading: Putting the Inference Cache on an All-Flash Array
- When GPU Memory Runs Out: The HBM-Equivalence Methodology of an External KV Tier
- AI Inference Storage Acceleration: The Third Way to Raise Cluster Output Without Adding GPUs
Joint test first, decisions second: gate-based acceptance with built-in stop-loss
The full costing model is provided as reproducible Python after NDA — customers can rerun it with their own parameters. Every key figure on this site carries a report ID and is open to third-party verification.
This site presents business-cooperation information and constitutes neither an investment offer nor any promise of returns. Measured data come from signed / official test reports (see the Evidence Library); vendor specs, public sources and estimates are labeled as such.