KV Cache Capacity Planning: How Big and How to Configure the External Tier
KV capacity = concurrent sessions × per-session KV volume × retention-window factor. Production ratio anchor: one FX100 (184.3 TB fully populated) serves 8 GPU nodes; the sizing model is reproducible Python.
Three input variables
Per-session KV volume: set by model architecture and context length — a single long session on a 480B-class MoE model runs to GBs (R2 load-design basis); quantization (FP8 KV) compresses proportionally.
Session volume and retention window: daily active sessions × average revisit interval determine how long to retain. The external tier costs about ¥2,014/TB (FX100 fully-populated basis), so widening the window to days carries low marginal cost — the tier's order-of-magnitude advantage over HBM.
Ratio anchors and sizing tools
Bandwidth-side anchor: a single 100 GbE port sustains KV read-back for 8 GPU nodes (derived from R2/R3 measured bandwidth); provision at 1:8 in production. For higher bandwidth needs, step up to FX200 (200 Gb) / FX300 (400 Gb) tiers.
Capacity is not guesswork: Mingxin provides a reproducible Python sizing model (source delivered after NDA; rerun with your parameters), and the on-site ROI calculator gives a quick range from GPU count and cluster size.
FAQ
What happens if I under-provision?
The external tier evicts more often, hit rates fall and gains degrade toward the re-compute baseline — the system still works, just with reduced acceleration. Capacity expands online; starting with a small joint test then scaling is the standard path.
Which RAID level?
The measured form is 4-drive RAID0 (KV is recomputable — the loss cost is just a cold start). For strong retention requirements, trade capacity for a redundant level.
How do I size for mixed multi-model deployment?
Size each model's KV independently and sum; the weight pool and KV pool can share one array in separate partitions (model-load acceleration anchor 6.2–9.3×, R9).
Data sources (verifiable)
Related reading
- KV Cache Tiering: The Three-Level HBM → RAM → All-Flash Architecture
- Inference Storage Planning for a Thousand-GPU AI Datacenter
- When GPU Memory Runs Out: The HBM-Equivalence Methodology of an External KV Tier
Joint test first, decisions second: gate-based acceptance with built-in stop-loss
The full costing model is provided as reproducible Python after NDA — customers can rerun it with their own parameters. Every key figure on this site carries a report ID and is open to third-party verification.
This site presents business-cooperation information and constitutes neither an investment offer nor any promise of returns. Measured data come from signed / official test reports (see the Evidence Library); vendor specs, public sources and estimates are labeled as such.