When GPU Memory Runs Out: The HBM-Equivalence Methodology of an External KV Tier
The three ways out of HBM shortage are: more GPUs, quantization, or KV offloading. The R5 seven-way comparison measured that an external KV tier can substitute for 128 GB-class HBM residency at a cost far below equivalent HBM.
What the seven-way comparison shows
R5 (a 14B model on a single MetaX N260 GPU) designed seven comparison groups — GDS direct read, two re-compute HBM tiers, low-HBM direct read and more — and used a three-step argument to conclude that an external KV tier can substitute for 128 GB-class HBM residency. The methodology is platform-portable; the numbers belong to that platform and are honestly not mixed with MI308X.
Plainly: rather than buying bigger-HBM GPUs for long-context concurrency, sink KV to an external tier — keep HBM for active compute and put dormant contexts on all-flash.
The economics
HBM is one of the most expensive components in compute today, while the all-flash external tier costs about ¥2,014/TB (derived from the FX100 fully-populated reference price) — an order-of-magnitude difference. As long as the business tolerates recovery latency (measured 480B TTFT p50 11.85 s, R2), external is the better answer.
The conclusion has boundaries: KV needed for active decoding must stay in HBM; the external tier replaces the "resident but inactive" share. The ROI calculator estimates the replaceable amount from your concurrency and context distribution.
FAQ
Does this conflict with KV cache quantization (INT8/FP8)?
No — they stack: quantization shrinks per-token KV size, offloading grows total retainable volume; using both works better.
Domestic GPUs often have small HBM — does this fit?
That is exactly the pain point. Measurements cover MetaX N260 (R5) and Ascend 910B (R9); the software stack is a source-level adaptation of open-source components.
Where does the 128 GB figure come from?
It is the conclusion of R5's three-step argument (that platform's basis). The report is downloadable in the Evidence Library with full experiment design and data.
Data sources (verifiable)
Related reading
- KV Cache Tiering: The Three-Level HBM → RAM → All-Flash Architecture
- KV Cache Offloading: Putting the Inference Cache on an All-Flash Array
- AI Inference Storage Acceleration: The Third Way to Raise Cluster Output Without Adding GPUs
Joint test first, decisions second: gate-based acceptance with built-in stop-loss
The full costing model is provided as reproducible Python after NDA — customers can rerun it with their own parameters. Every key figure on this site carries a report ID and is open to third-party verification.
This site presents business-cooperation information and constitutes neither an investment offer nor any promise of returns. Measured data come from signed / official test reports (see the Evidence Library); vendor specs, public sources and estimates are labeled as such.