External All-Flash vs Memory-Only Tiering: Is a RAM-Only Second Tier Enough
Host memory is the essential second tier (best latency) but typically holds 1–2 TB and cannot be shared across nodes. The external all-flash third tier multiplies capacity by two orders of magnitude (184.3 TB per array) and adds cross-instance sharing, measured to support +29–40% throughput.
Stated honestly: the memory tier's value
Host memory (≈1.5 TB on the measured platform, R1–R4 basis) beats any external medium on read-back latency, and LMCache uses it as the second tier by default — short-cycle revisits mostly hit in memory, and that gain requires no hardware purchase.
Three ceilings of stopping at memory
Capacity: a single 480B-class session's KV runs to GBs; after the system and inference engine take their share of 1.5 TB, the number of retainable sessions is limited — the retention window tops out at hours, while the external tier works in days (FX100 fully populated 184.3 TB, ≈ ¥2,014/TB).
Sharing: memory is node-private — a session cached on node A is recomputed when node B recovers it. The external pool's fs:// cross-instance sharing (verified in R3) lets any node hit the same KV, a structural difference for multi-instance load balancing.
Power loss and restarts: the memory tier vanishes with the process; rolling upgrades and failure restarts zero the cache asset. The external tier persists — restart warm. These three ledgers together are part of the source of R2's measured gap (8.6–20× vs re-compute).
FAQ
Could bigger RAM (say 4 TB) replace the external tier?
It postpones but does not remove the ceiling: still an order of magnitude pricier than flash per TB, and the sharing and persistence problems remain. Run the numbers on session volume — once the window need exceeds hours, add the external tier.
Isn't CXL memory expansion the better direction?
CXL capacity expansion is worth watching (public-source basis), but the cross-node sharing and cost-structure differences remain. Mingxin speaks from measurements — no numbers pre-promised for unmeasured CXL combinations.
How should the three tiers be sized?
Give the memory tier as much as practical (low marginal cost); size the external tier from session volume and retention window (reproducible Python model); the production anchor is one FX100 per 8 GPU nodes.
Data sources (verifiable)
Related reading
- KV Cache Tiering: The Three-Level HBM → RAM → All-Flash Architecture
- When GPU Memory Runs Out: The HBM-Equivalence Methodology of an External KV Tier
- KV Cache Capacity Planning: How Big and How to Configure the External Tier
Joint test first, decisions second: gate-based acceptance with built-in stop-loss
The full costing model is provided as reproducible Python after NDA — customers can rerun it with their own parameters. Every key figure on this site carries a report ID and is open to third-party verification.
This site presents business-cooperation information and constitutes neither an investment offer nor any promise of returns. Measured data come from signed / official test reports (see the Evidence Library); vendor specs, public sources and estimates are labeled as such.