The Storage Challenge of MoE Inference: A 480B Measured Case
MoE models are huge in parameters (480B-class weights ≈450 GB) but sparse in activation; once weights occupy HBM, KV space gets tighter. External tiering measured: throughput +29–40%, TTFT down 26–32% (Qwen3-Coder-480B-FP8).
MoE amplifies storage pressure
MoE trades sparse activation for parameter scale: Qwen3-Coder-480B-FP8 weights are ≈450 GB. After 8× MI308X (1536 GB HBM total) hold the weights, the HBM share left for KV is markedly smaller than for an equal-HBM dense model — under long-context concurrency, KV eviction starts earlier and happens more often.
That is exactly why 480B was chosen as the test load: it represents the real pressure of the MoE era. Results: external KV tiering delivered throughput +29–40%, TTFT p50 down 26–32%, and 8.6–20× vs no-external-storage re-compute (R2/R3, TP=8 production form).
Weight loading is an MoE pain point too
Loading 450 GB of weights from slow storage takes tens of minutes (NFS anchor: a 70B model at 1399 s, R9), and MoE's larger weights scale the problem up — restarts, elastic scale-out and failure migration all wait on storage.
The external all-flash pool plays a double role: second-scale weight distribution (6.2–9.3× anchor, R9) plus KV tiering (R2/R3) — one investment, two returns. The TP4×2 dual-instance whole-machine basis measured +35–36% (R3).
FAQ
Can expert weights also sink to the external tier?
Expert offloading is an active research direction, but the decode path is extremely latency-sensitive. Mingxin's current measured boundary is KV and weight loading; we do not pre-promise expert offloading conclusions.
Does FP8 quantization affect KV tiering?
Positively: KV volume halves with quantization, doubling the external tier's effective capacity. The measured load itself is FP8 (Qwen3-Coder-480B-FP8, R2).
Do the conclusions hold for larger (trillion-scale) MoE?
The mechanism holds (bigger weights → tighter KV → higher tiering gains), but numbers do not extrapolate; trillion-scale loads await joint-test measurements.
Data sources (verifiable)
Related reading
- KV Cache Offloading: Putting the Inference Cache on an All-Flash Array
- Model-Load Acceleration: A Measured Path from 23 Minutes to 2.5 Minutes
- When GPU Memory Runs Out: The HBM-Equivalence Methodology of an External KV Tier
Joint test first, decisions second: gate-based acceptance with built-in stop-loss
The full costing model is provided as reproducible Python after NDA — customers can rerun it with their own parameters. Every key figure on this site carries a report ID and is open to third-party verification.
This site presents business-cooperation information and constitutes neither an investment offer nor any promise of returns. Measured data come from signed / official test reports (see the Evidence Library); vendor specs, public sources and estimates are labeled as such.