Storage Acceleration for AMD MI308X Inference Clusters

Direct answer

An MI308X (192 GB HBM per GPU) inference cluster with external KV tiering: ROCm 7.2 + vLLM + LMCache + FX100 measured throughput +29–40% and TTFT −26–32% on a 480B model (signed reports R1–R4).

Why storage deserves serious work on MI308X

MI308X's 192 GB HBM per GPU is its relative advantage, yet with a 480B-class MoE model (weights ≈450 GB) plus long-context concurrency, HBM still runs tight. The ROCm ecosystem has fewer storage-acceleration options than CUDA — exactly why Mingxin invested in source-level adaptation.

The complete measured stack: 8× MI308X (gfx942), 2× EPYC 9654, ROCm 7.2, vLLM 0.20.1+rocm721, LMCache upstream mainline build, FX100 NVMe-oF array — hardware and software versions published in full (R1–R4).

The results obtained

Qwen3-Coder-480B-FP8 in production deployment form (TP=8): throughput +29–40%, TTFT p50 down 26–32%, 8.6–20× vs no-external-storage re-compute (R2/R3). Multimodal: all 7 ComfyUI + LTX-Video 2.3 models ran end-to-end on MI308X with no dtype/operator/kernel errors (R6/R7).

The LMCache parallel-read patch (TTFT 4.1×, R1) was validated on ROCm and contributed back to the community — MI308X users benefit directly.

FAQ

Where can I find measured data for 480B models on MI308X?

Signed reports R2/R3/R4 (Qwen3-Coder-480B-FP8, TP8 and TP4×2 forms, nine load levels with full metrics) — PDFs downloadable in the Evidence Library.

Is LMCache easy to install on ROCm?

It needs a source build and several fixes; Mingxin supplies the pre-built adaptation layer, patches and deployment scripts, delivered with the joint test (R8 export package).

Are newer GPUs like MI350/MI355 supported?

The stack builds on ROCm's general interfaces and gfx architecture upgrades are expected to be compatible; new platforms are backfilled by measurements — we do not pre-promise unmeasured numbers.