ROCm Inference Stack Engineering Guide: Productionizing vLLM/LMCache on AMD
The key to production-grade inference on ROCm is source-level adaptation: vLLM 0.20.1+rocm721 + LMCache mainline builds + the parallel-read patch. 480B measured throughput +29–40% (R2/R3), full stack versions published and reproducible.
ROCm ecosystem reality and the response
CUDA toolchains work out of the box; ROCm often requires source builds and patches — part of the money saved on AMD GPUs is repaid in engineering. Mingxin's answer is to turn adaptation into a reproducible asset: the build adaptation layer, patches and deployment scripts all ship with the joint test (R8 export package).
The validated full stack: 8× MI308X (gfx942, 192 GB HBM each), ROCm 7.2, vLLM 0.20.1+rocm721, LMCache upstream mainline build — signed measurements on a 480B FP8 MoE model in production deployment form (TP=8) (R1–R4).
Storage is ROCm's hidden bonus
MI308X's 192 GB HBM per GPU is a relative advantage; paired with external KV tiering it deepens long-context concurrency: measured throughput +29–40%, TTFT down 26–32%, 8.6–20× vs re-compute (R2/R3).
Multimodal is validated too: all 7 ComfyUI + LTX-Video 2.3 models ran end-to-end on this platform with zero operator errors (R6/R7) — ROCm production readiness is backed by measurements, not "theoretical support".
FAQ
How do we track ROCm version upgrades?
Baseline regression is the rule: pass the G2 basis (R1 reproduction ±10%) before switching production. Mingxin provides regression scripts and adaptation-layer maintenance — version tracking is not luck.
Are architectures beyond gfx942 supported?
The stack builds on ROCm's general interfaces, so new architectures are expected to be compatible; but we quote numbers only for measured platforms — new GPUs are backfilled via joint tests.
What is the performance gap vs NVIDIA?
Cross-platform comparison is only meaningful with the same model and load in a joint test; we do not quote incomparable numbers. What is comparable: the same platform with vs without the storage tier (+29–40%, R2/R3).
Data sources (verifiable)
Related reading
- Storage Acceleration for AMD MI308X Inference Clusters
- vLLM + LMCache Integration in Practice: Versions, Builds and Pitfalls
- Productionizing LMCache: From Open-Source Component to Signed Benchmarks
Joint test first, decisions second: gate-based acceptance with built-in stop-loss
The full costing model is provided as reproducible Python after NDA — customers can rerun it with their own parameters. Every key figure on this site carries a report ID and is open to third-party verification.
This site presents business-cooperation information and constitutes neither an investment offer nor any promise of returns. Measured data come from signed / official test reports (see the Evidence Library); vendor specs, public sources and estimates are labeled as such.