Productionizing LMCache: From Open-Source Component to Signed Benchmarks
LMCache is the open-source KV cache tiering framework of the vLLM ecosystem. Mingxin completed source-level ROCm build adaptation and contributed a parallel-read patch, measuring a 4.1× TTFT improvement, with code reproducible via an export package.
What separates open source from production
LMCache upstream targets the CUDA ecosystem first; landing it on ROCm / domestic platforms requires source builds and fixes. Mingxin built from upstream mainline (2026-06-29) on ROCm 7.2 (vLLM 0.20.1+rocm721) and isolated an fs:// backend defect along the way (recorded in R1/R8).
The performance-critical finding: default serial disk reads cannot saturate array bandwidth. The parallel-read patch lifted single-GPU concurrency-16 cold-read bandwidth from 0.98 GB/s to 5.23 GB/s (5.3×) and cut TTFT from 37.97 s to 9.30 s (4.1×, R1).
Reproducibility first
The patch ships as a git patch plus full before/after sources, packaged with load clients, orchestration scripts, forensic probes and raw data as a code export (R8) — any third party can independently reproduce every conclusion. This is how Mingxin differs from "slideware performance".
System-level validation on the 480B production form is in R2/R3: throughput +29–40%, TTFT down 26–32%, two independent runs within 5% deviation.
FAQ
Will the patch be contributed upstream?
Yes. The parallel-read patch goes back to the LMCache community as a PR / standalone repository, while the export package remains directly available (see the GitHub organization page for the open-source plan).
What about vLLM version upgrades?
The patch targets the LMCache storage-backend interface and is weakly coupled to vLLM versions; Mingxin provides adaptation service and regression-test scripts.
Can I use only open source without buying hardware?
Yes — LMCache is open source and runs on local drives too. The array's incremental value is pooled capacity, cross-instance sharing and higher bandwidth (see the R2 local-drive control group for the measured gap).
Data sources (verifiable)
Related reading
- KV Cache Offloading: Putting the Inference Cache on an All-Flash Array
- TTFT Optimization: Engineering Paths to Cut LLM First-Token Latency
- NVMe-oF Inference Storage: Measured Results of a RoCEv2 All-Flash Array
Joint test first, decisions second: gate-based acceptance with built-in stop-loss
The full costing model is provided as reproducible Python after NDA — customers can rerun it with their own parameters. Every key figure on this site carries a report ID and is open to third-party verification.
This site presents business-cooperation information and constitutes neither an investment offer nor any promise of returns. Measured data come from signed / official test reports (see the Evidence Library); vendor specs, public sources and estimates are labeled as such.