KV Cache Offloading: Putting the Inference Cache on an All-Flash Array

Direct answer

KV cache offloading moves the key-value cache produced during LLM inference from GPU memory down to an external all-flash array, so a cache hit skips re-computation. Mingxin FX100 measured a 29–40% throughput gain on a 480B model in production form.

Why move the KV cache to external storage

During inference every token's attention key-value (KV) pairs are cached; the longer the context and the higher the concurrency, the faster GPU memory runs out. Once an evicted KV block is needed again, the whole prefill must be recomputed — that is the root cause of TTFT blow-ups in long-context workloads.

External KV caching adds an all-flash tier beyond GPU and host memory: when a session returns, its KV is read back from the array by prefix hash instead of being recomputed. Under a long-context cold-recovery load on Qwen3-Coder-480B-FP8 (TP=8 production deployment form), Mingxin FX100 was 8.6–20× faster than the no-external-storage re-compute baseline, cutting TTFT p50 from 149.5 s to 11.85 s (measured, R2, concurrency 16).

How big the gain is: 480B production-form measurements

Compared with the common practice of storing KV on a local NVMe drive, FX100 (NVMe-oF, 4-drive RAID0, 100 GbE) posted lower TTFT p50 at all three concurrency levels (8/16/32: 7.53 s vs 10.17 s, 11.85 s vs 17.31 s, 26.35 s vs 35.73 s) and a 29–40% throughput gain (measured, R2/R3; two independent runs within 5% deviation).

In procurement language: storage spend within roughly 10% of the GPU budget (joint-test basis) buys nearly a third more verifiable effective capacity — more auditable output for the same GPU spend.

Engineering essentials

The software stack is open source — vLLM plus LMCache. Mingxin did source-level adaptation on the ROCm platform and contributed a parallel-read patch: single-GPU, concurrency-16 cold reads saw TTFT drop from 37.97 s to 9.30 s (4.1×, measured, R1). The patch and reproduction scripts ship as a code export package for third-party verification (R8).

Cross-instance sharing is a benefit unique to the external tier: the fs:// shared pool lets multiple inference instances hot-share the same KV, verified in R3. GPU memory cannot be shared across machines — a structural advantage of the external tier.

FAQ

Isn't offloading KV cache to SSD too slow?

The key is read bandwidth and parallelism. FX100 measured lower TTFT than a local drive by 26–32% under cold-recovery load and 8.6–20× faster than re-compute (R2). Reading back is always cheaper than recomputing the whole prefill.

Do I need to modify the inference engine?

No rewrite needed. The open-source vLLM + LMCache combination connects to the external tier; Mingxin provides ROCm build adaptation and patches, delivered with reproduction scripts (R1/R8).

Which workloads benefit most?

Long-context, multi-turn, high-concurrency cold recovery (agents, code assistants, customer service) benefit most. Short low-concurrency chats have low hit rates and limited gains — we honestly recommend testing before buying.