Replacing KV Cache GPU Memory: How an External Storage Layer Frees 128GB-Class GPU Memory Usage

Introduction

The GPU memory bottleneck for large model inference is shifting from "model weights" to "KV Cache." Taking a 480B MoE model as an example, a single long-context inference can require hundreds of GB of KV Cache, far exceeding the memory capacity of a single GPU. The traditional approach relies on GPU memory residency, which either limits concurrency or sacrifices context length. A frequently discussed but rarely implemented idea is to use an external storage layer to replace part of the GPU memory residency, offloading the KV Cache to high-speed NVMe-oF storage and loading it only when needed. The core challenge of this approach is whether storage latency can meet the real-time requirements of inference. Measured data from Mingxin FX100 on the AMD MI308X platform provides quantifiable answers.

Feasibility of Replacing GPU Memory with External Storage: From Latency to Bandwidth

The core contradiction of KV Cache offloading lies in the latency gap: GPU memory latency is approximately 100-300ns, while NVMe SSD latency is about 10-100μs, a difference of two orders of magnitude. However, modern inference frameworks (such as vLLM and LMCache) can significantly mask this gap through prefetching, hierarchical caching, and parallel read optimization. In the R2 test, Mingxin FX100, using the LMCache parallel read patch, reduced the TTFT in cold read scenarios from 37.97 seconds to 9.30 seconds (a 4.1x improvement), and increased bandwidth from 0.98 GB/s to 5.23 GB/s (a 5.3x improvement) [measured, report R1]. This means that when the KV Cache is offloaded to external storage, the latency of a single load has shifted from "unacceptable" to "tolerable."

For replacing 128GB-class GPU memory residency, the key lies in "equivalent capacity." Taking the 480B model as an example, a single inference requires approximately 450GB of KV Cache (FP8 weights are about 450GB, but KV Cache grows dynamically with sequence length). In a TP8 configuration, the total memory of 8 GPUs is about 1.5TB (192GB HBM per GPU), but in the available memory, over 30% is occupied by KV Cache. By offloading to external storage, the GPU memory usage of KV Cache can be reduced to near zero, freeing up memory for higher concurrency or larger models. In measured tests, with a concurrency of 16, FX100 reduced TTFT from 149.5 seconds (baseline without external storage recomputation) to 11.85 seconds (an acceleration of 8.6-20x), and increased throughput from 4.1 tok/s to 74.9 tok/s [measured, report R2]. This demonstrates that external storage can not only replace GPU memory but also maintain or even improve inference performance after replacement.

Measured Data: Quantified Boundaries of TTFT Reduction and Throughput Improvement

In the R2/R3 tests, Mingxin FX100, configured with 480B·TP8, measured clear performance boundaries for long-context cold recovery workloads. The TTFT p50 decreased from 10.17-35.73 seconds to 7.53-26.35 seconds, a reduction of 26-32% [measured, report R2]. This reduction is not linear: at low concurrency (8), the TTFT reduction is near the lower bound (26%); at high concurrency (16), it reaches the upper bound (32%). The reason is that at higher concurrency, GPU memory pressure is greater, making the offloading effect of external storage more significant.

In terms of throughput, KV hierarchical acceleration improved inference throughput by 29-40% [measured, reports R2/R3]. Similarly, the lower bound occurred at concurrency 8 (+29%), and the upper bound at concurrency 16 (+40%). At the TP4×2 full-machine level, throughput improved by 35-36% [measured, report R3]. These numbers indicate that the effect of replacing GPU memory with external storage is positively correlated with concurrency: the higher the concurrency, the greater the memory pressure, and the more pronounced the benefit of offloading.

Notably, these data were obtained under "no external storage recomputation" conditions. In traditional approaches, if the KV Cache is evicted, it must be recomputed from model weights, resulting in a TTFT of up to 149.5 seconds (concurrency 16) [measured, report R2]. By directly loading from external storage, FX100 avoids recomputation, reducing TTFT to 11.85 seconds, an acceleration ratio of 12.6x. This validates the feasibility of using external storage as a "second-level GPU memory."

Architecture Design: Collaborative Path of LMCache and vLLM

Replacing GPU memory with external storage is not simply a matter of mounting NVMe drives; it requires deep integration with the inference framework. Mingxin FX100's measurements are based on a combination of LMCache (compiled from upstream mainline source code dated 2026-06-29) and vLLM 0.20.1+rocm721 [measured platforms, reports R1-R4]. The LMCache parallel read patch is key: it changes single-threaded disk reads to multi-threaded concurrency, increasing bandwidth from 0.98 GB/s to 5.23 GB/s (a 5.3x improvement) [measured, report R1]. This optimization directly reduces TTFT.

In training scenarios, external storage is also effective. In 8-card 32B LoRA training, checkpoint saving was reduced from 178 seconds to 94 seconds (a 1.9x acceleration), and sustained write bandwidth increased from 3.26 GB/s to 6.40 GB/s (+96%) [measured, report R1]. This shows that external storage is not only suitable for inference but can also accelerate persistence operations during training.

For computing center decision-makers, the key question is: Is the investment in external storage worthwhile? The reference price for Mingxin FX100 is approximately ¥2,014/TB (FX100 fully configured system) [FX product line], while the cost of equivalent GPU memory (based on HBM3e at approximately $20/GB) is about ¥14,000/TB. The cost of external storage is only about 14% of that of GPU memory. Considering that measured KV Cache offloading can free up 128GB-class GPU memory, with TTFT reduced by 26-32% and throughput improved by 29-40%, the ROI is positive in most production scenarios.

Conclusion

Replacing GPU memory residency with external storage is not a theoretical fantasy. Measured data from Mingxin FX100 shows that, through LMCache parallel read optimization and vLLM framework collaboration, NVMe-oF storage can replace 128GB-class GPU memory usage while reducing TTFT by 26-32% and improving throughput by 29-40%. The cost of this solution is only about 14% of equivalent GPU memory, and the benefits are more significant in high-concurrency scenarios. Mingxin offers approximately 10 weeks of gated joint testing (from G1 arrival acceptance to G3 main gate: measured in-band TTFT reduction ≥25%, throughput +29-40%), with a stop-loss mechanism if targets are not met. Computing center teams are welcome to contact us for NDA-based Python reproducible evaluations tailored to your models and workloads.