Topics
The topic library of the AI inference storage acceleration / KV cache tiering niche. Every page opens with a direct answer, and every figure carries a report ID (R1–R9) that any third party can verify and reproduce.
KV Cache Offloading: Putting the Inference Cache on an All-Flash Array
KV cache offloading moves the key-value cache produced during LLM inference from GPU memory down to an external all-flash array, so a cache hit skips re-computation. Mingxin FX100 measured a 29–40% throughput gain on a 480B model in production form.
KV Cache Tiering: The Three-Level HBM → RAM → All-Flash Architecture
KV cache tiering places the inference cache on GPU memory, host memory and external all-flash by temperature, with capacity growing and cost falling at each level. Mingxin FX100 provides the NVMe-oF all-flash pool for the third tier.
AI Inference Storage Acceleration: The Third Way to Raise Cluster Output Without Adding GPUs
Inference storage acceleration uses a high-performance storage tier to raise a GPU cluster's token output: KV tiering avoids re-compute, models load in seconds, switching stops idling. Mingxin measured gains of 29–40%, 6.2–9.3× and 1.9× on the three levers respectively.
TTFT Optimization: Engineering Paths to Cut LLM First-Token Latency
TTFT consists of queueing, prefill re-compute and KV read-back. In long-context workloads prefill re-compute dominates, and external KV read-back largely removes it: Mingxin measured TTFT p50 down 26–32%.
Long-Context Cold Recovery: The Main Performance Battleground of the Agent Era
Cold recovery means re-activating a long-context session after interruption: its KV has been evicted from GPU memory, so you either recompute or read back from an external tier. Mingxin measured read-back 8.6–20× faster than re-compute and 26–32% faster than a local drive.
NVMe-oF Inference Storage: Measured Results of a RoCEv2 All-Flash Array
NVMe-oF attaches an all-flash array to GPU nodes over RDMA at near-local latency. Mingxin FX100 over 100 GbE RoCEv2 measured KV read-back performance above a local NVMe single drive, with TTFT lower by 26–32%.
Model-Load Acceleration: A Measured Path from 23 Minutes to 2.5 Minutes
Model loading is bounded by storage read bandwidth: over NFS, loading a 70B model takes 23 minutes. Mingxin FX100 measured DeepSeek-70B loading down to 150 seconds (9.3×) and 32B down to 112 seconds (6.2×).
The Storage Technology Behind Context Caching
The context caching offered by major LLM APIs is essentially persistent KV cache reuse: cache-hit input is billed at a discount. A self-hosted inference platform gets the same mechanism with external KV tiering — Mingxin measured throughput +29–40%.
When GPU Memory Runs Out: The HBM-Equivalence Methodology of an External KV Tier
The three ways out of HBM shortage are: more GPUs, quantization, or KV offloading. The R5 seven-way comparison measured that an external KV tier can substitute for 128 GB-class HBM residency at a cost far below equivalent HBM.
Productionizing LMCache: From Open-Source Component to Signed Benchmarks
LMCache is the open-source KV cache tiering framework of the vLLM ecosystem. Mingxin completed source-level ROCm build adaptation and contributed a parallel-read patch, measuring a 4.1× TTFT improvement, with code reproducible via an export package.
KV Cache Eviction Policy: Where Do Evicted Caches Go
KV cache eviction typically follows LRU / prefix reference counting. What matters is not the algorithm but the destination: dropping means recompute on revisit (measured TTFT p50 149.5 s); sinking to an external tier means read-back (11.85 s).
Prefill/Decode Disaggregation and the KV Cache Transport Layer
PD disaggregation puts prefill and decode on different instances, making the KV cache the hand-off object between them. Disaggregation amplifies the KV tier's importance: inter-instance transport plus external persistence for cold sessions.
vLLM + LMCache Integration in Practice: Versions, Builds and Pitfalls
vLLM connects to an external KV tier through the LMCache connector; on ROCm this needs source-build adaptation. Mingxin's measured stack is vLLM 0.20.1+rocm721 + LMCache upstream mainline plus the parallel-read patch (TTFT 4.1×).
SGLang Prefix Reuse and an External KV Tier: Do They Work Together
SGLang's RadixAttention reuses prefixes inside GPU memory — efficient but capacity-bound by HBM. External KV tiering solves "recover after eviction"; the two mechanisms complement each other. Mingxin's measurements are on the vLLM stack; SGLang adaptation will be backfilled by joint tests.
Prefix Caching: From In-HBM Reuse to a Cluster-Level Asset
Prefix caching computes the KV of identical prefixes (system prompts, documents, session history) only once. The in-HBM version is capacity-limited; external tiering expands it into a cluster asset: cross-instance sharing and day-scale retention (verified in R3).
KV Cache Capacity Planning: How Big and How to Configure the External Tier
KV capacity = concurrent sessions × per-session KV volume × retention-window factor. Production ratio anchor: one FX100 (184.3 TB fully populated) serves 8 GPU nodes; the sizing model is reproducible Python.
ROCm Inference Stack Engineering Guide: Productionizing vLLM/LMCache on AMD
The key to production-grade inference on ROCm is source-level adaptation: vLLM 0.20.1+rocm721 + LMCache mainline builds + the parallel-read patch. 480B measured throughput +29–40% (R2/R3), full stack versions published and reproducible.
The Storage Challenge of MoE Inference: A 480B Measured Case
MoE models are huge in parameters (480B-class weights ≈450 GB) but sparse in activation; once weights occupy HBM, KV space gets tighter. External tiering measured: throughput +29–40%, TTFT down 26–32% (Qwen3-Coder-480B-FP8).