KV Prefix Reuse in 29.8K Token Dialogue: 7.15GB KV失踪

In multi-turn dialogue scenarios, the storage and reuse of KV Cache is key to improving inference efficiency. Taking a 29.8K token session as an example, the approximately 7.15GB of KV data it generates could theoretically avoid over 90% of redundant computation if fully reused. In actual tests, the Mingxin FX100, through LMCache parallel read patching and its all-flash NVMe-oF architecture, reduced cold read disk TTFT by 4.1x and compressed first token latency by 26–32%, validating the engineering feasibility of storage acceleration for long-context interactions.

Principles of KV Cache Storage and Reuse in Multi-turn Dialogues

KV Cache is a critical optimization technique in large model inference: during autoregressive generation, the model needs to repeatedly compute the Key and Value matrices for each token. If the KV values of historical tokens are cached to external storage, new requests can directly reuse them, avoiding redundant computation. Taking a 480B MoE model (Qwen3-Coder-480B-FP8) as an example, the KV data per token is approximately 245 KB (estimated based on 8 attention layers, 16 heads per layer, FP8 quantization). A single 29.8K token session accumulates approximately 7.15 GB of KV data.

In practical deployment, this data is typically stored in GPU memory (HBM), but memory capacity is limited (e.g., AMD MI308X offers 192 GB per card, 1.5 TB total for 8 cards), making it difficult to accommodate the full KV Cache of long dialogues. Therefore, the industry adopts a tiered caching strategy: hot data resides in memory, while cold data is offloaded to SSD or NVMe-oF storage. Tests on the Mingxin FX100 show that through the LMCache framework and NVMe-oF protocol, the latency of reading cold KV data from storage can be effectively controlled, thereby supporting long-context reuse.

Where Did the KV Data from the 29.8K Token Session Go? – An Analysis of Storage Acceleration Tests

In tests on the Mingxin FX100 (measured, report R1), we simulated a multi-turn dialogue scenario with 29.8K tokens. The baseline was a no-external-storage recomputation scheme (i.e., computing all KV from scratch for each request), while the FX100 scheme, via LMCache parallel read patching, stored cold KV data on an all-flash NVMe-oF array (4-drive RAID0, 14 TB, RoCEv2 single-port 100 GbE).

Key Data Flow:

• After generating 29.8K tokens in the session, 7.15 GB of KV data was written to the FX100 array, with a write bandwidth of approximately 3.2 GB/s (single-drive measured value).
• When subsequent requests reused this KV, the read bandwidth from storage increased to 5.23 GB/s (measured, report R1), a 5.3x improvement compared to a local NVMe single drive (0.98 GB/s).
• Cold read disk TTFT decreased from 37.97s to 9.30s, a reduction of 4.1x (measured, report R1).

Why is Storage Acceleration Effective? In traditional schemes, cold KV data is stored on local NVMe drives, but the single-drive bandwidth (approximately 1 GB/s) cannot match the GPU's memory loading demands. The FX100 employs the NVMe-oF protocol to aggregate the bandwidth of a multi-drive RAID0 array onto a 100 GbE network and optimizes data sharding and scheduling using LMCache's parallel read patching. In tests, with a single-card concurrency of 16, the read bandwidth reached 5.23 GB/s, approaching 60% of the network limit (theoretical bandwidth for 100 GbE is approximately 12.5 GB/s, limited by the PCIe 3.0 interface).

Quantifying the Benefits of Storage Acceleration for Multi-turn Dialogues: Throughput and Latency

The core metrics for multi-turn dialogue scenarios are Time to First Token (TTFT) and throughput (tok/s). In tests with a 480B model, TP8, across three concurrency levels (8/16/32) (measured, report R2), the FX100, compared to the no-external-storage recomputation baseline, reduced TTFT by 26–32% (p50 decreased from 10.17–35.73s to 7.53–26.35s) and improved throughput by 29–40% (+29% at concurrency level 8, +40% at concurrency level 16).

Direct Impact on a 29.8K Token Session: Assuming a multi-turn dialogue contains 5 rounds of interaction, with approximately 6K new tokens per round. Under a no-reuse scheme, each round requires recomputing the KV for all 29.8K tokens, with a cumulative TTFT of approximately 35.73s × 5 ≈ 179s. Under the FX100 scheme, only the first round computes the full KV (TTFT approximately 26.35s), and subsequent rounds reuse the 7.15 GB of data from storage, reducing TTFT per round to approximately 9.30s. The total TTFT becomes approximately 26.35s + 9.30s × 4 ≈ 63.5s, a reduction of 64.5%.

Analogy to Training Checkpoints: Similarly, checkpoint saving in training scenarios also benefits from storage acceleration. Tests (measured, report R1) show that for an 8-card 32B LoRA full model snapshot (65.6 GB), the save time decreased from 178s to 94s, with sustained write bandwidth improving by 96%. This further validates the FX100's generality for data-intensive workloads.

Conclusion

Storage reuse of KV Cache in multi-turn dialogues is essentially an engineering trade-off of "exchanging storage bandwidth for computational latency." Through its all-flash NVMe-oF architecture and LMCache optimizations, the Mingxin FX100 transforms the 7.15 GB of KV data from a 29.8K token session from a "computational burden" into a "reusable asset," achieving measured benefits of a 26–32% reduction in TTFT and a 29–40% improvement in throughput on a 480B model. For computing centers deploying long-context interactions, this solution provides a reproducible acceleration path. For details on joint testing, please contact the technical team via the Mingxin website.

Key Q&A

Q: In multi-turn dialogues, how is the 7.15 GB of KV data from a 29.8K token session reused? A: Through the LMCache framework, cold KV data is stored on the Mingxin FX100 all-flash NVMe-oF array. When subsequent requests read from storage, the bandwidth reaches 5.23 GB/s (measured, report R1), and cold read disk TTFT is reduced by 4.1x.

Q: What are the specific improvements in TTFT and throughput for multi-turn dialogues due to storage acceleration? A: On a 480B model with TP8 across three concurrency levels, the FX100, compared to no-external-storage recomputation, reduced TTFT by 26–32% (measured, report R2) and improved throughput by 29–40% (measured, report R2).

Q: Does the Mingxin FX100 offer similar benefits in training scenarios? A: Yes. In 8-card 32B LoRA training, checkpoint save time decreased from 178s to 94s, with sustained write bandwidth improving by 96% (measured, report R1), validating the generality of storage acceleration for both training and inference scenarios.