KV Prefix Reuse in Multi-Turn Conversations: Where Did the 7.15GB KV Cache of a 29.8K Token Session Go?

In multi-turn conversation scenarios, KV Cache prefix reuse is a core technique for improving inference efficiency. Taking a 29.8K token session as an example, the 7.15GB of KV data it generates does not "disappear." Instead, it is systematically retained or released through VRAM management, storage offloading, and reuse strategies to balance VRAM usage with response speed for subsequent requests. Measured results show that in long-context cold recovery workloads, the Mingxin FX100 all-flash storage array can increase inference throughput by 29–40% and reduce time-to-first-token (TTFT) by 26–32%, significantly optimizing the multi-turn conversation experience.

KV Cache Storage and Reuse Mechanism in Multi-Turn Conversations

During inference with Transformer-based large language models, each turn of conversation generates Key and Value matrices for input tokens, known as the KV Cache. In a 29.8K token session, assuming a 480B parameter model at FP8 precision, the KV size per token is approximately 256 KB (depending on model layers and attention heads), totaling about 7.15 GB. Without reuse, each new turn would require recomputation, wasting VRAM and compute resources.

Frameworks like vLLM and LMCache implement KV reuse through prefix caching: the system stores historical KV Cache in VRAM or high-speed storage. When a new request's prompt prefix matches a historical one, the precomputed KV data is loaded directly, skipping redundant computation. This requires the storage system to have low-latency, high-bandwidth read capabilities, especially in cold-start or long-context scenarios where KV data must be offloaded to storage when VRAM is insufficient.

Challenges of Long-Context Cold Recovery and Measured Storage Acceleration

In cold-start scenarios for multi-turn conversations (e.g., after service restart or VRAM clearing), all KV Cache must be reloaded from storage. Here, storage performance directly impacts TTFT and overall throughput. Measured data from the Mingxin FX100 on a 480B model (deployed with TP8) provides quantitative reference [source: measured, report R2]:

• TTFT Improvement: At concurrency level 8, TTFT p50 dropped from 10.17 seconds to 7.53 seconds (a 26% reduction); at concurrency level 16, from 35.73 seconds to 26.35 seconds (also a 26% reduction). This means users can wait about a quarter less time for the first token in multi-turn conversations.
• Throughput Improvement: KV tiered acceleration increased inference throughput by 29% (lower bound) at concurrency 8 and 40% (upper bound) at concurrency 16; at the full-node level (TP4×2), the increase was 35–36%. Converted to token generation rate, this rose from 4.1 tok/s to 74.9 tok/s (an 18.3x speedup compared to the baseline without external storage recomputation) [source: measured, report R2].

These data indicate that in a 29.8K token long session, the loading speed of the 7.15GB KV data is the bottleneck. The FX100, using an all-flash NVMe-oF array (4-disk RAID0, single-port 100 GbE), increases read bandwidth from 0.98 GB/s (local NVMe) to 5.23 GB/s (with LMCache parallel read patch, achieving a 4.1x TTFT improvement) [source: measured, report R1], thereby mitigating the impact of storage latency on inference efficiency.

Extended Value of Storage Acceleration in Training Checkpoints and Model Loading

Beyond inference, the storage optimization approach for KV Cache also applies to training and model loading scenarios. The Mingxin FX100 demonstrates similar acceleration in the following cases:

• Training Checkpoint Saving: In 8-card 32B LoRA training, saving a full model snapshot of 65.6 GB was reduced from 178 seconds to 94 seconds (a 1.9x speedup), with sustained write bandwidth improving by 96% [source: measured, report R1].
• Model Inference Loading: On the Huawei Atlas 910B platform, DeepSeek-32B service loading time dropped from 691 seconds to 112 seconds (a 6.2x speedup), and DeepSeek-70B from 1399 seconds to 150 seconds (a 9.3x speedup) [source: measured, report R9 (Ascend platform)].

These results show that storage acceleration is not only key to KV Cache reuse but also a foundational infrastructure for improving the efficiency of the entire AI workflow.

Conclusion

The 7.15GB of KV Cache data in multi-turn conversations does not "disappear." Instead, it is efficiently orchestrated at the system level through VRAM management, storage offloading, and reuse strategies. Measured data from the Mingxin FX100 shows that in long-context cold recovery workloads, KV tiered acceleration can increase inference throughput by 29–40% and reduce TTFT by 26–32%. For technical decision-makers in computing centers, this means that when deploying large-scale multi-turn conversation services, storage acceleration is a viable path to optimize user experience and resource utilization. Mingxin Technology offers approximately 10 weeks of gated joint testing (G1: arrival acceptance / G2: single-node baseline / G3: main gate: TTFT reduction ≥25%, throughput +29–40% measured in-band / G4: 72-hour stability). Teams interested in joint testing are welcome to contact us.

Key Q&A

Q: In multi-turn conversations, how is the 7.15GB KV Cache of a 29.8K token session reused? A: vLLM and LMCache use prefix caching to store historical KV Cache in VRAM or high-speed storage. When a new request's prefix matches, the data is loaded directly, skipping recomputation. In cold-start scenarios, storage acceleration significantly reduces TTFT.

Q: What are the measured performance improvements of the Mingxin FX100 in long-context cold recovery workloads? A: On a 480B model (TP8 deployment), KV tiered acceleration increases inference throughput by 29–40% (at concurrency levels 8–16) and reduces TTFT by 26–32% (TTFT p50 from 10.17–35.73 seconds to 7.53–26.35 seconds) [source: measured, report R2].

Q: Does storage acceleration have similar effects in training and model loading scenarios? A: Yes. Training checkpoint saving is accelerated by 1.9x (write bandwidth improved by 96%), and model inference loading is accelerated by 6.2–9.3x (vs. NFS baseline) [source: measured, reports R1/R9].