Solution for 'Restore Storm' in Long-Context Sessions

Cold restore for long-context sessions is a critical performance bottleneck currently faced by Agent and code assistant applications: when a service needs to rebuild its context due to failure, scaling, or maintenance, reloading or recomputing the KV Cache from external storage can cause the Time-To-First-Token (TTFT) to surge from seconds to minutes, creating a so-called "restore storm." Based on measured data from Mingxin FX100 on a 480B MoE model (Qwen3-Coder-480B-FP8), this article demonstrates that an NVMe-oF storage acceleration solution can reduce TTFT in cold restore scenarios by 26-32% and achieve an 8.6-20x speedup without external storage recomputation, providing an engineerable solution for the elasticity and availability of long-context sessions.

Background: Why Agents and Code Assistants Encounter a "Restore Storm"

The core working mode of Agents and code assistants (such as Cursor, Copilot-type applications) involves maintaining long-duration, multi-turn session context. Each user interaction relies on historical dialogue, code snippets, tool call results, and other information. In inference frameworks like vLLM, this context is encoded as a KV Cache and resides in GPU memory. However, when a service node fails, undergoes a rolling upgrade, or is migrated for load balancing, the KV Cache in memory is lost, and the system must reload or recompute it from external storage (typically NFS or local SSD).

Recomputation is the most straightforward recovery method, but it is extremely costly. Taking Qwen3-Coder-480B-FP8 as an example, its weights are approximately 450 GB, and a single forward pass can generate hundreds of MB of KV Cache. For long contexts (e.g., 128K tokens), recomputing the entire sequence can result in a TTFT of 149.5 seconds (concurrency 16, measured baseline, report R2). In production environments, this means users may need to wait several minutes for the first response, directly leading to session timeouts or a breakdown in user experience.

More critically, Agent scenarios often require high-frequency recovery: when multiple user sessions restore simultaneously, GPU computational resources are fully occupied by recomputation tasks, unable to handle new inference requests, creating a chain reaction of the "restore storm." Therefore, how to efficiently persist and quickly restore the KV Cache becomes a core issue for improving the resilience of Agent systems.

KV Cache Storage Acceleration: Quantifiable Improvement from NFS to NVMe-oF

Traditional KV Cache persistence solutions rely on NFS (Network File System), but its latency and bandwidth bottlenecks are fully exposed in long-context scenarios. The Mingxin FX100 all-flash NVMe-oF array (4-disk RAID0, 14 TB, RoCEv2, single-port 100 GbE) on an AMD MI308X ×8 platform, through the LMCache framework, implements parallel read patching for KV Cache. Compared to a local single NVMe disk baseline, TTFT improvement reached 4.1x (measured, report R1: Qwen2.5-32B, concurrency 16, cold disk read TTFT reduced from 37.97s to 9.30s, bandwidth increased from 0.98 GB/s to 5.23 GB/s).

In a 480B production deployment configuration (TP4×2 full-machine scale), FX100's KV tiering accelerated inference throughput by 29-40% (measured, reports R2/R3). Specifically for cold restore scenarios, TTFT was reduced by 26-32%: under three concurrency levels for 480B·TP8, TTFT p50 decreased from 10.17-35.73s to 7.53-26.35s (measured, report R2). For extreme cases without external storage recomputation, the speedup factor reached 8.6-20x: recomputation baseline TTFT p50 149.5s (concurrency 16) compared to FX100's 11.85s, throughput increased from 4.1 to 74.9 tok/s (measured, report R2).

The core mechanism behind this improvement is: NVMe-oF provides lower latency and higher IOPS than NFS, while LMCache's parallel read patch changes KV Cache reading from serial to parallel, fully utilizing the bandwidth advantage of the all-flash array. For Agent scenarios, this means the wait time for the first request after a service restart is compressed from minutes to seconds, eliminating the perception window of the "restore storm."

Synergistic Acceleration for Training Checkpoint and Inference Loading

The cold restore problem exists not only in the inference phase; the saving and loading of training checkpoints also affect the iteration efficiency of Agent systems. In 8-card 32B LoRA training, the save time for each 65.6 GB full-model snapshot decreased from 178s to 94s, with sustained write bandwidth improving by 96% (measured, report R1). This acceleration capability means frequent checkpoint saving (e.g., after each training round) no longer becomes a bottleneck in the training pipeline, thereby supporting shorter iteration cycles.

For model inference loading, the acceleration effect of FX100 compared to NFS is even more significant: on the Huawei Atlas 910B platform, DeepSeek-32B service loading decreased from 691s to 112s (6.2x), and DeepSeek-70B decreased from 1399s to 150s (9.3x) (measured, report R9). For Agent services that require frequent model switching or A/B testing, this acceleration directly shortens service readiness time and reduces operational complexity.

Conclusion

Cold restore for long-context sessions is an obstacle that must be overcome for Agents and code assistants to move into production. Mingxin FX100, through the synergistic optimization of NVMe-oF storage acceleration and LMCache, achieved a quantifiable improvement of 26-32% TTFT reduction and 8.6-20x speedup without external storage recomputation on a 480B model, providing a verifiable engineering solution for the "restore storm" problem. Mingxin (Tianjin) Semiconductor Equipment Co., Ltd., focusing on storage acceleration and domestic computing power, provides full-industry-chain services for computing centers. Teams with joint testing needs are welcome to verify practical results through a gated cooperation model (approximately 10 weeks, including G1-G4 acceptance milestones).

Key Q&A

Q: What specifically does the "restore storm" in cold restore for long-context sessions refer to? A: It refers to the chain reaction where, when an Agent/code assistant service needs to rebuild its KV Cache due to failure or migration, recomputation causes Time-To-First-Token (TTFT) to surge from seconds to minutes (e.g., baseline for a 480B model reaching 149.5s), resulting in user wait timeouts and computational resources being fully occupied.

Q: How does Mingxin FX100 quantitatively solve the cold restore latency problem? A: Under 480B·TP8 production load, FX100, through the NVMe-oF array and LMCache parallel read patch, reduces TTFT by 26-32% (from 10.17-35.73s to 7.53-26.35s, measured, report R2); in scenarios without external storage recomputation, acceleration is 8.6-20x (TTFT reduced from 149.5s to 11.85s, throughput increased from 4.1 to 74.9 tok/s, measured, report R2).

Q: Does this solution offer synergistic benefits for training checkpoint saving? A: Yes. In 8-card 32B LoRA training, the save time for each 65.6 GB snapshot decreased from 178s to 94s (1.9x speedup, write bandwidth improved by 96%, measured, report R1), supporting shorter training iteration cycles.