Long-Context Cold Recovery: The Main Performance Battleground of the Agent Era
Cold recovery means re-activating a long-context session after interruption: its KV has been evicted from GPU memory, so you either recompute or read back from an external tier. Mingxin measured read-back 8.6–20× faster than re-compute and 26–32% faster than a local drive.
Why cold recovery keeps getting more important
Agent workflows inherently follow an interrupt–resume pattern: waiting on tool calls, human confirmation, multi-agent handoffs — every resume reloads tens of thousands of tokens of context. Code assistants behave the same: the developer closes the window today and reopens tomorrow with the session history intact.
The workload signature: long contexts, high revisit rates, unpredictable resume timing. GPU memory cannot hold KV for every dormant session, so cold recovery is inevitable — the only question is re-compute or read-back.
Measured: read-back vs re-compute vs local drive
R2 specifically tested a cold-recovery storm on the 480B production form: at concurrency 16, the re-compute baseline posted TTFT p50 of 149.5 s and 4.1 tok/s — essentially unusable; FX100 read-back: 11.85 s and 74.9 tok/s; local NVMe: 17.31 s and 53.6 tok/s.
Stated by basis: 8.6–20× vs re-compute; vs local drive, TTFT lower by 26–32% and throughput higher by 29–40%. The external pool additionally offers cross-instance sharing — any instance recovering a session hits the same KV copy (R3 fs:// verification).
FAQ
How long is it economical to retain sessions?
The external tier costs about ¥2,014/TB (derived from the FX100 fully-populated reference price), orders of magnitude below HBM, so retention windows can stretch to days. Use the ROI calculator with your workload parameters for specifics.
Is there a difference between multi-turn chat and agent workloads?
Mechanically the same (prefix reuse); agents resume more frequently with longer contexts, so the external tier pays off more. The R2 load design simulates exactly this session-recovery storm.
Can cold recovery be made imperceptible to users?
TTFT p50 of 11.85 s on a 480B model (concurrency 16) versus 149.5 s re-compute is a qualitative difference, but "imperceptible" depends on model size and concurrency; smaller models recover faster (9.30 s in the single-GPU 32B scenario, R1).
Data sources (verifiable)
Related reading
- TTFT Optimization: Engineering Paths to Cut LLM First-Token Latency
- KV Cache Offloading: Putting the Inference Cache on an All-Flash Array
- The Storage Technology Behind Context Caching
Joint test first, decisions second: gate-based acceptance with built-in stop-loss
The full costing model is provided as reproducible Python after NDA — customers can rerun it with their own parameters. Every key figure on this site carries a report ID and is open to third-party verification.
This site presents business-cooperation information and constitutes neither an investment offer nor any promise of returns. Measured data come from signed / official test reports (see the Evidence Library); vendor specs, public sources and estimates are labeled as such.