Session-Recovery Acceleration for Agent and Code-Assistant Platforms
Agent and code-assistant workloads combine long contexts with frequent interrupt-resume cycles. An external KV tier turns recovery from full re-compute into read-back: measured on 480B, TTFT p50 fell from 149.5 s to 11.85 s.
Why agent workloads are hard to serve
A single agent session easily reaches tens of thousands of tokens (system prompt, tool outputs, code context) and keeps cycling through think–call-tool–wait–resume. If every resume recomputes prefill, GPU compute is burned on duplicate work and users see minute-scale stalls.
R2's cold-recovery storm load simulates exactly this pattern: a 480B model at concurrency 16, where the re-compute baseline posts TTFT p50 149.5 s and 4.1 tok/s — unacceptable for an agent platform.
The external-KV solution and its ledger
The FX100 read-back path: TTFT p50 11.85 s, throughput 74.9 tok/s (12.6× TTFT and 18× throughput vs re-compute), plus a 32% TTFT edge over a local drive (R2). Cross-instance sharing (R3) means the agent scheduler can route recovery requests to any idle instance.
On cost: retaining KV on the flash tier runs about ¥2,014/TB — orders of magnitude cheaper than expanding HBM for peak concurrency. The ROI calculator estimates the benefit range from your session distribution.
FAQ
Do multi-agent setups benefit more?
Yes: shared system prompts and knowledge-base prefixes repeat heavily across agents, so pooled KV hits more often (prefix hashing deduplicates naturally).
How is conversation privacy isolated?
KV blocks are isolated by tenant namespace; in a private deployment data never leaves the cluster — a cleaner boundary than calling public-cloud caching APIs.
Do small-model agents (7B–32B) benefit too?
Yes: R1 measured single-GPU 32B cold-read TTFT 37.97 s → 9.30 s (4.1×). The smaller the model, the faster re-compute becomes and the smaller the gain — test with your own workload.
Data sources (verifiable)
Related reading
- Long-Context Cold Recovery: The Main Performance Battleground of the Agent Era
- The Storage Technology Behind Context Caching
- TTFT Optimization: Engineering Paths to Cut LLM First-Token Latency
Joint test first, decisions second: gate-based acceptance with built-in stop-loss
The full costing model is provided as reproducible Python after NDA — customers can rerun it with their own parameters. Every key figure on this site carries a report ID and is open to third-party verification.
This site presents business-cooperation information and constitutes neither an investment offer nor any promise of returns. Measured data come from signed / official test reports (see the Evidence Library); vendor specs, public sources and estimates are labeled as such.