Inference Acceleration for RAG Knowledge-Base Q&A: Prefix Reuse's Best Scenario
Every RAG query stuffs retrieved document passages into the context, so the same popular documents get prefilled over and over — one of the highest-payoff workloads for prefix reuse. An external KV tier computes a hot document's KV once for the whole cluster.
RAG's hidden waste
A typical enterprise RAG context consists of: system prompt (globally identical) + retrieved document passages (concentrated on hot spots) + the user question (the only truly variable part). The first two dominate the context and repeat heavily, but in-HBM prefix caching is capacity-limited, so hot documents' KV keeps getting evicted and recomputed.
The cost of recompute has a measured anchor: the 480B long-context re-compute baseline reaches TTFT p50 149.5 s (concurrency 16, R2) — RAG contexts commonly exceed ten thousand tokens, squarely in this magnitude.
The payoff structure of external tiering in RAG
Once hot documents' KV sinks into the external pool: any instance hits the same copy (R3 cross-instance sharing verified); the retention window runs in days (knowledge-base update cycles are usually days/weeks — a perfect match); and read-back TTFT measured 11.85 s vs 149.5 s recompute (R2).
On knowledge-base updates, invalidate the affected documents' KV blocks at document granularity (content-addressed prefix hashing) — no full flush needed. The accompanying throughput band: +29–40% (R2/R3), translating directly into higher QPS for high-concurrency RAG on the same cluster.
FAQ
What happens to the cache when documents change?
Content addressing handles it naturally: a changed document changes its hash, so the old KV simply stops hitting and is LRU-evicted. Stale-cache reads cannot happen.
Retrieval results differ per query — can prefixes still hit?
Yes: hits are block-granular and each retrieved passage forms its own blocks; whenever a passage is retrieved again (inevitable for hot documents), its KV hits.
At what concurrency do gains become clear?
The measured band covers concurrency 8–32 (R2, holding at all three levels); for smaller scale, joint-test with your own Q&A traces (G3 gate: TTFT reduction ≥25% to pass).
Data sources (verifiable)
Related reading
- Prefix Caching: From In-HBM Reuse to a Cluster-Level Asset
- The Storage Technology Behind Context Caching
- Session-Recovery Acceleration for Agent and Code-Assistant Platforms
Joint test first, decisions second: gate-based acceptance with built-in stop-loss
The full costing model is provided as reproducible Python after NDA — customers can rerun it with their own parameters. Every key figure on this site carries a report ID and is open to third-party verification.
This site presents business-cooperation information and constitutes neither an investment offer nor any promise of returns. Measured data come from signed / official test reports (see the Evidence Library); vendor specs, public sources and estimates are labeled as such.