Prefix Caching: From In-HBM Reuse to a Cluster-Level Asset
Prefix caching computes the KV of identical prefixes (system prompts, documents, session history) only once. The in-HBM version is capacity-limited; external tiering expands it into a cluster asset: cross-instance sharing and day-scale retention (verified in R3).
The value and the ceiling of prefix reuse
Production workloads repeat prefixes far more than intuition suggests: system prompts are globally identical, knowledge-base documents are referenced repeatedly, and every turn of a conversation carries the full history. vLLM's prefix caching identifies identical prefixes by block hash and skips prefill on hit — this part the open-source engine gives you for free.
The ceiling is capacity: HBM retains prefixes measured in GB, and a single long session on a 480B model eats a large share. As concurrency rises, prefix blocks crowd each other out and the hit rate decays fast — a media-capacity problem, not an algorithm problem.
Growing it into a cluster-level asset
External tiering swaps the medium from HBM to a three-level "HBM + RAM + all-flash" stack: FX100 fully populated holds 184.3 TB raw, relaxing prefix retention from minutes to days. The hit path is measured: 480B cold-recovery read-back TTFT p50 11.85 s vs 149.5 s re-compute (R2).
Sharing matters even more: the fs:// shared pool lets any instance in the cluster hit the same prefix KV (verified in R3) — a newly scaled-out instance comes online with the full cache asset from second one, which in-HBM schemes structurally cannot do.
FAQ
What prefix hit rate is typical?
Workload-dependent: system-prompt and knowledge-base scenarios hit high; one-shot long-tail Q&A hits low. No universal number — test with real traces (the G3 gate sets TTFT reduction ≥25% as the acceptance line).
What if one character of the prefix changes?
Mismatch is block-granular: blocks after the change point are invalidated, blocks before it still hit. Hence the prompt-engineering rule — put stable content first and variable content last for higher hit rates.
How does this relate to vendors' context-caching APIs?
Two commercial forms of one mechanism: API vendors bill hits at a discount; a self-hosted platform saves the compute directly. The self-hosted version measured throughput +29–40% (R2/R3).
Data sources (verifiable)
Related reading
- The Storage Technology Behind Context Caching
- KV Cache Tiering: The Three-Level HBM → RAM → All-Flash Architecture
- SGLang Prefix Reuse and an External KV Tier: Do They Work Together
Joint test first, decisions second: gate-based acceptance with built-in stop-loss
The full costing model is provided as reproducible Python after NDA — customers can rerun it with their own parameters. Every key figure on this site carries a report ID and is open to third-party verification.
This site presents business-cooperation information and constitutes neither an investment offer nor any promise of returns. Measured data come from signed / official test reports (see the Evidence Library); vendor specs, public sources and estimates are labeled as such.