TTFT Optimization: Cutting LLM First-Token Latency
TTFT consists of queueing, prefill re-compute and KV read-back. In long-context workloads prefill re-compute dominates, and external KV read-back largely removes it: Mingxin measured TTFT p50 down 26–32%.
Where high TTFT comes from
What users perceive as "lag" is mostly TTFT. For long contexts (tens of thousands of tokens of session history or codebase context), prefill runs a forward pass over the entire history — tens of seconds of GPU time; once concurrency rises, queueing pushes TTFT p99 into collapse.
R2 quantifies it: on a 480B model at concurrency 16, the no-external-storage re-compute baseline hits TTFT p50 of 149.5 s — the price of ignoring KV.
Measured effect of tiered read-back
Under the same load, external KV read-back on FX100 pressed TTFT p50 to 11.85 s (12.6× vs re-compute) and beat the local NVMe drive (17.31 s) by 32%. It holds across all three concurrency levels (8/16/32), with throughput up 29–40% (R2/R3).
Read-back bandwidth is the key engineering quantity: the LMCache parallel-read patch raised single-GPU cold-read bandwidth from 0.98 GB/s to 5.23 GB/s (5.3×), improving TTFT by 4.1× (R1). The patch is provided as a full git patch (R8).
FAQ
Does TTFT optimization help short conversations?
Only marginally — short-context prefill is fast anyway. The battleground is long context and session recovery, whose share keeps rising in agent and code-assistant workloads.
p50 went down — what about p99?
R3 measured nine load levels with full metrics (TTFT p50/p90/p99, TPOT, throughput, disk bandwidth); download the report in the Evidence Library to inspect every percentile.
On what platform were these numbers measured?
8× AMD MI308X, ROCm 7.2, vLLM 0.20.1, Qwen3-Coder-480B-FP8, standard production deployment form (TP=8) — all conditions published in full (R2).
Data sources (verifiable)
Related reading
- Long-Context Cold Recovery: The Agent-Era Bottleneck
- KV Cache Offloading: Inference Cache on All-Flash
- Productionizing LMCache: Open Source to Signed Results
Joint test first, decisions second: gate-based acceptance with built-in stop-loss
The full costing model is provided as reproducible Python after NDA — customers can rerun it with their own parameters. Every key figure on this site carries a report ID and is open to third-party verification.
This site presents business-cooperation information and constitutes neither an investment offer nor any promise of returns. Measured data come from signed / official test reports (see the Evidence Library); vendor specs, public sources and estimates are labeled as such.