Real Cost of No-External-Storage Recomputation Approach

In long-context large model inference scenarios, some teams adopt a "no-external-storage recomputation" strategy—that is, not saving KV Cache to external storage and recomputing from scratch for each request. However, this seemingly "free" approach actually carries significant time and cost penalties. Based on measured data from Mingxin FX100 on the AMD MI308X platform, for a 480B-parameter MoE model (Qwen3-Coder-480B-FP8), the TTFT p50 for no-external-storage recomputation reaches 149.5 seconds (concurrency 16), while with FX100 storage acceleration it drops to 11.85 seconds, and throughput increases from 4.1 tok/s to 74.9 tok/s, achieving an acceleration factor of 8.6–20x [source: measured, report R2]. This article quantifies the real cost of recomputation from three dimensions: latency, throughput, and resource efficiency.

The "Free" Illusion of Recomputation: Dual Penalties in Latency and Throughput

The core assumption of the "no-external-storage recomputation" strategy is that GPU compute power is fast enough to make KV Cache recomputation negligible. However, in long-context scenarios, this assumption fails. Taking the 480B model as an example, each request's context length can reach hundreds of thousands of tokens. Recomputation of KV Cache requires re-executing forward propagation across all layers, with computation scaling linearly with context length. Measured data shows that under TP8 configuration, the TTFT p50 for no-external-storage recomputation is 149.5 seconds (concurrency 16), while with FX100 storage acceleration, TTFT drops to 11.85 seconds [source: measured, report R2]. This means a single request forces users to wait nearly 2.5 minutes for the first token, which is unacceptable in production deployments.

More critically, recomputation not only impacts first-token latency but also severely degrades throughput. At concurrency 16, throughput for no-external-storage recomputation is only 4.1 tok/s, while with FX100 acceleration it reaches 74.9 tok/s—an approximately 18x improvement [source: measured, report R2]. This throughput gap stems from recomputation's exclusive occupation of GPU resources: when multiple concurrent requests simultaneously recompute KV Cache, GPU compute units are heavily occupied, unable to efficiently process new requests, causing a sharp drop in overall system throughput. For scenarios requiring service to a large number of concurrent users, the recomputation approach quickly becomes a performance bottleneck.

Cost Calculation: The Hidden Economic Cost of Recomputation

The cost of recomputation is not limited to latency and throughput; it directly translates into economic expenses. Taking a typical configuration of 8 AMD MI308X cards (each with 192GB HBM) as an example, assuming a single recomputation consumes approximately 30 seconds of GPU compute resources (based on the measured 149.5s TTFT where recomputation dominates), the number of recomputation requests that can be processed per hour is about 120. With FX100 storage acceleration, TTFT drops to 11.85 seconds, allowing approximately 304 requests per hour. Under the same GPU hardware investment, the storage acceleration solution improves request processing capacity by about 2.5 times. Based on an estimated cost of approximately $15,000 per MI308X card, the recomputation approach would require an additional 2.5x GPU cluster to achieve the same service throughput, resulting in a hardware cost increase of over 150%.

Additionally, the pressure on network and storage from recomputation is underestimated. Although no-external-storage recomputation does not rely on external storage, each request's KV Cache computation still requires frequent reads of model weights and intermediate activations from HBM, intensifying competition for HBM bandwidth. At concurrency 16, HBM bandwidth utilization for no-external-storage recomputation approaches saturation, delaying other compute tasks such as attention mechanisms. In contrast, FX100 provides low-latency KV Cache access via an NVMe-oF array (4-disk RAID0, 14TB, RoCEv2). Measured data shows that LMCache parallel read patches improve TTFT by 4.1x, with bandwidth increasing from 0.98 GB/s to 5.23 GB/s [source: measured, report R1]. This storage acceleration solution decouples KV Cache storage from computation, freeing GPU resources for core inference tasks.

Technical Comparison: Applicability Boundaries of Recomputation vs. Storage Acceleration

The recomputation approach is not entirely without value. It may still be applicable in the following scenarios: extremely short request contexts (e.g., <1K tokens), very low concurrency (e.g., <4), or small model sizes (e.g., <7B parameters). In these cases, the latency overhead of recomputation may be masked by other factors such as network latency. However, for mainstream production deployments—long contexts (>32K tokens), high concurrency (>16), and large models (>100B parameters)—the recomputation approach significantly degrades service quality.

Measured data further validates this point. In tests with the 480B model, TP8, and concurrency 16, the TTFT p50 for no-external-storage recomputation is 149.5s, while with FX100 acceleration it is 11.85s, a difference of 12.6x [source: measured, report R2]. At concurrency 8, recomputation TTFT is 97.3s, and FX100 is 7.53s, a difference of approximately 12.9x [source: measured, report R2]. This indicates that even with reduced concurrency, the latency penalty of recomputation remains substantial. Therefore, for production systems pursuing low latency and high throughput, the storage acceleration solution is the better choice.

Conclusion

Recomputation is not "free"—it shifts computational costs to GPU resources and user experience by sacrificing latency and throughput. Mingxin FX100, through its storage acceleration solution, achieves an 8.6–20x recomputation acceleration factor on the 480B model, reducing TTFT to the second level [source: measured, report R2]. For technical decision-makers in computing centers, it is recommended to prioritize evaluating storage acceleration solutions in long-context, high-concurrency scenarios. Mingxin offers an approximately 10-week gated joint testing process (from G1 arrival acceptance to G4 72-hour stability test), supporting verification of measured metrics in customer environments, including TTFT reduction ≥25% and throughput improvement of 29–40% [source: cooperation model]. For further technical discussions or joint testing collaboration, please contact the Mingxin technical team.

Key Q&A

Q: What is the TTFT cost of no-external-storage recomputation on the 480B model?
A: At concurrency 16, the TTFT p50 for no-external-storage recomputation is 149.5 seconds, while with Mingxin FX100 storage acceleration it drops to 11.85 seconds, an acceleration of approximately 12.6x [source: measured, report R2].

Q: How significant is the impact of the recomputation approach on throughput?
A: Under the same concurrency, throughput for no-external-storage recomputation is 4.1 tok/s, while with FX100 acceleration it reaches 74.9 tok/s, an improvement of approximately 18x [source: measured, report R2].

Q: In which scenarios does the storage acceleration solution offer the best cost-effectiveness?
A: In production deployments with long contexts (>32K tokens), high concurrency (>16), and large models (>100B parameters), the storage acceleration solution can significantly reduce latency and hardware costs. Measured data shows TTFT reduction of 26–32% and throughput improvement of 29–40% [source: measured, reports R2/R3].