Why "Recomputation" Is Not Free: The Real Cost of a No-External-Storage Strategy

In large model inference, the "no-external-storage recomputation" strategy for KV Cache is often mistakenly considered "free" or "low-cost." However, actual measurements show its cost is far higher than expected: using a 480B parameter model as an example, the TTFT (time to first token) for no-external-storage recomputation can reach 149.5 seconds, while a storage acceleration solution (such as Mingxin FX100) can reduce latency to 11.85 seconds, improving throughput by over 18x. This article quantifies the real cost of recomputation from three dimensions—latency, throughput, and energy consumption—and compares optimization paths under frameworks like LMCache and vLLM.

The "Hidden Bill" of Recomputation: TTFT Latency and Throughput Loss

In long-context inference scenarios, KV Cache recomputation means that each request must recalculate the key-value pairs for all historical tokens. Taking the Qwen3-Coder-480B-FP8 model (MoE, weights approximately 450 GB) on an 8×AMD MI308X platform as an example, the TTFT p50 for no-external-storage recomputation at a concurrency of 16 reaches 149.5 seconds [source: measured, report R2]. In contrast, using the Mingxin FX100 all-flash NVMe-oF array (4-disk RAID0, 14 TB, RoCEv2), the TTFT p50 drops to 11.85 seconds, a speedup of 12.6x. More critically, throughput jumps from 4.1 tok/s to 74.9 tok/s, an improvement of 18.3x [source: measured, report R2].

This is not an isolated case. In a production deployment of the 480B model (TP4×2 full-machine configuration), the KV hierarchical acceleration with FX100 improves inference throughput by 35–36% [source: measured, report R3]. The "hidden bill" of recomputation lies in the fact that GPU compute resources are heavily consumed by repetitive calculations rather than generating new tokens. For compute centers deploying long-context applications (e.g., code generation, document analysis), this means user wait times jump from seconds to minutes, directly degrading the user experience.

Why LMCache and vLLM Cannot Fully Eliminate the Cost of Recomputation?

LMCache and vLLM are currently mainstream KV Cache management frameworks, but they rely on local DRAM or SSDs as cache media. Measurements show that when cache hit rates are insufficient, the cost of recomputation remains significant. For example, in the LMCache parallel read-patch test, with a single GPU at concurrency 16 and a cold-read scenario (Qwen2.5-32B), TTFT dropped from 37.97 seconds to 9.30 seconds, and bandwidth increased from 0.98 GB/s to 5.23 GB/s (↑5.3x) [source: measured, report R1]. However, this optimization depends on cache pre-filling strategies; if request sequence lengths or context distributions change, cold starts will still trigger recomputation.

More critically, local storage (e.g., a single NVMe drive) has limited bandwidth and IOPS. In the 480B model test, the baseline using a local NVMe single drive (PCIe Gen4, 2 TB) achieved a sustained write bandwidth of only 3.26 GB/s, while the FX100 array reached 6.40 GB/s (+96%) [source: measured, report R1]. This means that even with optimized cache management from LMCache and vLLM, the underlying storage bottleneck can amplify the cost of recomputation. For inference services targeting low latency, software optimization alone cannot solve the hardware I/O capacity deficiency.

Energy Consumption and Cost: The "Hidden Inflation" of Recomputation

Recomputation not only consumes time but also directly increases GPU energy consumption. Taking the 8×AMD MI308X platform (each card with 192 GB HBM) as an example, during recomputation, GPU utilization approaches 100%, with power consumption estimated at approximately 1.5 kW. If each request requires 149.5 seconds of recomputation, the energy per request is about 0.062 kWh. For a production environment processing 100,000 requests per day, the annual energy cost of the recomputation scheme (estimated at $0.1/kWh) would be approximately $22,600. In contrast, with storage acceleration (TTFT 11.85 seconds), the annual energy cost drops to about $1,800, a savings of 92%.

From a hardware procurement cost perspective, the reference price for a fully configured Mingxin FX100 system is ¥371,200 (approximately ¥2,014/TB) [source: fact sheet]. Meanwhile, the recomputation scheme requires more GPUs to compensate for throughput loss. For the 480B model, if the target throughput is 100 tok/s, the recomputation scheme would require approximately 24 GPUs (based on 4.1 tok/s per machine), while the storage acceleration scheme needs only 2 GPUs (based on 74.9 tok/s). The GPU cost difference can exceed 10x. Thus, the "hidden inflation" of recomputation is that it seemingly saves on storage investment but demands several times more compute resources.

Conclusion

No-external-storage recomputation is not "free"—it carries significant costs in latency, throughput, energy consumption, and hardware expenditure. For technical decision-makers at compute centers deploying long-context inference services, storage acceleration solutions (such as the Mingxin FX100 all-flash NVMe-oF array) should be prioritized to reduce recomputation overhead. Mingxin Technology offers approximately 10 weeks of gated joint testing (G1: arrival acceptance / G2: single-machine baseline / G3: main gate: TTFT reduction ≥25%, throughput +29–40% measured in-band / G4: 72-hour stability), with a stop-loss mechanism if targets are not met. Compute centers and model service providers are welcome to schedule joint testing to verify the economics of storage acceleration.

Key Q&A from This Article

Q: What are the main costs of the no-external-storage recomputation scheme?
A: The main costs include TTFT latency up to 149.5 seconds (480B model, concurrency 16) and throughput of only 4.1 tok/s, compared to the storage acceleration scheme (TTFT 11.85 seconds, throughput 74.9 tok/s), a gap of 12–18x [source: measured, report R2].

Q: Can LMCache and vLLM fully eliminate the cost of recomputation?
A: No. Even though LMCache optimizes cache read-patching (TTFT improved by 4.1x), cold-start scenarios still depend on storage I/O capability; a local NVMe single drive has only 3.26 GB/s bandwidth, while the FX100 array reaches 6.40 GB/s (+96%) [source: measured, report R1].

Q: What is the impact of the recomputation scheme on energy consumption and hardware costs?
A: The annual energy cost of the recomputation scheme (100,000 requests per day) is approximately $22,600, while the storage acceleration scheme reduces it to about $1,800 (a 92% savings); GPU deployment numbers may increase by over 10x due to throughput loss.