KV Cache Hit Rate and Cache Pricing: Storage Economics in API Cost Structures

In the API cost structure of large model inference, the KV Cache hit rate is a key variable determining first-token latency (TTFT) and throughput efficiency, while the rationality of cache pricing is directly tied to the economics of storage acceleration. Based on measured data from Mingxin FX100 on a 480B model, tiered KV Cache acceleration can improve throughput by 29–40% (measured, report R2/R3) and reduce TTFT by 26–32% (measured, report R2). This indicates that an efficient KV Cache storage solution can significantly lower the inference cost per token. This article explores three dimensions: the quantitative relationship between hit rate and latency, the storage economics model of cache pricing, and the practice of storage acceleration in the vLLM and LMCache ecosystem, providing a reusable analytical framework for technical decision-makers.

How Does KV Cache Hit Rate Affect First-Token Latency and Throughput?

The KV Cache hit rate directly impacts two core metrics of an inference system: first-token latency (TTFT) and throughput (tok/s). When the cache hits, the model does not need to recompute the Key-Value pairs of historical tokens; it directly reuses the cached intermediate results. When the cache misses (i.e., cold start or context switch), the system must compute from scratch, causing a significant increase in TTFT.

In measured data from Mingxin FX100 on a 480B model (TP8, concurrency 16), the baseline TTFT p50 without external memory recomputation was 149.5 seconds. After enabling FX100 storage acceleration, it dropped to 11.85 seconds, achieving a speedup of 12.6x (measured, report R2). This difference directly maps to throughput: baseline throughput was 4.1 tok/s, while FX100 increased it to 74.9 tok/s, an improvement of approximately 18.3x. In the more production-like tiered KV acceleration scenario, throughput improvement ranged from 29–40% (measured, reports R2/R3), and TTFT was reduced by 26–32% (measured, report R2). These data show that for every percentage point increase in KV Cache hit rate, the system's real-time performance gains grow nonlinearly.

From a system architecture perspective, the choice of storage medium for KV Cache is critical. Traditional solutions rely on local NVMe or NFS, but are limited by bandwidth and latency. Mingxin FX100, through an all-flash NVMe-oF array (4 disks RAID0, single port 100 GbE), achieves a parallel read bandwidth of 5.3 GB/s (after LMCache patch, measured, report R1), far exceeding the 0.98 GB/s of a local single disk (PCIe Gen4, 2 TB). This bandwidth advantage directly translates to a higher cache hit rate—in long-context workloads, more historical tokens can be loaded quickly, reducing recomputation.

Storage Economics of Cache Pricing: From Unit Cost to Total Cost of Ownership

In the API cost structure, KV Cache pricing is typically based on token count or cache size, but behind it lies the economics of storage acceleration. The core value of storage acceleration is to lower the inference cost per token, achieved through two paths: first, reducing the waste of compute resources from recomputation, and second, increasing throughput to amortize fixed compute costs.

Taking the pricing of Mingxin FX100 as an example, the reference price for a fully configured unit is approximately ¥371,200 (about ¥2,014/TB, measured, report R2). Compared to the throughput improvement it brings (29–40%), the storage cost per token is significantly lower than the recomputation cost. In the 480B model scenario, if baseline throughput is 4.1 tok/s and FX100 increases it to 74.9 tok/s, this means approximately 255,000 more tokens can be processed per hour. Based on typical API pricing (e.g., $0.002/1K tokens), the incremental revenue per hour is about $510, while the hardware cost, depreciated over 3 years, is about $15/hour, making the payback period reasonable.

From a cache pricing strategy perspective, users need to weigh the following factors:

• Cache Size: Larger cache capacity can accommodate more context but incurs higher storage costs. Mingxin FX100 has a single array capacity of 14 TB, suitable for long-context scenarios of the 480B model (weights approximately 450 GB).
• Hit Rate Threshold: When the hit rate falls below 50%, the marginal benefit of storage acceleration diminishes. Measured data shows a throughput improvement of 29% (lower bound) at concurrency 8 and 40% (upper bound) at concurrency 16, indicating higher cache reuse rates and better economics in high-concurrency scenarios (measured, reports R2/R3).
• Latency Sensitivity: In scenarios with strict TTFT requirements (e.g., real-time dialogue), the value of storage acceleration is higher. FX100 reduces TTFT from 149.5 seconds to 11.85 seconds, directly meeting the latency requirements for production deployment.

Storage Acceleration Practices in the vLLM and LMCache Ecosystem

vLLM, as a mainstream inference framework, uses PagedAttention by default to manage KV Cache, but its caching strategy relies on local DRAM. When context length exceeds VRAM capacity, it must offload to SSD or network storage. LMCache optimizes this process through a parallel read patch. On the Qwen2.5-32B model, TTFT dropped from 37.97 seconds to 9.30 seconds, with a 5.3x bandwidth improvement (measured, report R1).

The integration of Mingxin FX100 with LMCache further amplifies the storage acceleration effect. On the AMD MI308X platform, FX100 provides low-latency NVMe-oF access via the RoCEv2 protocol, enabling LMCache's cold read disk bandwidth to reach 5.23 GB/s, while a local NVMe single disk achieves only 0.98 GB/s. This architecture design elevates the KV Cache hit rate from a theoretical value (based on VRAM capacity) to a practically achievable value (based on storage bandwidth), making it particularly suitable for long-context scenarios (e.g., multi-turn dialogue with a 480B model).

Additionally, FX100's 1.9x speedup in training checkpoint saving scenarios (measured, report R1) indirectly improves the economics of cache pricing—faster saving times mean shorter training interruptions, reducing overall compute costs.

Conclusion

The KV Cache hit rate and cache pricing are not isolated parameters but core levers in the cost structure of large model inference. Through storage acceleration technology (such as Mingxin FX100's all-flash NVMe-oF array), users can reduce TTFT by 26–32% and improve throughput by 29–40%, achieving a significant advantage in per-token cost. For technical decision-makers in compute centers, it is recommended to focus on the alignment of storage bandwidth, cache hit rate, and pricing model during selection, and to validate actual benefits through joint testing. Mingxin offers approximately 10 weeks of joint testing cooperation (including G1–G4 gated acceptance), ensuring measured in-band compliance with TTFT reduction ≥25% and throughput improvement of 29–40%, with loss mitigation if targets are not met.

Key Q&A from This Article

Q: What is the specific impact of KV Cache hit rate on inference latency?
A: In tests on the 480B model, TTFT dropped from 149.5 seconds to 11.85 seconds (12.6x speedup) without external memory recomputation. After enabling tiered acceleration, TTFT was reduced by 26–32% (measured, reports R2/R3). Every 10% increase in hit rate results in a nonlinear improvement in latency.

Q: How to evaluate the economics of cache pricing?
A: The core metric is cost per token. Mingxin FX100, fully configured at approximately ¥371,200, can process about 255,000 more tokens per hour after a 29–40% throughput improvement. Based on typical API pricing, the payback period is reasonable (measured, reports R2/R3).

Q: What are the advantages of storage acceleration in the vLLM and LMCache ecosystem?
A: Through an NVMe-oF array achieving 5.23 GB/s parallel read bandwidth (5.3x improvement over a local single disk), LMCache's cold read TTFT dropped from 37.97 seconds to 9.30 seconds (measured, report R1). This directly improves cache hit rates and inference efficiency in long-context scenarios.