Technical Challenges and Innovation Directions for Domestic KV Cache Chips

Domestic KV Cache chips are at a critical crossroads: they must solve the memory wall problem caused by KV Cache in large model inference, while finding a viable technical path under the constraints of an underdeveloped domestic computing ecosystem. Measured data from Mingxin's FX100 series on the AMD MI308X platform shows that through storage tiering and acceleration technologies, the KV Cache bottleneck is not insurmountable—under a 480B model long-context cold-restore workload, inference throughput can be improved by 29–40% (measured, reports R2/R3). This result offers direct reference value for domestic chip manufacturers.

Three Technical Challenges Facing Domestic KV Cache Chips

KV Cache is a memory structure that stores historical attention key-value pairs during large model inference, with its capacity growing linearly with sequence length. According to Efficient Memory Management for Large Language Model Serving with PagedAttention, the memory fragmentation issue of KV Cache can significantly reduce memory utilization in long-context scenarios. Domestic chips face three challenges in this domain:

Memory bandwidth bottleneck. KV Cache reads are high-frequency operations during inference and are highly sensitive to memory bandwidth. Domestic chips have a generational gap in HBM bandwidth compared to leading products, which directly constrains KV Cache read speed. Mingxin's R2 measurements show that under 480B·TP8 with three concurrency levels, TTFT p50 dropped from 10.17–35.73s to 7.53–26.35s, a reduction of 26–32%—this is the gain achieved through storage optimization under bandwidth-constrained conditions.

Capacity versus cost trade-off. KV Cache requires large storage capacity, but HBM is expensive. If domestic chips adopt an HBM solution, overall system cost loses competitiveness; if they adopt a DDR solution, bandwidth is insufficient to support efficient inference. The Mingxin FX100 full-configuration reference price is ¥371,200 (approximately ¥2,014/TB), achieving 16M IOPS over PCIe 3.0, providing a reference for cost-sensitive scenarios (per vendor quotation).

Software ecosystem adaptation cost. According to official Ascend community documentation, the software stack of domestic computing platforms differs significantly from the CUDA ecosystem. Model migration, operator optimization, and framework adaptation all require additional engineering investment. In Mingxin's R9 measurements on the Huawei Atlas 910B platform, model inference loading acceleration (vs NFS) reached 6.2–9.3×, but this result is built on the adaptation work already completed on the Ascend platform.

Storage Tiering: A Viable Innovation Path for Domestic KV Cache Chips

Facing these challenges, storage tiering architecture has proven to be a pragmatic technical path. The core concept is: tier KV Cache by access frequency, keeping hot data in memory and offloading cold data to high-speed storage devices, using intelligent scheduling to expand capacity boundaries while maintaining latency guarantees.

Mingxin FX100 measured data validates the effectiveness of this path. Under a 480B production-deployment long-context cold-restore workload, throughput improved +29% (lower bound) at concurrency 8, +40% (upper bound) at the optimal operating point of concurrency 16, and +35–36% (measured, reports R2/R3) at the full-machine level with TP4×2. This improvement does not come from the chip itself, but from the coordinated optimization between the storage layer and the inference framework.

The acceleration effect for scenarios without external memory recomputation is even more pronounced: the recomputation baseline TTFT p50 of 149.5s (conc16) compared to FX100's 11.85s, with throughput increasing from 4.1 to 74.9 tok/s, an acceleration factor of 8.6–20× (measured, report R2). This demonstrates that in long-context scenarios, the benefit of avoiding recomputation far exceeds that of pure storage read/write optimization.

Domestic Substitution: From Point Breakthroughs to Full-Stack Collaboration

Innovation in domestic KV Cache chips cannot proceed in isolation; it must form synergy with systems, frameworks, and applications. According to Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving, a KVCache-centric disaggregated architecture can further optimize resource utilization at the system level through prefix cache reuse and cross-node KV pooling. Combining this approach with the hardware characteristics of domestic chips may yield differentiated advantages.

In the practice of domestic substitution, Mingxin's collaboration model provides a risk-controlled validation path: approximately 10 weeks of gated joint testing, with G1 arrival acceptance, G2 single-machine baseline, G3 main gate (TTFT reduction ≥25%, throughput +29–40% measured in-band), and G4 72-hour stability testing, with stop-loss if targets are not met. This mechanism allows buyers to validate the feasibility of domestic solutions with small-scale investment, reducing substitution risk.

It is worth noting that, according to SGLang: Efficient Execution of Structured Language Model Programs, RadixAttention's prefix tree reuse mechanism can significantly improve hit rates in multi-turn dialogue and shared-prefix scenarios. If domestic chip manufacturers can provide targeted hardware-level optimization for such reuse mechanisms, they may develop competitiveness beyond general-purpose solutions in specific scenarios.

Conclusion

The technical challenges facing domestic KV Cache chips are systemic, but the innovation path is equally clear: resolve the capacity-cost contradiction through storage tiering, break through bandwidth bottlenecks through software-hardware co-design, and reduce substitution risk through gated joint testing. Mingxin Technology continues to invest in storage acceleration and domestic computing, with the FX100 series having completed multiple rounds of measured validation on both AMD and Ascend platforms. Customers with domestic substitution needs are welcome to contact us for joint testing collaboration.

Key Q&A

Q: What are the core technical challenges facing domestic KV Cache chips? A: Three challenges: memory bandwidth bottlenecks constraining KV Cache read speed; the contradiction between high HBM costs and capacity requirements; and a generational gap between the software ecosystem and CUDA, resulting in high adaptation costs.

Q: How much actual performance improvement does storage tiering bring to KV Cache inference? A: Mingxin FX100 achieved measured throughput improvements of 29–40% (measured, reports R2/R3) on a 480B model long-context cold-restore workload; for scenarios without external memory recomputation, acceleration reached 8.6–20× (measured, report R2).

Q: How can domestic substitution control technical validation risk? A: By adopting a gated joint testing model with phased acceptance (arrival acceptance, single-machine baseline, main gate, stability testing), setting clear pass criteria (e.g., TTFT reduction ≥25%), and applying stop-loss if targets are not met.

References

1. SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
2. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
3. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
4. Ascend Documentation - Ascend Community — https://www.hiascend.com/document
5. CANN - Ascend Heterogeneous Computing Architecture - Ascend Community — https://www.hiascend.com/software/cann