Technical Challenges and Innovation Directions for Domestic KV Cache Chips

Domestic KV Cache chips are at a critical technical fork: simply stacking compute power can no longer solve the memory wall problem in large-model inference, while innovations in KV Cache storage and interconnect architecture are becoming the main battlefield for domestic chips to achieve differentiated breakthroughs. Measured data from Mingxin's FX100 on AMD MI308X platforms shows that by tiered offloading of KV Cache to NVMe-oF all-flash arrays, long-context inference throughput for a 480B model can be improved by 29–40% [measured, report R2/R3]. This result reveals the strategic value of the storage subsystem in the domestic compute stack.

Why Domestic Chips Cannot Avoid the KV Cache Hurdle

The memory bottleneck in large-model inference is an industry consensus. According to "Efficient Memory Management for Large Language Model Serving with PagedAttention," paged management of KV Cache is the core mechanism for solving memory fragmentation, and mainstream inference frameworks such as vLLM are built on this foundation. However, domestic chips face more severe challenges: constrained HBM supply, and single-card memory capacity and bandwidth that lag leading products by a generation. This means that at equivalent model scales, KV Cache capacity on domestic cards reaches its limit earlier, forcing more frequent memory swapping.

Mingxin observed a critical phenomenon in R2 testing: under a 480B·TP8 long-context workload, the baseline TTFT p50 without external memory recomputation reached as high as 149.5 seconds (at concurrency level 16), while it dropped to 11.85 seconds after connecting the FX100 tiered storage [measured, report R2]. The significance of this data lies in quantifying the catastrophic degradation caused by insufficient KV Cache capacity—when the cache cannot hold the data, the model is forced to recompute entire segments, and latency degrades from seconds to minutes. If domestic chips are to support long-context production workloads, this physical constraint must be addressed head-on.

Why Tiered Storage Is the Practical Path for Domestic Chips

Offloading KV Cache from HBM to external storage is not a new concept. According to "Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving," the KVCache-centric disaggregated architecture alleviates single-card capacity pressure through prefix cache reuse and cross-node KV pooling. However, Mooncake's architecture inherently relies on high-speed RDMA networks and centralized cache pools, which places high demands on interconnect bandwidth in domestic clusters.

Mingxin's differentiated approach lies in tiered offloading of KV Cache to NVMe-oF all-flash arrays, bypassing the latency overhead of traditional file systems through a dedicated hardware path. In R1 testing, the LMCache parallel read patch under a single-card, concurrency-16, cold-read scenario (Qwen2.5-32B) reduced TTFT from 37.97 seconds to 9.30 seconds, and increased bandwidth from 0.98 GB/s to 5.23 GB/s [measured, report R1]. The deeper implication of this result is that even in a single-card configuration, storage-side optimization can deliver a 4.1x improvement in TTFT without modifying the model or inference framework—which is particularly important for domestic platforms with relatively thinner software stacks.

It is worth noting that NVIDIA proposed the CMX Context Memory Storage Platform in the G3.5 architecture. According to its official page, the platform is positioned as an AI-native context memory storage layer, claiming up to 5x higher throughput and up to 5x better power efficiency compared to traditional storage. According to the NVIDIA Technical Blog, CMX uses BlueField-4 and DOCA Memos to prefetch KV back to HBM, working in coordination with Dynamo/NIXL/Grove. This confirms that tiered KV storage is becoming a mainstream industry direction, but NVIDIA's path depends on its proprietary hardware ecosystem (Spectrum-X networking, BlueField DPUs), which domestic chips cannot directly replicate and must address through their own technical route.

Three Innovation Entry Points for Domestic KV Cache Chips

Based on Mingxin's engineering practice with the FX100 series, we believe innovation in domestic KV Cache chips should focus on three levels:

First, deep optimization of the storage protocol stack. Domestic platforms (such as Ascend) have significant differences in software stacks compared to the CUDA ecosystem. According to the official positioning of "CANN-Ascend Heterogeneous Computing Architecture," CANN provides the core software stack for heterogeneous computing, but optimization of its storage access paths still requires coordination with specific hardware. Mingxin's testing on the Ascend 910B platform shows that by replacing NFS with FX100-HBMM as the model loading storage, DeepSeek-70B service loading time dropped from 1399 seconds to 150 seconds (a 9.3x speedup) [measured, report R9]. This result demonstrates that domestic platforms are not inherently incapable of performance—rather, the storage subsystem has long been overlooked.

Second, semantic awareness of KV data. According to "SGLang: Efficient Execution of Structured Language Model Programs," RadixAttention reuses shared prefixes through a prefix tree, significantly improving hit rates in multi-turn conversation scenarios. If domestic KV Cache chips can perceive the semantic structure of KV at the hardware level (rather than treating it merely as a block device), they can make better decisions on cache replacement policies and prefetch timing. In Mingxin's R3 testing, throughput improvement reached 35–36% at the full-machine level under TP4×2 [measured, report R3], with part of the gains coming from targeted optimization of KV access patterns.

Third, coordination with the domestic interconnect ecosystem. Domestic cluster interconnects (such as Ascend's HCCS) differ from InfiniBand/RoCE in topology and semantics. The efficiency of cross-node KV Cache migration is directly limited by interconnect bandwidth and latency. Mingxin's R4 multi-instance testing validated the benefit boundaries under different concurrency configurations [measured, report R4], providing important reference for domestic chip designers: the benefits of KV offloading are not linear but strongly correlated with concurrency patterns and model scale, requiring co-design between hardware and schedulers.

Conclusion

The breakthrough point for domestic KV Cache chips lies not in compute stacking, but in system-level innovation in storage and interconnect. The Mingxin FX100 series (including FX100/FX200/FX300/FX400) achieves efficient tiered offloading of KV Cache through all-flash NVMe-oF arrays, with measured throughput improvements of 29–40% and TTFT reductions of 26–32% on 480B models [measured, report R2/R3], providing a reproducible performance optimization path for domestic inference platforms. We welcome compute centers and model vendors to conduct gated joint testing, using G1–G4 phased acceptance over approximately 10 weeks to verify the real-world performance of these benefits on their own workloads.

References

1. SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
2. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
3. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
4. NVIDIA CMX Context Memory Storage Platform — https://www.nvidia.com/en-us/data-center/ai-storage/cmx/
5. CANN-Ascend Heterogeneous Computing Architecture-Ascend Community — https://www.hiascend.com/software/cann
6. Introducing NVIDIA BlueField-4-Powered Inference Context Memory Storage Platform for the Next Frontier of AI — https://developer.nvidia.com/blog/introducing-nvidia-bluefield-4-powered-inference-context-memory-storage-platform-for-the-next-frontier-of-ai/

Key Q&A

Q: What are the core technical challenges facing domestic KV Cache chips? A: Insufficient KV Cache capacity caused by limited memory capacity and bandwidth, as well as cross-node migration efficiency issues arising from interconnect ecosystem differences. Mingxin's measurements show that without external memory recomputation, the TTFT p50 for a 480B model reaches as high as 149.5 seconds [measured, report R2].

Q: How significant are the benefits of tiered storage for domestic inference platforms? A: Mingxin's FX100 achieves throughput improvements of 29–40% and TTFT reductions of 26–32% on 480B models [measured, report R2/R3]; on the Ascend 910B platform, model loading is accelerated by 6.2–9.3x [measured, report R9].

Q: What are the innovation directions for domestic KV Cache chips? A: Three levels: deep optimization of the storage protocol stack, semantic awareness of KV data (similar to RadixAttention's prefix reuse), and co-design with the domestic interconnect ecosystem.