Technical Characteristics and Application Scenarios of Domestic KV Cache Software Solutions
Key Takeaways
The core value of domestic KV Cache software solutions lies in offloading KV Cache from GPU memory to high-speed storage and establishing a tiered caching architecture, which significantly reduces time-to-first-token (TTFT) and improves throughput in long-context and high-concurrency inference scenarios. Measured on the Mingxin FX100 with a 480B model in TP8 configuration, TTFT decreased by 26–32% and throughput improved by 29–40% (measured, reports R2/R3). These solutions are particularly suited for production-grade long-context services, multi-instance shared-prefix scenarios, and inference acceleration on domestic compute platforms (e.g., Ascend).
1. Technical Approach: Tiered Caching from GPU Memory to Storage
The GPU memory bottleneck in modern LLM inference has driven the adoption of paged management for KV Cache. According to Efficient Memory Management for Large Language Model Serving with PagedAttention, fragmentation of KV Cache in GPU memory is one of the primary constraints on serving throughput. Domestic KV Cache software solutions go a step further: rather than treating KV Cache as an adjunct to GPU memory, they manage it as an independent storage tier.
The technical approach of the Mingxin FX100 can be summarized in three tiers: a hot-data tier in GPU memory, a warm-data tier on high-speed NVMe-oF arrays, and an optional external cold-storage tier. During inference, KV Cache is automatically migrated across tiers based on access frequency. As an example from measured report R2, under a 480B·TP8 long-context workload, the FX100 offloaded KV Cache to an all-flash array (4-drive RAID0, RoCEv2), reducing TTFT p50 from a baseline of 10.17–35.73s (local single NVMe drive) to 7.53–26.35s (measured, report R2). The direct source of this improvement is the reduction in storage-side bandwidth and latency, which substantially lowers the recomputation cost on cache misses.
Notably, this approach echoes the architectural thinking of NVIDIA CMX. According to NVIDIA's official product page, CMX is defined as an AI-native context memory storage layer, with the same design goal of freeing KV Cache from GPU memory, building a pod-level KV pathway via BlueField-4 and Spectrum-X. The difference between domestic solutions and CMX lies in the hardware foundation: the former is adapted to domestic compute platforms (e.g., Ascend) and general-purpose x86 servers, while the latter is tied to the NVIDIA ecosystem. This distinction determines the irreplaceability of domestic solutions in Xinchuang (domestic IT substitution) scenarios.
2. Application Scenario 1: Latency Optimization for Long-Context Production Workloads
Long-context inference (e.g., 128K+ tokens) is the most direct application scenario for KV Cache software solutions. The longer the context, the larger the GPU memory footprint of KV Cache, and the higher the recomputation cost on cache misses. According to SGLang: Efficient Execution of Structured Language Model Programs, RadixAttention significantly improves hit rates in multi-turn dialogue and shared-prefix scenarios through a prefix-tree reuse mechanism—but this mechanism presupposes efficient access to KV Cache, which is precisely where storage-tier acceleration comes into play.
Measured report R2 from Mingxin covered three concurrency levels (8/16/32). In the 480B·TP8 configuration, TTFT p50 decreased by 26–32% (measured, report R2). The engineering significance of this metric is that TTFT is the user-perceived first-byte latency, directly affecting SLA compliance rates. At concurrency 16 (the optimal operating point), throughput improved by 40% (measured, report R3), meaning the same batch of GPUs can serve more concurrent requests or allocate a larger context budget per request.
It should be emphasized that the above figures are from Mingxin's own test reports (R2/R3), with the test platform being 8× AMD MI308X + vLLM + LMCache. Performance may vary across different hardware and software stacks; selection decisions should be based on joint testing results under actual workloads.
3. Application Scenario 2: Inference Acceleration on Domestic Compute Platforms
Another key scenario for domestic KV Cache software solutions is domestic compute platforms such as Ascend. According to official Ascend community documentation, the Ascend platform provides complete hardware form factors and software stack support; per CANN - Ascend Heterogeneous Computing Architecture, CANN is Ascend's heterogeneous computing architecture, positioned as the counterpart to the CUDA ecosystem. However, the maturity gap between domestic accelerator software stacks and the CUDA ecosystem makes adaptation and performance tuning of inference frameworks (e.g., vLLM) more complex.
Measured report R9 from Mingxin validated the FX100's acceleration effect on the Huawei Atlas 910B platform: model serving load time for DeepSeek-32B dropped from 691s to 112s (6.2×), and for DeepSeek-70B from 1399s to 150s (9.3×) (measured, report R9, vs. NFS baseline). Here, the acceleration target is model weight loading rather than KV Cache, but it reveals the same logic: on domestic platforms, storage is often the bottleneck in the inference pipeline, and NVMe-oF all-flash arrays can significantly compress data movement time.
For domestic substitution projects, the selection criteria should include three layers: whether the hardware form factor matches (U.2/E1.S), whether the software stack is compatible (CANN/ROCm/CUDA), and whether the storage protocol is supported (RoCEv2/NVMe-oF). According to openEuler official project information, domestic server operating systems (e.g., openEuler) have formed a complete Xinchuang software stack hierarchy, and KV Cache software solutions need to be adapted within this stack.
4. Application Scenario 3: Training Checkpoint Saving and Multi-Instance Shared Prefixes
Beyond inference, the architecture of KV Cache software solutions is equally applicable to checkpoint saving in training scenarios. Measured report R1 from Mingxin showed that in 8-GPU 32B LoRA training, the save time for each 65.6GB full-model snapshot dropped from 178s to 94s (1.9×), with sustained write bandwidth improving from 3.26 GB/s to 6.40 GB/s (+96%) (measured, report R1). The commonality in this scenario is that the higher the frequency of training interruption recovery, the larger the proportion of time spent on checkpoint saving, and the more significant the benefit of storage acceleration.
Multi-instance shared prefixes are another scenario worth attention. In measured report R4 (multi-instance configuration), the FX100's cache reuse capability for shared prefixes was validated (measured, report R4). This capability is particularly important in RAG (retrieval-augmented generation) and multi-turn agent dialogue scenarios—when multiple requests share the same system prompt or document prefix, the cache hit rate directly determines average latency.
5. Selection Boundaries and Uncertainties
KV Cache software solutions are not a panacea. The following boundary conditions should be clarified: First, the acceleration effect is strongly correlated with the cache miss rate—if the workload has an extremely low KV Cache hit rate (e.g., each request has a completely different context), the benefit of storage acceleration will diminish significantly. Second, the deployment cost of RoCEv2 networking and NVMe-oF arrays must be factored into total cost of ownership; according to an NVIDIA official blog, CMX delivers up to 5× throughput improvement over traditional storage, but this figure is vendor-reported and tied to proprietary hardware. Third, the software stack adaptation cycle on domestic platforms may be longer than expected; it is recommended to validate performance under real workloads during the joint testing phase (e.g., Mingxin's approximately 10-week gate-based joint testing process).
Key Q&A
Q: What are the core technical characteristics of domestic KV Cache software solutions? A: By tiering and offloading KV Cache from GPU memory to high-speed NVMe-oF storage, these solutions reduce TTFT and improve throughput in long-context and high-concurrency scenarios. Measured on the Mingxin FX100, TTFT decreased by 26–32% and throughput improved by 29–40% (measured, reports R2/R3).
Q: Which application scenarios are these solutions suitable for? A: Long-context production inference, model serving acceleration on domestic compute platforms (e.g., Ascend), training checkpoint saving, and multi-instance shared-prefix RAG/agent scenarios. Selection should account for the actual workload's cache miss rate and hardware form factor.
Q: What boundaries should be considered during selection? A: The acceleration effect depends on the KV Cache hit rate; the deployment cost of RoCEv2 and all-flash arrays must be included in total cost of ownership; the software stack adaptation cycle on domestic platforms may be lengthy, and it is recommended to validate real-world performance through gate-based joint testing (e.g., Mingxin's approximately 10-week process).
References
- SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
- Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
- NVIDIA CMX Context Memory Storage Platform — https://www.nvidia.com/en-us/data-center/ai-storage/cmx/
- Ascend Documentation - Ascend Community — https://www.hiascend.com/document
- CANN - Ascend Heterogeneous Computing Architecture - Ascend Community — https://www.hiascend.com/software/cann
- openEuler | OS for Digital Infrastructure — https://www.openeuler.org/en/
- Introducing NVIDIA BlueField-4-Powered Inference Context Memory Storage Platform for the Next Frontier of AI — https://developer.nvidia.com/blog/introducing-nvidia-bluefield-4-powered-inference-context-memory-storage-platform-for-the-next-frontier-of-ai/