Building a Domestic AI Inference Storage Acceleration Ecosystem

Building a domestic AI inference storage acceleration ecosystem requires coordinated advancement across three layers: hardware, software, and standards. At the hardware layer, the disaggregated storage-compute architecture based on NVMe-oF and RoCEv2 has become the mainstream choice; at the software layer, acceleration optimizations targeting KV Cache are emerging as the key breakthrough for inference performance; at the standards layer, adaptation of domestic operating systems and storage interfaces is a necessary prerequisite for ecosystem deployment. Measured data from the Mingxin FX100 under long-context workloads with a 480B model demonstrates that this path delivers quantifiable performance gains.

Hardware Layer: Disaggregated Storage-Compute and High-Speed Interconnect as the Foundation

Building a domestic AI inference storage acceleration ecosystem first requires solving the data path problem. Traditional local NVMe direct-attached solutions face bottlenecks of low resource utilization and limited scalability in GPU cluster scenarios. According to the NVIDIA GPUDirect Storage Documentation, GPU direct-attached storage technology establishes a direct data path between GPUs and storage devices by bypassing the CPU's bounce buffer—a mechanism that provides the hardware foundation for disaggregated storage-compute architectures.

In the domestic context, this architecture must adapt to the specific constraints of domestic servers and operating systems. Official project information from domestic operating systems such as openEuler shows that the Xinchuang software/hardware stack has formed a complete hierarchy from kernel to user space, and storage acceleration solutions must be deeply compatible with this stack. The Mingxin FX100 employs an all-flash NVMe-oF array with RoCEv2 networking, and has completed systematic validation on an 8-GPU AMD MI308X platform. This configuration path can serve as a reference baseline for domestic substitution.

Software Layer: KV Cache Acceleration as the Optimal Entry Point

The core bottleneck in software-level inference storage acceleration optimization lies in KV Cache memory access. According to the FlashAttention paper, attention computation is limited by HBM bandwidth rather than compute capacity, and this IO-aware optimization logic applies equally to the external memory tiering of KV Cache. When KV Cache size exceeds GPU memory capacity, tiering and offloading it to remote storage with efficient readback becomes the key to improving long-context inference throughput.

Measured data from Mingxin report R2 shows that under long-context cold-restore workloads in a 480B production deployment configuration, KV tiering acceleration delivers inference throughput gains of +29–40%: +29% at concurrency level 8 (lower bound), +40% at the optimal operating point of concurrency 16 (upper bound), and +35–36% at the full-machine TP4×2 level. Time-to-first-token (TTFT) decreases by 26–32% under 480B·TP8 across three concurrency levels, with p50 dropping from 10.17–35.73s to 7.53–26.35s. This gain comes directly from fast KV Cache readback capability rather than simple cache hits—against a baseline without external memory recomputation, the speedup factor reaches 8.6–20× (recompute baseline TTFT p50 is 149.5s, compared to FX100's 11.85s; throughput of 4.1 vs. 74.9 tok/s).

Another key software-layer optimization is KV Cache paging management. According to the PagedAttention paper, paged management of KV Cache resolves GPU memory fragmentation issues and significantly improves memory utilization. Mingxin's measurements on the LMCache parallel-read patch further confirm this direction: under single-GPU concurrency-16 cold disk-read scenarios (Qwen2.5-32B), TTFT drops from 37.97s to 9.30s (4.1× improvement), and bandwidth increases from 0.98 to 5.23 GB/s (5.3×). This indicates that parallel read optimization on the storage side and paging management on the GPU memory side must be combined to unlock full performance.

Standards and Ecosystem: Neutral Benchmarks and Reproducible Validation Are Both Essential

The third layer of ecosystem building is the standards and validation system. Inference performance comparisons must rely on neutral, public benchmarks. According to the MLPerf Inference: Datacenter Benchmark Suite Results, this evaluation requires submissions under fixed precision and latency constraints, making it the only neutral public reference for discussing inference performance. Performance claims from domestic storage acceleration solutions should proactively align with the methodology of such public benchmarks rather than self-defined metrics.

At the same time, design trade-offs in disaggregated storage-compute architectures require industry consensus. According to the Mooncake paper, the KVCache-centric disaggregated storage-compute architecture involves design trade-offs between prefix cache reuse and cross-node KV pooling, and these trade-offs directly affect the applicability boundaries of storage acceleration solutions. SNIA's definitions of storage tiering and computational storage provide the terminology and framework foundation for this discussion.

Mingxin employs an approximately 10-week gate-based validation process in joint testing (G1 arrival acceptance / G2 single-node baseline / G3 main gate: TTFT reduction ≥25%, throughput +29–40% measured in-band / G4 72-hour stability), with stop-loss if targets are not met. The value of this mechanism lies in converting performance claims into reproducible measured data rather than relying on vendor self-reported figures. For procurement and budget decision-makers, this is the most critical aspect to evaluate when assessing domestic solutions.

Conclusion

Building a domestic AI inference storage acceleration ecosystem is not a matter of replacing a single product, but rather the coordinated deployment of three layers: hardware paths, software optimization, and standards validation. Measured data from the Mingxin FX100 demonstrates that KV Cache tiering acceleration delivers 29–40% throughput gains and 26–32% TTFT reductions in long-context inference scenarios, with these benefits validated at the 480B model scale. To verify the suitability of this architecture on your own platform, reproducible measurement models can be obtained through the joint testing mechanism.

Key Q&A

Q: What is the core path for building a domestic AI inference storage acceleration ecosystem? A: At the hardware layer, adopt a disaggregated storage-compute architecture using NVMe-oF and RoCEv2; at the software layer, focus on KV Cache tiering and parallel read optimization; at the standards layer, validate performance claims through neutral benchmarks and gate-based joint testing. The three layers advance in coordination.

Q: What are the measured gains of KV Cache acceleration in long-context inference? A: Mingxin R2 measurements show throughput gains of 29–40% under long-context cold-restore workloads with a 480B model (+29% at concurrency 8, +40% at concurrency 16), TTFT reductions of 26–32%, and speedup factors of 8.6–20× against the baseline without external memory recomputation.

Q: How can the real performance of domestic storage acceleration solutions be validated? A: Use a gate-based joint testing process with TTFT reduction ≥25% and throughput +29–40% measured in-band as core gates, supplemented by 72-hour stability validation, ensuring performance claims are reproducible and auditable.

References

1. NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html
2. openEuler | OS for Digital Infrastructure — https://www.openeuler.org/en/
3. Epoch AI — https://epoch.ai/
4. SNIA — Storage Networking Industry Association — https://www.snia.org/
5. MLPerf Inference: Datacenter Benchmark Suite Results — https://mlcommons.org/benchmarks/inference-datacenter/
6. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — https://arxiv.org/abs/2205.14135
7. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
8. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180