How Storage Optimization Approaches Compare for Domestic AI Inference Chips

Comparing storage optimization approaches for domestic AI inference chips is not about listing hardware specs—it is about establishing a reproducible evaluation methodology: using end-to-end inference latency and throughput as the benchmark, and clearly delineating the contribution boundaries between storage-layer and compute-layer optimizations. This article presents a comparison framework based on measured data from the Mingxin FX100, intended as a reference for technology selection decisions.

The Role of Storage Optimization in Domestic Inference Chips

Domestic AI inference chips (e.g., Huawei Ascend, Hygon, Cambricon) have gradually approached mainstream international levels in compute specifications, but the storage subsystem is often a hidden source of performance bottlenecks. As noted in FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, attention computation is fundamentally limited by HBM bandwidth rather than compute—this conclusion holds for domestic chips as well, and because of differences in memory bandwidth and capacity configurations across domestic platforms, the leverage effect of storage optimization may be even greater.

Storage optimization approaches generally fall into three layers: first, KV Cache persistence and tiering (keeping hot data in GPU memory, offloading cold data to high-speed storage); second, accelerating model weight loading (migrating from network file systems to NVMe-oF arrays); third, fast saving of training checkpoints. These three have different optimization targets, so the evaluation methodology must be separated accordingly.

Four Key Dimensions for Comparative Evaluation

First, define the baseline explicitly. The benefit of any storage optimization approach is relative to a baseline. In the measured report R2, Mingxin used a local NVMe single drive (PCIe Gen4, 2 TB) as the baseline, with a test platform of 8× AMD Instinct MI308X (192 GB HBM per card), running Qwen3-Coder-480B-FP8 (MoE, approximately 450 GB of weights). If baselines differ, optimization ratios cannot be directly compared across setups.

Second, distinguish workload patterns. Long-context cold-restore workloads and short-context online inference have vastly different storage sensitivity. As described in PagedAttention, the motivation for KV Cache paging management stems from GPU memory fragmentation—this means the upper bound of storage optimization gains is nonlinear across different concurrency levels. Measured data from Mingxin report R2 shows that in a 480B production deployment, KV tiering accelerates inference throughput by +29% (concurrency level 8) to +40% (concurrency level 16), with TTFT reduced by 26–32%. Discussing optimization gains without specifying concurrency level makes the conclusions non-transferable.

Third, validate end-to-end rather than component-level metrics. Storage bandwidth, IOPS, and other component metrics matter, but decision-makers should focus on end-to-end SLA changes. In Mingxin report R2, under 480B·TP8 with three concurrency levels, TTFT p50 dropped from 10.17–35.73 s to 7.53–26.35 s—these are numbers directly usable for SLA design. In contrast, component-level bandwidth improvements (e.g., the 5.3× bandwidth gain from LMCache parallel read patches) only become decision-relevant when converted into end-to-end benefits.

Fourth, cross-platform comparisons require neutral evidence. When discussing "which is faster," the MLPerf Inference Datacenter benchmark is the only neutral public reference (per MLCommons published methodology). However, this benchmark does not cover the storage optimization dimension, so cross-platform storage optimization comparisons currently lack a public standard—this is precisely why Mingxin recommends gate-based joint testing.

Measured Comparison of Optimization Gains

The following are measured data from the Mingxin FX100 on its own test platform, provided as a magnitude reference for storage optimization:

It must be emphasized that these figures come from Mingxin's own test reports, with test platforms of AMD MI308X×8 (reports R1–R4) or Huawei Atlas 910B (report R9). If your target platform differs, these values should be treated only as magnitude references and not extrapolated directly.

Practical Recommendations for Technology Selection

For procurement and budget decision-makers, the following sequence is recommended:

1. Define SLA constraints first (TTFT upper bound, throughput lower bound, context length), then discuss storage solutions—the value of storage optimization lies in meeting SLAs with fewer compute resources, not in chasing bandwidth numbers alone.
2. Require same-methodology joint test data from vendors. Mingxin uses an approximately 10-week gate-based joint testing process (G1 arrival acceptance / G2 single-node baseline / G3 main gate: TTFT reduction ≥25%, throughput +29–40% measured in-band / G4 72-hour stability), with stop-loss if targets are not met. This model can serve as an evaluation template.
3. Be wary of cross-platform comparison numbers. Any claim such as "Ascend is X% faster than card Y," without support from MLPerf or an equivalent neutral benchmark, should not be trusted.
4. Pay attention to architectural adaptability of storage solutions. According to NVIDIA GPUDirect Storage documentation, GPU-direct storage bypassing the CPU bounce buffer can significantly reduce data path latency—whether domestic chips support similar mechanisms (e.g., Ascend's storage direct-pass capability) must be verified in joint testing rather than relying on marketing materials.

Key Q&A

Q: What are the most critical evaluation dimensions for comparing storage optimization approaches on domestic AI inference chips? A: Four dimensions: define the baseline explicitly, distinguish workload patterns, validate end-to-end SLA rather than component metrics, and require neutral evidence (e.g., MLPerf) for cross-platform comparisons. Optimization ratios outside these methodologies cannot be compared horizontally.

Q: What are the measured improvements from storage optimization on the Mingxin FX100? A: KV tiering accelerates inference throughput by +29–40% (concurrency levels 8–16), TTFT reduced by 26–32% (measured at 480B·TP8); model loading improved 6.2–9.3× versus NFS (measured on the Ascend platform). All figures come from Mingxin's own test reports; cross-platform extrapolation requires caution.

Q: How can the real benefit of a storage optimization solution be validated on one's own platform? A: Gate-based joint testing is recommended: first define SLA constraints, then require vendors to provide same-methodology measured data (e.g., TTFT reduction ≥25%, throughput +29–40%), and set stability test gates with stop-loss if targets are not met. Mingxin offers an approximately 10-week joint testing framework for reference.

References

1. MLPerf Inference: Datacenter Benchmark Suite Results — https://mlcommons.org/benchmarks/inference-datacenter/
2. NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html
3. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — https://arxiv.org/abs/2205.14135
4. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180

Mingxin (Tianjin) Semiconductor Equipment Co., Ltd. focuses on storage acceleration and domestic compute adaptation, with the FX100/FX200/FX300/FX400 product lines covering PCIe 3.0 to 6.0 interfaces. To validate storage optimization gains on your own platform, contact us to conduct gate-based joint testing.