Deploying MoE Large Models on Domestic Platforms: Inference Acceleration Paths for 480B/744B Models

With the increasing adoption of Mixture of Experts (MoE) architectures in 480B/744B large models, domestic computing platforms—whether the AMD ROCm ecosystem or Huawei Ascend 910B—face a common challenge: the KV Cache storage bottleneck in long-context inference. While MoE models reduce computation through sparse activation of expert parameters, the KV Cache size grows linearly with sequence length. In scenarios such as cold starts, long conversation recovery, or high concurrency, GPU memory is insufficient to cache all historical states, leading to frequent disk I/O readbacks. Based on measured data from Mingxin FX100 on AMD MI308X and Huawei 910B, this article demonstrates a validated acceleration path: optimizing KV Cache tiered acceleration via an all-flash NVMe-oF storage array can achieve 26–40% end-to-end performance improvement on domestic platforms.

Why Do MoE Large Models Face KV Cache Bottlenecks on Domestic Platforms?

During inference of MoE models (e.g., 480B Qwen3-Coder-FP8, with weights ~450 GB), each token activates only a subset of experts, but the KV Cache must still store attention key-value pairs for all sequences. For example, with a 480B model deployed as TP8 and 16 concurrent sessions, a single long-context cold recovery (e.g., 128K context) can require tens of GB of KV Cache, far exceeding the per-GPU memory (e.g., 192 GB HBM on MI308X). When memory is insufficient, the system must offload part of the KV Cache to external storage—a step that can become a performance bottleneck on traditional NFS or local NVMe.

The software stacks of domestic computing platforms (ROCm, Ascend) have not yet been as deeply optimized for storage paths as NVIDIA GPUDirect Storage, making storage latency more impactful on time to first token (TTFT). Our tests show that with a single local NVMe drive as baseline, the 480B model at 8 concurrent sessions achieves a TTFT p50 of 10.17 seconds. However, when relying entirely on disk readback (no external memory recomputation), TTFT can surge to 149.5 seconds (16 concurrent sessions), and throughput drops from 74.9 tok/s to 4.1 tok/s [source: measured, report R2]. This demonstrates that storage I/O is a key bottleneck for deploying MoE models on domestic platforms.

How Does Storage Acceleration Solve the KV Cache Bottleneck? Measured Data Analysis

Tests with the Mingxin FX100 all-flash NVMe-oF array (4-disk RAID0, 14 TB, RoCEv2, single-port 100 GbE) on the AMD MI308X platform provide quantitative reference. The core conclusion is: by tiering KV Cache acceleration (migrating part of the cache from local NVMe to low-latency network storage), performance can be significantly improved without sacrificing model accuracy.

1. Inference Throughput Improvement of 29–40%

In long-context cold recovery workloads under 480B production deployment, FX100 achieved significant throughput gains over the local NVMe baseline. Specifically:

• At 8 concurrent sessions, throughput improved by 29% (lower bound);
• At the optimal operating point of 16 concurrent sessions, throughput improved by 40% (upper bound);
• Under the full-machine TP4×2 configuration, throughput improved by 35–36% [source: measured, reports R2/R3].

This improvement stems from FX100’s high bandwidth (measured read bandwidth 5.23 GB/s, 5.3× higher than local NVMe’s 0.98 GB/s) and low-latency network protocol (RoCEv2). For MoE models, due to irregular expert activation patterns, KV Cache access exhibits random read/write characteristics, and NVMe-oF’s parallel read capability aligns well with this requirement.

2. First Token Latency Reduction of 26–32%

TTFT is a critical metric for interactive applications (e.g., chat, code generation). In tests with 480B·TP8 and three concurrent sessions, FX100 reduced TTFT p50 from 10.17–35.73 seconds to 7.53–26.35 seconds, a reduction of 26–32% [source: measured, report R2]. For scenarios relying entirely on disk readback (no external memory recomputation), the improvement is even more dramatic: TTFT dropped from 149.5 seconds (16 concurrent sessions) to 11.85 seconds, an acceleration factor of 12.6× [source: measured, report R2].

3. Model Loading Acceleration of 6.2–9.3× (Ascend Platform)

On the Huawei Atlas 910B platform, FX100’s acceleration of model inference loading compared to the NFS baseline is particularly striking:

• DeepSeek-32B service loading time decreased from 691 seconds to 112 seconds (6.2×);
• DeepSeek-70B from 1399 seconds to 150 seconds (9.3×) [source: measured, report R9 (Ascend platform)].

This data indicates that storage acceleration is cross-platform universal on domestic GPUs—whether ROCm or Ascend, storage I/O is a common optimizable link.

Practical Path for Domestic Computing Deployment: From Gate Testing to Scale

Based on the above data, we recommend the following steps for deploying MoE large models on domestic platforms:

• Step 1: Identify the bottleneck. Use profiling tools (e.g., ROCProfiler, Ascend MindStudio) to determine the proportion of KV Cache disk readback. If I/O wait time in TTFT exceeds 30%, prioritize storage acceleration.
• Step 2: Gate-based validation. Follow the approximately 10-week joint testing process provided by Mingxin (G1: arrival acceptance → G2: single-machine baseline → G3: main gate: TTFT reduction ≥25%, throughput +29–40% measured in-band → G4: 72-hour stability test) to reproduce acceleration effects in your own environment. This model allows users to verify benefits without assuming upfront risk.
• Step 3: Scale up. For 744B MoE models (weights ~700 GB), consider higher-bandwidth NVMe-oF solutions (e.g., FX200 with 200 Gb interface or FX300 with 400 Gb interface). According to Mingxin’s product line, the full-configuration reference price for FX200 is ¥331,200 (approximately ¥1,797/TB), offering better cost-effectiveness compared to FX100’s ¥2,014/TB.

Conclusion

The core challenge of deploying MoE large models on domestic computing platforms is not computational power, but storage I/O support for KV Cache. Measured data from Mingxin FX100 on AMD MI308X and Huawei 910B shows that optimizing the storage path with an all-flash NVMe-oF array can achieve 26–40% inference performance improvement and significantly reduce model loading time. For technical decision-makers evaluating domestic computing solutions, we recommend prioritizing verification of storage acceleration benefits—this may be the optimization step with the highest return on investment. For further details on joint testing or to obtain a Python-reproducible measurement model, please contact the Mingxin technical team.

Key Q&A

Q: What is the main performance bottleneck for MoE large models on domestic GPU platforms?
A: The main bottleneck is KV Cache disk I/O readback. The KV Cache size of MoE models grows with context length, and when GPU memory is insufficient, frequent external storage reads/writes cause a significant increase in time to first token (TTFT), which can surge to 149.5 seconds in measured tests.

Q: What are the measured acceleration effects of Mingxin FX100 on domestic platforms?
A: On the AMD MI308X platform, FX100 improves inference throughput of the 480B model by 29–40% and reduces first token latency by 26–32%; on the Huawei 910B platform, model loading time is shortened by 6.2–9.3× (vs. NFS baseline) [source: measured, reports R2/R3/R9].

Q: How can I verify whether the storage acceleration solution is suitable for my deployment environment?
A: We recommend adopting a gate-based joint testing process, with acceptance criteria of TTFT reduction ≥25% and throughput improvement ≥29%. Mingxin provides an approximately 10-week testing cycle, including a 72-hour stability test, with the option to stop if targets are not met.