Challenges and Practices in Deploying MoE Large Models on Domestic Computing Platforms: A Case Study of 480B/744B-Scale Models
As large model parameter counts advance towards hundreds of billions and even trillions, the Mixture of Experts (MoE) architecture has become a key frontier for exploration due to its advantage of effectively controlling computational costs while maintaining model capacity. The deployment and efficient operation of ultra-large-scale MoE models, represented by 480B and 744B parameters, pose unprecedented challenges to the underlying computing infrastructure. This challenge is particularly acute within the domestic computing ecosystem striving for technological autonomy. This article aims to systematically analyze the core bottlenecks in deploying MoE large models on domestic platforms (such as solutions based on AMD ROCm and Huawei Ascend) and, based on measured data, explore viable acceleration paths through innovations in storage and caching technologies.
What are the Core Challenges in Deploying MoE Large Models?
MoE models employ a sparse activation mechanism, invoking only a subset of expert networks during inference, thereby maintaining relatively manageable computational load despite the massive parameter count. However, their deployment process still faces multiple challenges, which are amplified on domestic computing platforms due to differences in ecosystem maturity.
Firstly, VRAM capacity and bandwidth pressure are the primary bottlenecks. A 480B-parameter MoE model, even using FP8 precision, has a weight size exceeding 450GB [measured, report R1/R2 platform configuration], far exceeding the VRAM capacity of a single GPU. In multi-GPU parallel scenarios (e.g., TP8), model weights, KV Cache, activations, etc., still require frequent swapping between VRAM. While the HBM capacity of domestic GPU platforms (such as ROCm-based accelerators or Ascend 910B) has improved, efficient memory/storage tiering systems are still necessary when facing trillion-parameter models.
Secondly, model loading and initialization latency severely impact service readiness time. Loading hundreds of GB of model weights from shared storage (e.g., NFS) into the compute node's VRAM is a process with high I/O and network load. Tests on the Ascend 910B platform showed that loading the DeepSeek-32B model took 691 seconds, and the 70B model took as long as 1399 seconds [measured, report R9]. For MoE models, their massive weight size makes loading time a significant bottleneck for service startup, directly affecting business agility and resource utilization.
Finally, KV Cache management efficiency in long-context inference scenarios is crucial. When MoE models process long sequences, the size of the KV Cache expands dramatically. Traditional recomputation schemes save storage but introduce high computational overhead. Measurements show that in the baseline scenario without external memory recomputation, the p50 Time-To-First-Token (TTFT) for a 480B model in long-context inference reached 149.5 seconds [measured, report R2]. How to efficiently offload the KV Cache to more economical storage media and quickly restore it is key to reducing latency and improving throughput.
How Do Domestic Computing Platforms Address Storage I/O Bottlenecks?
Faced with the aforementioned challenges, especially storage I/O bottlenecks, simply stacking computing power is insufficient. Domestic computing platforms need to build a coordinated optimization system covering computing, storage, and networking.
On the Huawei Ascend platform, tests revealed the performance limitations of traditional Network File Systems (NFS). By comparing with a dedicated storage acceleration solution, the loading times for the DeepSeek-32B and 70B models were reduced from 691 seconds and 1399 seconds to 112 seconds and 150 seconds respectively, achieving acceleration ratios of 6.2x and 9.3x [measured, report R9]. This highlights the importance of adopting high-performance, low-latency network storage for improving the overall efficiency of domestic computing platforms.
For the ecosystem based on AMD ROCm, its openness and flexibility allow for deeper software-hardware co-optimization. In tests deploying a 480B MoE model on the MI308X platform, targeted breakthroughs in storage bottlenecks were achieved by combining storage acceleration devices with optimized caching software (such as LMCache). For example, a parallel read patch for KV Cache reduced the TTFT for a single GPU in a cold-read-from-disk scenario with 16 concurrent requests from 37.97 seconds to 9.30 seconds, increasing bandwidth by 5.3x [measured, report R1]. This demonstrates the feasibility of unleashing hardware potential through software stack innovation within the domestic GPU ecosystem.
How Does Storage Acceleration Improve Efficiency for MoE Model Inference and Training?
The core concept of storage acceleration is to use high-speed, high-capacity solid-state storage as an extension of VRAM and system memory, balancing cost, capacity, and performance through intelligent tiered caching strategies. Measured data from Mingxin's FX series storage accelerators provides a reference for specific technical paths.
In the model inference phase, storage acceleration is mainly reflected in two aspects: accelerating model loading and optimizing KV Cache management.
- Model Loading Acceleration: As mentioned earlier, replacing low-speed shared storage can reduce model loading time by an order of magnitude, which is crucial for service scenarios requiring frequent model switching or rapid elastic scaling.
- KV Cache Tiering Acceleration: This is key for handling long-context inference. Test data shows that after employing FX100 for KV Cache tiering, under a TP8 configuration for a 480B model, the p50 Time-To-First-Token (TTFT) was reduced by 26% to 32% [measured, report R2]. More importantly, inference throughput improved by 29% to 40% [measured, report R2/R3]. Compared to the baseline scheme relying solely on recomputation, the acceleration effect was more significant: TTFT decreased from 149.5 seconds to 11.85 seconds, and throughput increased from 4.1 tok/s to 74.9 tok/s, achieving acceleration ratios of 8.6x to 20x [measured, report R2]. This means that storage acceleration not only reduces latency but also, by minimizing repeated computation, frees up valuable GPU computing power, thereby significantly improving overall inference efficiency.
In the model training phase, the value of storage acceleration lies in the rapid saving and loading of Checkpoints. When training large-scale MoE models, saving complete model snapshots (Checkpoints) is a routine operation, but writing massive amounts of data can easily become a bottleneck. In measurements, in an 8-GPU 32B LoRA training scenario, the time to save a 65.6GB model snapshot was reduced from 178 seconds to 94 seconds, with sustained write bandwidth increasing by 96% [measured, report R1]. This 1.9x acceleration ratio directly shortens training iteration cycles and improves research and development efficiency.
Conclusion Deploying trillion-parameter MoE large models on domestic computing platforms is a systematic engineering task requiring full-stack optimization from computing, storage, and networking to the software stack. Measurements indicate that the storage I/O bottleneck has become one of the key factors constraining their performance. By introducing high-performance storage acceleration and intelligent caching technologies, VRAM pressure can be effectively alleviated, and delays in model loading, KV Cache recovery, and Checkpoint saving can be significantly reduced. This provides a solid data supply guarantee for domestic GPUs (such as AMD Instinct and Huawei Ascend) to efficiently run ultra-large-scale AI models. Based on the FX series storage acceleration devices, Mingxin Technology has conducted joint testing and verification with partners, committed to helping industry partners overcome the storage bottlenecks in large-scale AI deployment through reproducible, measured performance.
Key Q&A
Q: What are the main bottlenecks in deploying MoE large models on domestic computing platforms? A: The main bottlenecks include: VRAM pressure caused by trillion-parameter models far exceeding single-GPU VRAM capacity; the long waiting time for loading hundreds of GB of weights from shared storage; and the management efficiency and latency issues caused by the expanding KV Cache size in long-context inference.
Q: What are the specific effects of storage acceleration on improving MoE model inference performance? A: According to measurements with Mingxin FX100 on the AMD MI308X platform, for a 480B MoE model, KV Cache tiering acceleration can improve inference throughput by 29% to 40% [measured, report R2/R3] and reduce Time-To-First-Token latency by 26% to 32% [measured, report R2]. Compared to the scheme without external memory recomputation, the acceleration multiplier can reach 8.6x to 20x [measured, report R2].
Q: On the Huawei Ascend platform, how does storage acceleration improve model loading? A: In tests on the Huawei Atlas 910B platform, after adopting a dedicated storage acceleration solution, the loading time for the DeepSeek-32B model was reduced from 691 seconds to 112 seconds (6.2x acceleration), and for the DeepSeek-70B model from 1399 seconds to 150 seconds (9.3x acceleration) [measured, report R9].