ROCm Ecosystem Performance on Non-NVIDIA Cards

With the evolution of the global compute landscape and the increasing demand for supply chain autonomy, domestic compute solutions and non-NVIDIA GPU ecosystems, represented by AMD, are accelerating their entry into the core scenarios of enterprise AI deployment. However, migrating from the CUDA ecosystem to ROCm or Ascend platforms is not a simple hardware replacement; it involves complex system engineering challenges such as inference stack adaptation, software compatibility, and storage I/O path reconstruction. This article aims to analyze the current engineering status of inference stacks for non-NVIDIA compute cards based on publicly available test data and industry practices, and to explore how to achieve production-grade deployment performance targets through full-stack optimization.

What are the Core Challenges in Inference Stack Adaptation for ROCm and Ascend Ecosystems?

Transitioning from the closed CUDA ecosystem to open platforms like ROCm or Ascend presents the primary challenge of software stack maturity and compatibility. ROCm, as AMD's open-source computing platform, has gradually improved support for mainstream frameworks like PyTorch and TensorFlow through continuous iteration. However, in critical aspects of inference deployment—such as high-performance inference engines (e.g., corresponding versions of vLLM, TensorRT-LLM), operator optimization, and toolchain completeness—gaps with the CUDA ecosystem remain. For example, achieving expected performance in specific model architectures or mixed-precision scenarios may require additional kernel development or parameter tuning.

Secondly, the storage I/O path becomes a significant performance bottleneck. In large language model inference, VRAM capacity limitations make it common for KV Cache to spill over to host memory or external storage. In the NVIDIA ecosystem, technologies like NVLink and GPUDirect Storage optimize data transfer paths to some extent. On ROCm or Ascend platforms, the lack of equally mature hardware protocols for GPU direct storage access necessitates multiple data copies via the PCIe bus and host memory, which can easily lead to substantial increases in Time To First Token (TTFT) latency and reduced throughput. Measured, report R2 data shows that without optimized storage, the baseline TTFT p50 for a cold recovery load of a 480B model with long context can reach 149.5 seconds, severely constraining service responsiveness.

How to Quantify and Optimize Storage I/O Bottlenecks in Heterogeneous Compute Environments?

To address the above bottlenecks, it is first necessary to accurately quantify the I/O bottlenecks. Tests indicate that when running the Qwen3-Coder-480B model on the AMD MI308X platform, KV Cache loading and model weight reading are the two main I/O loads. Measured, report R1 points out that in a cold read scenario with single-card concurrency of 16, the read bandwidth of a single local NVMe drive is only 0.98 GB/s, resulting in a TTFT of 37.97 seconds. This reveals that even with high-performance local SSDs, their bandwidth and access latency are insufficient to meet the real-time loading demands of multi-concurrent, large-capacity model parameters.

In response, reconstructing the I/O path through dedicated storage acceleration devices has become an effective solution. The Mingxin FX series all-flash NVMe-oF arrays provide high-bandwidth, low-latency block storage access via the RoCEv2 network. In the same test scenario mentioned above, after connecting the FX100, the read bandwidth increased to 5.23 GB/s (a 5.3x improvement), and TTFT was reduced to 9.30 seconds (a 4.1x improvement) [Source: Measured, report R1]. The principle behind this is that the FX array aggregates bandwidth through multi-disk RAID0 and provides near-local access semantics via the NVMe-oF protocol, bypassing some host-side protocol stack overhead and directly alleviating the bandwidth constraints on the storage side.

Furthermore, optimization for KV Cache tiered management has been implemented. Measured reports R2 and R3 show that through collaborative optimization between the software stack and FX storage, KV tiered acceleration was achieved in the production deployment form of a 480B model, resulting in inference throughput improvements of 29% to 40% and first token latency reductions of 26% to 32% [Source: Measured, reports R2/R3]. This indicates that by storing active KV Cache in high-speed shared storage, data exchange between GPU VRAM and host memory is significantly reduced, improving overall inference efficiency.

From Model Loading to Training Checkpoints: The Full-Scenario Value of Storage Acceleration

Storage I/O bottlenecks affect not only inference but also other stages of the AI workflow. During model service startup, loading large model weights from a Network File System (NFS) is time-consuming. Measured, report R9 shows that on the Huawei Ascend Atlas 910B platform, using FX100-HBMM compared to the NFS baseline reduced the model loading time for DeepSeek-32B from 691 seconds to 112 seconds (a 6.2x acceleration), and for DeepSeek-70B from 1399 seconds to 150 seconds (a 9.3x acceleration) [Source: Measured, report R9]. This greatly improves development iteration efficiency and service elastic scaling.

In model training scenarios, the saving and restoring of checkpoints are also constrained by storage bandwidth. Measured, report R1 shows that in an 8-card 32B LoRA training task, the saving time for each 65.6GB model snapshot was reduced from 178 seconds to 94 seconds (a 1.9x acceleration) after using FX100, with sustained write bandwidth increasing from 3.26 GB/s to 6.40 GB/s [Source: Measured, report R1]. This translates to shorter training interruption times and higher GPU resource utilization, which is particularly important for large-scale distributed training.

Conclusion

In summary, when embracing domestic compute solutions and diverse compute ecosystems like ROCm, it is essential to adopt a system-level perspective and incorporate storage I/O performance into the scope of full-stack optimization. Measured data indicates that pure hardware compute upgrades may see significantly diminished effectiveness if they encounter storage bottlenecks. By adopting storage acceleration solutions designed for high-performance computing, such as the Mingxin FX series, significant performance improvements can be achieved in multiple key scenarios including KV Cache management, model loading, and training checkpoints, thereby truly unlocking the potential value of non-NVIDIA compute cards. For technical teams planning heterogeneous compute deployments, it is recommended to conduct gated joint testing based on their own workloads to drive decisions with data, ensuring stability and efficiency in production environments. The joint testing cooperation model provided by Mingxin Technology aims to help customers verify the actual acceleration effects on specific models and compute platforms through a standardized testing process of approximately 10 weeks.

Q&A on Key Points of This Article

Q: What are the main storage I/O bottlenecks when deploying large models on the AMD ROCm platform? A: The main bottlenecks are high latency and low bandwidth caused by KV Cache spilling and model weight loading. Measured, report R1 shows that in a cold read scenario, the bandwidth of a single local NVMe drive is only 0.98 GB/s, resulting in a TTFT as high as 37.97 seconds. Through dedicated storage acceleration, bandwidth can be increased to 5.23 GB/s, and TTFT can be reduced to 9.30 seconds.

Q: What improvements does the Mingxin FX storage acceleration solution bring to model loading on the Ascend platform? A: The improvements are significant. Measured, report R9 indicates that on the Huawei Atlas 910B platform, compared to the NFS baseline, FX100-HBMM accelerates the loading time of DeepSeek-32B by 6.2x (691s → 112s) and DeepSeek-70B by 9.3x (1399s → 150s), greatly improving service startup efficiency.

Q: How to conduct effective performance evaluation and optimization for inference deployment on non-NVIDIA compute cards? A: It is recommended to adopt a systematic, full-stack evaluation, focusing on quantifying the impact of storage I/O on inference throughput (e.g., KV Cache management) and latency (e.g., TTFT). Conducting gated testing based on real workloads, such as verifying whether the TTFT reduction is ≥25% or whether throughput improvement reaches the measured range of 29-40%, is a key step in ensuring production-grade performance.