AMD MI308X Inference in Practice: What Can a Single Card with 192GB HBM Run?
With the continuous growth in demand for domestic computing power, heterogeneous computing platforms represented by the AMD Instinct MI series are becoming important alternatives to NVIDIA and Ascend. Among them, the MI308X, equipped with 192GB of High Bandwidth Memory (HBM) per card, is often seen as a potential hardware candidate for deploying large model inference tasks due to its memory capacity advantage. However, memory capacity is not the sole factor determining inference performance; memory bandwidth, software ecosystem compatibility, and storage I/O bottlenecks are equally critical. This article, based on Mingxin Technology's measured data on the AMD platform, aims to analyze the capability boundaries of a single MI308X card under actual inference workloads and explore how to overcome performance bottlenecks through storage acceleration technology.
Theoretical Model Deployment Boundaries of 192GB HBM Memory
The 192GB HBM memory of the AMD MI308X is its most prominent feature. Theoretically, this allows a single card to host models with larger parameter counts. Calculating at FP16 precision, approximately 2GB of memory is required to store the weights for every 1 billion parameters. Therefore, a single card can theoretically accommodate model weights of about 96B parameters (FP16). If more efficient INT8 quantization is employed, the accommodated parameter count can be further increased.
However, more factors need to be considered in practical deployment. During inference, in addition to model weights, memory must also be allocated for the input sequence (Prompt), the generated KV Cache, and intermediate activations. For long-context inference tasks, the memory footprint of the KV Cache can grow dramatically, becoming a key bottleneck that limits the actual model size or context length that can be run. For example, running a 32B parameter model for long-text generation could easily occupy tens of GB of memory for its KV Cache, significantly reducing the available headroom. Therefore, in practical applications, 192GB HBM is more likely to smoothly run models with parameter sizes below 70B for standard-length inference, or attempt models with tens of billions of parameters after quantization, but strict management of context length is required to avoid memory overflow.
Measured Performance: Throughput, Latency, and Memory Bottleneck Verification
Mingxin Technology conducted inference tests on large models such as Qwen3-Coder-480B-FP8 (MoE, weights approx. 450GB) in a test environment based on 8 MI308X cards (AMD EPYC 9654 platform, ROCm 7.2) [measured, reports R1, R2, R3]. The tests revealed the real challenges faced by single cards and even multi-card clusters when dealing with ultra-large-scale models.
In the baseline scenario without external acceleration (using a local single NVMe drive), when running a 480B model (TP8 card parallel) under a long-context cold recovery load, the first token latency (TTFT p50) ranged from 10.17 seconds to 35.73 seconds, with throughput performance limited by storage I/O and memory swapping efficiency [measured, report R2]. When the memory cannot fully accommodate the required KV Cache or frequently needs to load weights from external storage, performance degradation is significant. This confirms that relying solely on large-capacity HBM, if storage I/O becomes a bottleneck, still makes it difficult to fully utilize the computing power, especially for scenarios requiring fast cold starts or processing ultra-long contexts.
How to Break Through MI308X Inference Bottlenecks with Storage Acceleration?
To unleash the inference potential of hardware with large memory like the MI308X, the key bottleneck of storage I/O must be addressed. The Mingxin FX100 all-flash NVMe-oF acceleration array provides a targeted solution for this scenario. Its core principle is to nearly seamlessly map the low-latency, high-bandwidth I/O capabilities of a remote flash array to compute nodes via high-speed networking (RoCEv2) and optimized protocols.
In actual tests, the FX100 demonstrated multi-dimensional acceleration effects for inference tasks on the MI308X platform:
- KV Cache Tiering Acceleration Boosts Throughput: In long-context cold recovery load tests for the 480B model in a production deployment configuration, the FX100 increased inference throughput by 29% to 40% (optimal operating point) [measured, reports R2/R3]. The throughput improvement for the entire machine (TP4×2) was 35-36% [measured, report R3].
- Significantly Reduces First Token Latency: Under the same test conditions, TTFT p50 was reduced by 26% to 32%, from 10.17–35.73 seconds down to 7.53–26.35 seconds [measured, report R2]. This directly improves user experience and system responsiveness.
- Absolute Advantage Compared to No-External-Memory Recalculation: In "recalculation" scenarios requiring complete data loading from storage, the FX100 accelerated TTFT by 8.6x to 20x (baseline 149.5 seconds vs FX100 11.85 seconds, concurrency 16), and throughput increased from 4.1 tok/s to 74.9 tok/s [measured, report R2].
- Optimizes Model Loading and Training Checkpoints: Comparative tests on the Huawei Ascend 910B platform showed that the FX100 accelerated model service loading times for DeepSeek-32B/70B by 6.2x to 9.3x compared to traditional NFS [measured, report R9]. In training scenarios, checkpoint save speed increased by 1.9x, and sustained write bandwidth improved by 96% [measured, report R1].
These improvements stem from the FX100's high performance of up to 16M IOPS (FX100 measured) and 100Gb per interface bandwidth, effectively alleviating congestion during data exchange with external storage, allowing the MI308X's computing power to be more consistently used for computation rather than waiting for data.
Conclusion
In summary, the AMD MI308X, with its 192GB HBM memory, possesses the hardware foundation for deploying medium and large models in AI inference scenarios, but its actual efficiency is heavily constrained by storage I/O performance. Especially in complex scenarios like long-context and cold-start inference, storage bottlenecks can severely hinder computing power utilization. The Mingxin FX100 series storage acceleration devices, validated through actual measurements, can effectively strengthen this aspect, delivering significant improvements in throughput, latency, and model loading, thereby helping users more fully utilize the hardware investment in domestic computing power platforms like the MI308X. Mingxin Technology offers a gated joint testing cooperation model lasting approximately 10 weeks, providing reproducible test validation of acceleration effects, offering a measured path selection for building efficient and balanced inference infrastructure in computing centers.
Key Q&A
Q: How large a parameter model can a single AMD MI308X card with 192GB HBM run? A: Theoretically, at FP16 precision, it can accommodate model weights of about 96B parameters. However, practical deployment requires reserving memory for KV Cache and activations. In actual measurements, it is more suitable for smooth inference of models with parameters below 70B, or for quantized models with tens of billions of parameters with controlled context length.
Q: How does the storage I/O bottleneck affect MI308X inference performance? A: When model weights or KV Cache exceed the memory capacity and need to be swapped with external storage, low-speed I/O becomes a key bottleneck. Mingxin's R2 measurements show that in long-context scenarios with a 480B model, the storage bottleneck can cause first token latency (TTFT) to reach tens of seconds, severely limiting throughput.
Q: How does Mingxin FX100 help improve MI308X inference efficiency? A: The FX100 provides low-latency, high-bandwidth storage access via high-speed networking and optimized protocols. In measured tests on the MI308X platform, it increased inference throughput for a 480B model by 29-40% (R2/R3 measured), reduced TTFT by 26-32% (R2 measured), and significantly accelerated model loading and training checkpoint saving.