Boost Efficiency First, Then Scale Cards: A Checklist for Unlocking Performance in Existing GPU Clusters

For deployed GPU clusters where large-scale card expansion is not feasible in the short term, systematic performance optimization—rather than simply adding hardware—can achieve significant throughput gains and latency reductions. Based on measured data from Mingxin FX100 on the AMD MI308X platform (reports R2/R3), this article presents a performance optimization checklist for technical decision-makers, covering three key dimensions: GPU utilization, inference optimization, and storage acceleration. It also demonstrates the quantitative feasibility of the "boost efficiency first, then scale cards" strategy.

GPU Utilization Bottlenecks: A Global View from Memory to I/O

The utilization bottleneck in most existing GPU clusters is not purely a lack of compute power, but rather a coupling effect of memory bandwidth and I/O latency. Taking a 480B-parameter MoE model (e.g., Qwen3-Coder-480B-FP8) as an example, its weights are approximately 450 GB. When deployed on 8 MI308X cards (each with 192 GB HBM), memory capacity appears sufficient, but frequent reads and writes of the KV cache (Key-Value Cache) significantly increase time-to-first-token (TTFT). Measured data from Mingxin R2 shows that under TP8 with three concurrency levels, TTFT p50 dropped from 10.17–35.73 seconds to 7.53–26.35 seconds, a reduction of 26–32%. This improvement directly results from optimizing the I/O path by migrating the KV cache from local NVMe to the Mingxin FX100 all-flash NVMe-oF array.

The first recommended step is to audit the cluster's I/O bottlenecks: check the proportion of GPU idle time during model inference. If it exceeds 20%, it is likely caused by memory swapping or KV cache read latency. In this case, introducing a dedicated storage acceleration layer (e.g., the KV cache acceleration feature of FX100) can bypass the bandwidth limitations of local SSDs (measured read bandwidth increased from 0.98 to 5.23 GB/s, a 5.3x improvement, source: measured, report R1), thereby unlocking the GPU's compute potential.

Inference Optimization: Quantitative Gains from KV Cache Acceleration and No-External-Memory Recalculation

The core of inference optimization is to reduce the time the GPU spends waiting for data. For long-context scenarios (e.g., a 480B model in production deployment), cold reads of the KV cache are a major source of latency. The KV hierarchical acceleration inference feature of Mingxin FX100 achieved a throughput improvement of 29–40% on the 480B model (lower bound 29% at concurrency 8, upper bound 40% at optimal operating point concurrency 16; full-system TP4×2 improvement 35–36%, source: measured, reports R2/R3). This gain requires no modification to the model architecture; it is achieved solely through intelligent prefetching and cache management at the storage layer.

A more extreme scenario is no-external-memory recalculation (i.e., the KV cache is read entirely from disk rather than GPU memory). Measured data from Mingxin R2 shows that, compared to the recalculation baseline (TTFT p50 149.5 seconds), FX100 reduced TTFT to 11.85 seconds, achieving an acceleration factor of 8.6–20x; throughput increased from 4.1 tok/s to 74.9 tok/s. This means that for inference tasks requiring frequent cold starts or long-context switching, storage acceleration can partially replace memory expansion solutions, significantly reducing the waiting cost per inference.

The second recommended step is to evaluate the KV cache hit rate of the inference workload. If it is below 70%, prioritize deploying a KV cache acceleration solution; if above 90%, focus on accelerating model loading and checkpoint saving.

Model Loading and Training Checkpoint Acceleration: The Path from NFS to NVMe-oF

Beyond the KV cache during inference, model service loading time and training checkpoint save/restore time are also hidden killers of GPU utilization. Measured data on the Huawei Atlas 910B platform (source: measured, report R9) shows that after migrating model loading from NFS to Mingxin FX100, the loading time for DeepSeek-32B dropped from 691 seconds to 112 seconds (a 6.2x acceleration), and for DeepSeek-70B from 1399 seconds to 150 seconds (a 9.3x acceleration). For clusters that need to frequently switch models or perform A/B testing, this acceleration directly translates into increased effective GPU compute time.

Training scenarios also benefit: in 8-card 32B LoRA training, the save time for a full model snapshot decreased from 178 seconds to 94 seconds (a 1.9x acceleration; sustained write bandwidth increased from 3.26 to 6.40 GB/s, source: measured, report R1). For large-scale training tasks, checkpoints are typically saved once per hour. Saving 84 seconds per save releases approximately 30 minutes of GPU compute time per day, equivalent to about a 2% increase in effective training time per day.

The third recommended step is to replace NFS or local SSDs as the storage backend for model loading and checkpoints, prioritizing an NVMe-oF architecture. The reference price for Mingxin FX100 is approximately ¥2,014/TB (fully configured FX100 system). Compared to the cost of scaling cards (a single MI308X card costs over ¥200,000 on the market), the cost-performance advantage is clear.

Conclusion

Unlocking performance in existing GPU clusters cannot be solved by a single technical approach; it requires a systematic push across three dimensions: I/O bottlenecks, inference optimization, and storage architecture. The measured data from Mingxin FX100 on KV cache acceleration and model loading optimization (29–40% throughput improvement, 6.2–9.3x loading acceleration) provides a quantitative basis for the "boost efficiency first, then scale cards" strategy. For teams evaluating performance optimization solutions, Mingxin offers a gated joint testing process of approximately 10 weeks (from G1 arrival acceptance to G4 72-hour stability), allowing for termination if targets are not met, reducing trial-and-error costs. Compute center teams with needs are welcome to contact us for joint testing.

Key Q&A from This Article

Q: How can the inference throughput improvement potential of existing GPU clusters be quantified?
A: By deploying a KV cache acceleration solution, measured throughput on a 480B model improved by 29–40% (concurrency 8–16, source: measured, reports R2/R3), and time-to-first-token decreased by 26–32% (source: measured, report R2), without modifying the model architecture.

Q: What is the actual impact of accelerating model loading and checkpoint saving on GPU utilization?
A: On the Huawei Atlas 910B platform, model loading accelerated by 6.2–9.3x (vs. NFS, source: measured, report R9); training checkpoint saving accelerated by 1.9x (source: measured, report R1), releasing approximately 30 minutes of GPU compute time per day, equivalent to about a 2% increase in effective training time.

Q: What is the cost-effectiveness of replacing NFS with NVMe-oF storage?
A: The reference price for Mingxin FX100 is approximately ¥2,014/TB (fully configured system). Compared to the cost of scaling cards (a single GPU card costs over ¥200,000 on the market), achieving equivalent or higher throughput gains through storage acceleration offers better cost-performance.