30% Faster Concurrent Model Loading: 8-GPU Cold Reads

In large-scale inference clusters, when 8 or more GPUs simultaneously cold-read model weights or KV Cache, the concurrent throughput of the storage system directly determines service startup latency and GPU idle time. Measured results show that replacing traditional NFS with an NVMe-oF all-flash array can reduce the first-token latency of 8-GPU concurrent cold loading of a 480B model by 26-32% and increase throughput by 29-40%. This is not merely a hardware upgrade but an explicit optimization of GPU utilization through storage engineering—by eliminating I/O bottlenecks, computing resources enter effective inference states faster.

Storage Bottleneck in Concurrent Cold Loading: Why Are GPUs Idle Over 30%?

In large model inference cold starts or long-context recovery scenarios, GPUs need to read hundreds of GB of model weights or KV Cache from storage media. Taking the 480B MoE model (Qwen3-Coder-480B-FP8, weights ~450GB) as an example, on an 8×AMD MI308X platform, using traditional NFS (Network File System) as the storage backend, the single-GPU cold read bandwidth is limited by network protocol overhead and file system metadata locks. Measured throughput is only about 0.98 GB/s (measured baseline, report R1). This means loading 450GB on a single GPU takes about 460 seconds, while with 8 GPUs concurrent, NFS shared lock contention and IOPS bottlenecks extend the total loading time to over 600 seconds.

During this process, GPUs are completely idle—VRAM unfilled, compute units idle, power consumption only at baseline. In production environments, this "GPU idle waiting for storage" time accounts for 30-50% of total service deployment time, directly reducing cluster effective utilization. Inference optimization often focuses on compute-side techniques like operator fusion and quantization, but the contribution of storage engineering to GPU utilization is frequently underestimated.

How Does NVMe-oF Architecture Achieve 30%+ Loading Acceleration?

The Mingxin FX100 all-flash NVMe-oF array addresses concurrent cold read bottlenecks through three engineering improvements:

1. Parallel Read Patch Eliminates Protocol Lock Contention

In concurrent read scenarios, traditional NFS makes the metadata server a single point of bottleneck. The FX100, based on the NVMe-oF (NVMe over Fabrics) protocol, uses RoCEv2 networking (single port 100GbE) to encapsulate NVMe commands directly as RDMA operations, bypassing file system layer locks. Measured results with the LMCache parallel read patch optimization (measured, report R1) show single-GPU cold read TTFT reduced from 37.97 seconds to 9.30 seconds, and bandwidth increased from 0.98 GB/s to 5.23 GB/s (a 5.3x improvement). This corresponds to a reduction in total loading time for 8 concurrent GPUs from about 600 seconds to about 120 seconds (theoretical value, subject to actual concurrent contention).

2. Linear Throughput Scaling with 4-Disk RAID0

The FX100 array is configured with 4 NVMe SSDs in RAID0 (XFS file system), leveraging multi-disk parallel I/O capability. In an 8-GPU concurrent scenario, measured sustained read bandwidth reaches 6.40 GB/s (measured, report R1, training checkpoint save scenario). Although limited by the single-port 100GbE network (theoretical 12.5 GB/s), this far exceeds NFS's 0.98 GB/s. For large sequential read workloads like model loading, the linear scaling of RAID0 provides a 6.2-9.3x throughput improvement (measured, report R9: Huawei 910B platform, DeepSeek-32B loading reduced from 691 seconds to 112 seconds, DeepSeek-70B from 1399 seconds to 150 seconds).

3. IOPS Guarantee from All-Flash Array

The challenge of concurrent cold reads is not only bandwidth but also IOPS in random read scenarios. When 8 GPUs simultaneously request different file blocks, NFS HDD or hybrid arrays suffer from IOPS drops due to mechanical seek. The FX100's all-flash architecture (NVMe SSDs) provides 16M IOPS (FX100 single interface), ensuring independent I/O paths for each GPU without contention. In the 480B model long-context workload (measured, report R2), concurrent 8-stream TTFT is reduced by 26-32% (p50 from 10.17-35.73 seconds to 7.53-26.35 seconds), a direct result of IOPS guarantee.

The Cascading Value of Storage Engineering for Inference and Efficiency Optimization

Concurrent loading acceleration is not an isolated metric; it impacts overall efficiency through the following chain:

• GPU Utilization Improvement: A 30%+ reduction in loading time means less GPU idle time, allowing the cluster to enter inference service faster. For inference clusters with frequent cold starts (e.g., multi-tenant environments), this can handle 10-20% more requests per day.
• Inference Optimization Headroom: When storage is no longer the bottleneck, inference optimization can focus more on the compute side—such as KV Cache hierarchical acceleration (measured throughput increase of 29-40% in reports R2/R3). The synergistic optimization of storage and computing can achieve total throughput gains of over 50%.
• Reduced Operational Complexity: The NVMe-oF architecture requires no modification to upper-layer inference frameworks (e.g., vLLM, LMCache) and integrates via standard NVMe drivers. Compared to NFS tuning (e.g., NFSv4.1 pNFS configuration), deployment costs are lower.

Conclusion

A 30% improvement in model concurrent loading speed is not the endpoint but the starting point for storage engineering to optimize compute center efficiency. The Mingxin FX100 series (FX100/FX200/FX300), through NVMe-oF all-flash arrays, has demonstrated full-link acceleration from loading to inference in measured results. For technical decision-makers focused on GPU utilization and inference optimization, it is recommended to include storage I/O as an independent dimension in compute center planning, rather than relying solely on compute-side optimization. Mingxin offers approximately 10 weeks of gated joint testing services (G1 arrival acceptance, G2 single-machine baseline, G3 main gate verification of TTFT reduction ≥25% and throughput +29-40%), supporting Python-reproducible measurement models under NDA. Teams with relevant needs are welcome to contact us.

Key Q&A

Q: How much does the storage bottleneck affect GPU utilization during 8-GPU concurrent cold loading?
A: In traditional NFS scenarios, GPU idle waiting time can account for 30-50% of total service deployment time; using an NVMe-oF all-flash array (e.g., Mingxin FX100), loading time is reduced by 6.2-9.3x (measured: DeepSeek-70B loading reduced from 1399 seconds to 150 seconds), significantly improving GPU utilization.

Q: How does NVMe-oF architecture achieve over 30% loading acceleration?
A: Through parallel read patches eliminating protocol lock contention (bandwidth increased 5.3x), 4-disk RAID0 linear throughput scaling (sustained read bandwidth up to 6.40 GB/s), and all-flash IOPS guarantee (16M IOPS), concurrent cold read TTFT is reduced by 26-32% (measured, report R2, 480B model).

Q: What are the specific benefits of storage engineering optimization for inference efficiency?
A: After eliminating storage bottlenecks, inference frameworks (e.g., vLLM) can achieve an additional 29-40% throughput improvement through KV Cache hierarchical acceleration (measured, reports R2/R3), with total throughput gains exceeding 50%, while reducing operational complexity (no upper-layer code modification required).