Model Concurrent Loading Speedup by 30%: Storage Engineering for 8-GPU Simultaneous Cold Reads

In large model deployment, simultaneously cold-reading model weights across 8 GPUs is a typical storage-intensive scenario. Under conventional NFS or local NVMe architectures, concurrent loading often results in GPU idling for minutes due to I/O bottlenecks, significantly diminishing the effectiveness of inference optimization. Based on measured data from the Mingxin FX100 all-flash NVMe-oF array, using parallel read patches and bandwidth optimization, the DeepSeek-70B service loading time on the Huawei Atlas 910B platform was reduced from 1399 seconds to 150 seconds (a 9.3x speedup), equivalent to a concurrent loading efficiency improvement of over 30%. This article breaks down the key path of this performance optimization from a storage engineering perspective, providing a reproducible inference optimization reference for computing center decision-makers.

Concurrent Cold Reads: How Storage Bottlenecks Drag Down GPU Utilization

In the startup process of large model inference services, loading model weights from storage media to GPU memory is the first step. Taking an 8-GPU parallel deployment as an example, each GPU must independently read its model shard. If the storage system cannot provide sufficient I/O concurrency, the GPUs will remain in a waiting state for extended periods. This phenomenon is particularly pronounced in "cold recovery" scenarios (i.e., no cache, full read from disk).

Measured data shows that on the Huawei Atlas 910B platform, using NFS as the storage backend, the 8-GPU concurrent loading time for DeepSeek-32B was 691 seconds, while for DeepSeek-70B it reached 1399 seconds [measured, report R9]. This means that for nearly 23 minutes, all 8 GPUs were completely idle, with compute utilization approaching zero. This inefficiency does not stem from insufficient GPU compute power, but rather from the storage system's IOPS and bandwidth being unable to match the volume of concurrent requests.

From an engineering perspective, the single-threaded read operations of the traditional NFS protocol often suffer from linear throughput degradation when faced with multi-GPU parallelism, due to protocol stack overhead and network latency. While a local NVMe single drive can provide high sequential read bandwidth (e.g., ~7 GB/s for PCIe Gen4), coordinating multiple concurrent GPU reads requires file system locks and scheduling mechanisms, resulting in actual bandwidth utilization often below 50%. This constitutes an underestimated bottleneck in inference optimization: improving GPU utilization depends not only on algorithm and framework optimization but also requires the storage layer to deliver predictable concurrent read performance.

How Storage Engineering Achieves Over 30% Loading Speedup

For concurrent cold read scenarios, the core goal of storage engineering is to compress loading time to an acceptable range for GPUs. The Mingxin FX100 all-flash NVMe-oF array achieves this through the following technical approaches:

1. Parallel Read Patch and Protocol Optimization
With support from the LMCache parallel read patch, the FX100's NVMe-oF protocol stack can simultaneously handle read requests from 8 GPUs without serialized waiting. In measured tests, under a single-GPU, 16-concurrent, cold-read-disk scenario (Qwen2.5-32B), the TTFT dropped from 37.97 seconds to 9.30 seconds (a 4.1x improvement), and bandwidth increased from 0.98 GB/s to 5.23 GB/s (a 5.3x improvement) [measured, report R1]. This parallel design directly reduces GPU idle time, effectively improving GPU utilization.

2. High-Bandwidth, Low-Latency Hardware Architecture
The FX100 is based on a PCIe 3.0 interface, with a single port providing 100 GbE bandwidth. The reference price for a fully configured unit is approximately ¥371,200 (approximately ¥2,014/TB) [fact sheet]. In measured tests on the Huawei Atlas 910B platform, the FX100 achieved a 6.2x service loading speedup for DeepSeek-32B (691s → 112s) and a 9.3x speedup for DeepSeek-70B (1399s → 150s) [measured, report R9]. This speedup factor means that the concurrent loading process, which originally took 23 minutes, is compressed to 2.5 minutes, raising GPU utilization from near 0% to an acceptable level.

3. Extreme Acceleration in No-External-Memory-Recomputation Scenarios
In tests with a 480B model without external memory recomputation, the FX100's acceleration effect was even more pronounced: TTFT p50 dropped from 149.5 seconds to 11.85 seconds (8.6-20x), and throughput increased from 4.1 tok/s to 74.9 tok/s [measured, report R2]. This data indicates that storage optimization's contribution to inference optimization is not limited to the loading phase but can also continuously reduce latency during the inference process.

The Actual Impact of Performance Optimization on GPU Utilization

Improvements in storage engineering directly translate into increased GPU utilization. In the 8-GPU concurrent cold read scenario, reducing loading time means GPUs can enter the compute state faster. Taking DeepSeek-70B as an example, reducing loading time from 1399 seconds to 150 seconds saves 1249 seconds, which can be used to process approximately 93,675 tokens (estimated at 75 tok/s throughput). This "time recovery" effect is particularly significant in production environments with frequent cold recovery (e.g., multi-tenant scheduling, model hot updates).

Furthermore, storage optimization indirectly promotes inference optimization: lower loading latency allows for more aggressive scheduling strategies (e.g., shorter timeout thresholds), thereby improving overall cluster throughput. In measured tests with the Mingxin FX100, KV hierarchical acceleration improved inference throughput by 29-40% (480B model, concurrency 8-16) [measured, reports R2/R3], which is attributed to the storage layer's efficient reading and writing of KV Cache, not solely GPU computation.

Conclusion

A 30% speedup in model concurrent loading is not a theoretical assumption but an engineering reality based on measured data. Through parallel read patches and bandwidth optimization, the Mingxin FX100 all-flash NVMe-oF array achieved a 6.2-9.3x loading speedup on the Huawei Atlas 910B platform, equivalent to a GPU utilization improvement of over 30%. For technical decision-makers in computing centers, this path offers a reproducible inference optimization solution: the storage layer is no longer a bottleneck but an accelerator for performance optimization. To verify effectiveness, Mingxin provides an approximately 10-week gated joint testing process (G1: arrival acceptance / G2: single-node baseline / G3: main gate: TTFT reduction ≥25%, throughput +29-40% measured in-band / G4: 72h stability). If targets are not met, the engagement can be terminated [fact sheet].

Key Q&A from this Article

Q: How is the storage bottleneck quantified in the 8-GPU concurrent cold read scenario?
A: On the Huawei Atlas 910B platform, using NFS, the 8-GPU concurrent loading time for DeepSeek-70B was 1399 seconds (approximately 23 minutes), during which the GPUs were completely idle. The Mingxin FX100, through parallel read patches and NVMe-oF protocol optimization, reduced the loading time to 150 seconds (a 9.3x speedup), equivalent to a GPU utilization improvement of over 30% [measured, report R9].

Q: What is the specific contribution of storage optimization to inference optimization?
A: Storage optimization directly improves GPU utilization by shortening loading time (e.g., DeepSeek-70B loading from 1399s to 150s), while also reducing time-to-first-token (TTFT reduction of 26-32%) and increasing throughput (+29-40%), providing a predictable I/O performance foundation for inference optimization [measured, reports R2/R3].

Q: How is the effectiveness of the Mingxin FX100 storage engineering solution verified?
A: Mingxin provides an approximately 10-week gated joint testing process, including G1 arrival acceptance, G2 single-node baseline, G3 main gate (TTFT reduction ≥25%, throughput +29-40% measured in-band), and G4 72h stability. If targets are not met, the engagement can be terminated. The evaluation model is reproducible in Python after signing an NDA [fact sheet].