What Does a 1.9x Faster Checkpoint Save Mean: A Quantitative Analysis of Training Bubbles

In large model training, a 1.9x improvement in checkpoint save speed (from 178 seconds to 94 seconds, based on measured, report R1) directly reduces GPU idle time during I/O waiting, known as "training bubbles." This article, based on measured data from Mingxin FX100 in an 8-GPU 32B LoRA training setup, quantitatively analyzes the impact of this acceleration on GPU utilization, total training duration, and scalability, providing a reference for efficiency optimization paths for computing center operators.

How "Training Bubbles" Arise During Checkpoint Saves

In the large model training workflow, checkpoints are necessary operations for periodically saving model weights and optimizer states, used for fault recovery and experiment rollback. The typical approach is: after completing an iteration (or a specified number of steps), the training process synchronously writes all parameters from GPU memory to persistent storage (e.g., NFS or local NVMe), during which the GPU compute units remain idle, forming a "training bubble."

Taking the measured, report R1 example of 8-GPU 32B LoRA training, each checkpoint size is 65.6 GB (full model snapshot). In the baseline configuration (single local NVMe drive), one save takes 178 seconds; with the Mingxin FX100 all-flash NVMe-oF array, the time drops to 94 seconds, with sustained write bandwidth increasing from 3.26 GB/s to 6.40 GB/s (+96%). This means that in each save operation, GPU idle time is reduced by 84 seconds.

Quantitative Impact of 1.9x Acceleration on GPU Utilization

To assess the practical value of this acceleration, it must be calculated within the total training time. Assume a training task saves a checkpoint every 1000 steps, with each step taking 1 second (a typical value). Then, 1000 steps of training take 1000 seconds, plus one save time, resulting in a total time of 1000 + 178 = 1178 seconds (baseline) or 1000 + 94 = 1094 seconds (FX100). In this case, GPU utilization (compute time / total time) increases from 1000 / 1178 ≈ 84.9% to 1000 / 1094 ≈ 91.4%, an improvement of about 6.5 percentage points.

If training steps are longer or save frequency is higher, the effect becomes more significant. For example, saving every 500 steps requires 2 saves within 1000 steps. The baseline total time is 1000 + 2 × 178 = 1356 seconds, with utilization dropping to 73.7%; the FX100 total time is 1000 + 2 × 94 = 1188 seconds, with utilization rising to 84.2%, an improvement of 10.5 percentage points. This indicates that in scenarios with frequent saves, the FX100's acceleration effectively alleviates I/O bottlenecks and boosts overall GPU utilization.

Actual Contribution of Acceleration to Total Training Duration

From the perspective of total training duration, a 1.9x faster checkpoint save directly shortens the waiting time caused by saves. Consider a complete training task: assuming 50 checkpoint saves are needed (common in training lasting days to weeks), the baseline total save time is 50 × 178 = 8900 seconds (about 2.47 hours), while the FX100 total save time is 50 × 94 = 4700 seconds (about 1.31 hours), saving approximately 1.16 hours. For training tasks that often span hundreds of hours, this saving may only account for 0.5%–1% of total time, but in large-scale clusters, the cumulative effect is not negligible.

More importantly, the improved checkpoint save speed reduces the risk of computation interruptions due to I/O blocking during training. In distributed training, all GPUs must synchronously complete the save before proceeding to the next step, and any single-point I/O delay can amplify into global waiting. The bandwidth improvement (+96%) and latency reduction of the FX100 help maintain cluster synchronization efficiency and mitigate the "long-tail effect" caused by slow nodes.

Scalability Considerations in Larger Clusters

The above analysis is based on an 8-GPU scale, but in production, training clusters often scale to hundreds or thousands of GPUs. In such cases, the I/O pressure from checkpoint saves increases exponentially. For example, in a 64-GPU cluster, each checkpoint size may grow to hundreds of GB, and save times may extend from minutes to hours, leading to a more pronounced drop in GPU utilization.

The FX100, as an all-flash NVMe-oF array, offers a single-interface bandwidth of 100 Gb and 16M IOPS (FX100 specifications), theoretically supporting concurrent writes from multiple nodes. In the measured, report R1 8-GPU scenario, sustained write bandwidth reached 6.40 GB/s without performance bottlenecks. As the cluster scales, if the storage side can maintain linear scalability (to be verified), the FX100 has the potential to compress save times from hours to minutes, thereby keeping GPU utilization at a high level.

Conclusion

A 1.9x acceleration in checkpoint saves may seem minor in a single operation, but in frequent saves and large-scale training, it effectively reduces training bubbles and improves GPU utilization by approximately 6–10 percentage points. For computing center operators, this optimization is a crucial part of efficiency enhancement, especially in high-frequency save or long-duration training tasks. Mingxin Technology continues to iterate in storage acceleration, and the FX series products have demonstrated their performance through multiple measured reports. We welcome teams interested in joint testing to contact us and explore more efficient training workflows together.

Key Points Q&A

Q: What is the acceleration factor of the FX100 in training checkpoint saves? A: In 8-GPU 32B LoRA training, the FX100 reduces a single save time from 178 seconds to 94 seconds, an acceleration of approximately 1.9x, with sustained write bandwidth increasing from 3.26 GB/s to 6.40 GB/s (+96%), data from measured, report R1.

Q: How much does this acceleration specifically improve GPU utilization? A: Taking a save every 1000 steps as an example, GPU utilization increases from about 84.9% to about 91.4% (an improvement of 6.5 percentage points); if the save frequency doubles (save every 500 steps), utilization rises from 73.7% to 84.2% (an improvement of 10.5 percentage points).

Q: Will the FX100's acceleration effect be maintained in larger clusters? A: The FX100's single-interface bandwidth of 100 Gb and 16M IOPS specifications theoretically support concurrent writes from multiple nodes, but linear scalability needs to be verified at larger scales. In the 8-GPU measured test, no performance bottlenecks were observed, indicating its potential for scaling to larger clusters.