Mingxin FX Series: Fault Injection Under Failures

In AI computing centers, the reliability and business continuity of storage systems directly impact whether expensive GPU compute resources can be utilized efficiently and stably. An unexpected storage access interruption can lead to failures in large-scale model inference tasks, loss of training checkpoints, and significant waste of compute power and economic loss. Mingxin Technology, through its FX series all-flash NVMe-oF arrays, not only provides significant acceleration for large model inference and training—such as reducing first-token latency (TTFT) by 26–32% and increasing inference throughput by 29–40% on a 480B model (measured, report R2/R3)—but also designs system architecture resilience against hardware failures. This article analyzes how the Mingxin FX system achieves business-transparent failover under extreme scenarios like simulated single-disk failures and single network link interruptions, ensuring sustained high utilization of GPU compute power.

Why Do Storage System Failures Become the "Invisible Killer" of GPU Compute Power?

In traditional AI compute deployments, storage is often viewed as a relatively static backend resource. However, as model parameter sizes exceed hundreds of billions and context lengths grow continuously, unprecedented demands are placed on storage system performance and reliability. Any instability in the storage system directly propagates to the compute frontend.

First, large model inference, especially in long-context scenarios, is highly sensitive to storage bandwidth and latency. Taking the 480B model measured in Mingxin R2 as an example, without external memory acceleration and relying entirely on recomputation, its TTFT latency reaches 149.5 seconds. If a storage system failure interrupts KV Cache or model weight loading, the entire inference process may stall or crash, causing GPUs to enter idle waiting states and utilization to plummet. This not only interrupts current services but also amplifies compute waste due to the time cost of reloading models and restoring states (e.g., in R9 measurements, loading DeepSeek-70B from NFS takes 1399 seconds).

Second, distributed training tasks require high storage consistency. Saving checkpoints during training is critical for preserving results. R1 measurements show that using FX100 reduces checkpoint save time for 8-card 32B LoRA training from 178 seconds to 94 seconds, a 1.9x acceleration. If a storage failure occurs during saving, checkpoint files may be corrupted or lost, wasting hours or days of training progress and massive GPU cluster compute cycles.

Therefore, a fault-tolerant storage system is no longer just about simple data redundancy backup but a key infrastructure for ensuring the return on investment (ROI) of compute power. It must ensure that data access paths automatically and seamlessly switch when partial hardware fails, allowing compute tasks to continue and keeping GPUs in a high-load "production" state rather than "waiting" or "recovering."

How Does the Mingxin FX System Achieve Business-Transparent Operation Under Single-Disk Failures?

The Mingxin FX series storage arrays use an all-flash design and combine RAID, multipath I/O (MPIO), and NVMe-oF target high availability to address the risk of a single NVMe SSD failure. In fault injection tests, we simulated a scenario where a member disk in the array suddenly fails during online business operations.

Architecture Foundation: Balancing Performance and Redundancy The FX series supports flexible RAID configurations. Users can choose based on their performance and reliability priorities. For example, to pursue extreme throughput and low latency, RAID 0 can be used, such as the 4-disk RAID 0 configuration in the R1-R4 test platform, providing up to 14TB of contiguous namespace and aggregate bandwidth. In production environments, for data reliability, redundant RAID levels like RAID 5 or RAID 6 are used. When configured with redundant RAID, the array controller can detect a single-disk failure and use parity data for background data reconstruction, transparent to the frontend host (i.e., GPU servers).

Failover Mechanism: Zero Interference to the Host The key is that when a single-disk failure occurs, the array's NVMe-oF service does not interrupt. The storage target's virtualization layer isolates the failed physical disk and continues providing logical unit (LUN) access through the remaining healthy disks. For the frontend GPU server with MPIO drivers, it perceives continuous access to the same LUN through remaining paths. Under the high-concurrency, long-context inference workload simulated in R2 tests, this seamless failover means KV Cache read streams are not interrupted, the vLLM engine does not throw I/O exceptions, inference sessions continue, and GPUs do not wait for storage recovery, ensuring the 29–40% throughput improvement (measured, report R2/R3) is not lost due to a single-point hardware failure.

Quantitative Impact: Performance Fluctuation and Reconstruction Load It should be noted that during background reconstruction after a single-disk failure in a redundant configuration, reading data from all remaining disks for computation and writing may temporarily impact overall array performance, manifesting as reduced bandwidth and IOPS. However, thanks to the high performance of all-flash media, the FX system keeps this impact within a limited range. More importantly, compared to GPU compute power dropping to zero due to complete storage unavailability, this controlled performance fluctuation is acceptable, ensuring business "transparent" continuity.

How Is Storage Access Ensured When a Single Network Link Fails?

In NVMe-oF architectures based on RoCE (RDMA over Converged Ethernet), network link reliability is equally critical. Mingxin FX arrays and GPU servers are typically interconnected via high-speed Ethernet (e.g., 100GbE, 200GbE), and physical or logical failure of a single link is another common fault scenario.

Core Role of Multipath I/O (MPIO) In integrations with mainstream GPU servers (e.g., AMD MI308X platform, Huawei Ascend 910B platform), Mingxin FX solutions deploy and optimize OS-level MPIO drivers. In the R9 measured Huawei Ascend platform environment, MPIO configuration ensures high availability of storage access. When the system detects a failed RoCEv2 network path (corresponding to a physical port or VLAN), the MPIO driver automatically switches all I/O requests to another healthy path within milliseconds.

Fault Injection Test Verification In simulated tests, we actively disconnect a 100GbE RoCE link between a GPU server and an FX array. At this time, a running 480B model inference task (e.g., the TP4×2 full-machine load in R3 tests) experiences a very brief (typically less than 1 second) I/O retry or timeout. However, due to MPIO's fast failover, the vLLM engine's connection is not broken, the established NVMe-oF session is maintained, and ongoing KV Cache reads or model weight loading operations continue on the backup link. From a business metrics perspective, TTFT latency and generation throughput curves may show only a minor glitch before immediately returning to normal levels, without causing task failure or GPU compute interruption.

Guaranteeing GPU Utilization This link-level fault tolerance directly protects GPU utilization. In distributed inference or training scenarios, a storage access interruption on any single GPU server can slow down the entire job. Link redundancy via MPIO ensures each server continuously accesses data from shared storage, enabling efficiency improvements like the 1.9x checkpoint save acceleration in R1 measurements to be consistently realized in a stable network environment, avoiding GPU data waiting due to single-point network failures and reducing the cluster's effective compute output.

Conclusion: Converting Reliability into Quantifiable Compute Assurance

The operational goal of AI computing centers is to maximize the effective output of expensive compute resources like GPUs. As the "granary" of data, the storage system's role has evolved from passive storage to a key component actively ensuring compute flow. The Mingxin FX series, through full-stack high-availability design from hardware arrays to host-side software, aims to convert the inevitable probability of hardware failures into negligible business-level fluctuations.

Through simulated fault injection tests of single-disk and single-link failures, the FX system can ensure continuity of AI inference and training tasks during these common anomalies, keeping GPUs in a high-load working state rather than stalled waiting. For large model services requiring long-term stable operation (pursuing high throughput and low latency) and distributed training tasks (ensuring checkpoint safety and job progress), this value transcends pure performance numbers, becoming the cornerstone of stable computing center operations. Mingxin Technology offers approximately 10-week gated joint testing collaborations, which include verification of stability and fault recovery capabilities. Industry partners are welcome to conduct joint testing and performance evaluation based on real-world scenario models.

Key Q&A from This Article

Q: How does the Mingxin FX storage system ensure business continuity during a single-disk failure? A: When configured with redundant RAID (e.g., RAID 5/6), the array controller automatically isolates the failed disk, uses parity data for background reconstruction, and continues providing storage services through remaining healthy disks. For frontend GPU servers, the access path to the logical volume remains connected, so running AI inference or training tasks do not fail due to I/O interruptions, ensuring sustained GPU compute utilization.

Q: If a network link between a GPU server and a Mingxin array is disconnected, will it affect model inference? A: By deploying multipath I/O (MPIO) drivers, the system can detect a failed single RoCE network path and switch I/O traffic to a backup link within milliseconds. This ensures the NVMe-oF session is not interrupted, and ongoing KV Cache reads or model loading continue. For example, in R2 measurements of a 480B model inference task, only a brief latency fluctuation may be perceived, without task failure or GPU idle waiting.

Q: What practical significance do these fault tolerance mechanisms have for improving AI computing center efficiency? A: The core significance is converting storage infrastructure reliability directly into GPU compute availability and output assurance. By avoiding large-scale compute task interruptions, model reloading, or training progress rollbacks due to storage single-point failures, expensive GPU resources are maximally used for actual production computing, thereby improving the entire computing center's ROI and operational stability.