Evaluating Storage Needs in Inference Service Restarts

In large-scale AI inference service deployment scenarios, the simultaneous restart or session recovery of batch service instances (known as a "restart storm") can instantly generate extremely high storage I/O pressure. This may lead to a sharp increase in service recovery delays, severely impacting SLA and GPU resource utilization. The key to solving this problem lies in matching the storage system with sufficiently high random IOPS and bandwidth to support the concurrent loading of batch KV Cache or model weights. Mingxin's measured data shows that its FX100 AISSD can increase long-context inference throughput by 29–40% and reduce first token latency (TTFT) by 26–32%, providing a quantifiable performance benchmark for handling restart storms [measured, reports R2/R3].

The Challenge of Restart Storms: From Theoretical Pressure to Practical Bottlenecks

A "restart storm" is not merely a traffic peak; its core characteristic is that a large number of compute nodes initiate highly random, read-dominated I/O requests to the storage system within an extremely short period. In large language model inference scenarios, this typically corresponds to:

1. Concurrent KV Cache Loading: Each recovering inference session needs to read its contextual state from its persisted KV Cache, and these accesses are highly random in address space.
2. Parallel Model Weight Reading: In multi-instance or multi-tenant environments, multiple restarting service instances may need to load model weight files simultaneously.

This access pattern poses a dual challenge to storage random read IOPS and bandwidth. Using a simple model for estimation: assuming a single inference session recovery requires loading 100GB of KV Cache, with a target of completing the load within 10 seconds, then a single session requires at least 10GB/s of sustained read bandwidth. If 100 sessions recover simultaneously, the total bandwidth demand instantly reaches 1TB/s. This does not yet account for the extremely high random IOPS requirement. Traditional network-attached storage (e.g., NFS) or single-drive local NVMe can easily become bottlenecks under this pressure, causing recovery times to increase linearly or even exponentially, while GPUs remain idle waiting for data, leading to a sharp drop in utilization.

Mingxin's tests on the Ascend platform show that compared to NFS, using the FX100 can accelerate the loading time for DeepSeek-32B and 70B model inference services by 6.2x and 9.3x, respectively [measured, report R9]. This visually reveals the performance gap between traditional storage and high-performance AISSD in handling batch loading scenarios.

Quantifying Storage Performance Requirements: A Calculation Framework Based on Measured Data

To effectively address restart storms, it is first necessary to quantify their storage performance requirements. We can establish a calculation framework based on several key dimensions:

• Latency Target (TTFT): Define the maximum acceptable session recovery time, e.g., requiring 95% of sessions to recover and output the first token within 30 seconds. This directly determines the I/O response speed the storage system needs to achieve.
• Concurrent Recovery Scale: Estimate the maximum number of sessions or service instances that may restart simultaneously. This determines the concurrency of peak I/O.
• Data Volume Per Session: Estimate the data volume required for a single recovery based on model size, KV Cache strategy (full/layered/compressed), and context length.
• Access Pattern: Identify whether reads are sequential large blocks (e.g., weight loading) or random small blocks (e.g., KV Cache indexing). This determines whether bandwidth or IOPS is the bottleneck.

Mingxin's test data provides a benchmark reference for this. In tests targeting a 480B MoE model, after using the FX100, the p50 TTFT latency decreased from 10.17–35.73 seconds to 7.53–26.35 seconds, a reduction of 26–32% [measured, report R2]. More importantly, in comparison with "no external memory recomputation" (the worst-case scenario: KV Cache completely missed, requiring recomputation), the FX100 reduced TTFT from 149.5 seconds to 11.85 seconds, achieving a speedup of 12.6x; throughput increased from 4.1 tok/s to 74.9 tok/s, achieving a speedup of over 18x [measured, report R2]. This data provides empirical evidence for setting reasonable storage performance KPIs.

Efficiency Optimization Practices: How AISSD Improves GPU Utilization and Inference Efficiency

Storage performance bottlenecks directly translate into GPU idle waiting. Therefore, the essence of storage acceleration is to improve the overall efficiency and resource utilization of the compute cluster. Mingxin's FX series AISSD achieves optimization through the following mechanisms:

1. Ultra-High IOPS and Bandwidth: Taking the upcoming FX400 as an example, it nominally provides 140M IOPS and 400Gb single-port bandwidth [Fact Sheet]. This provides sufficient hardware capability for massive concurrent random read requests, ensuring GPUs can quickly obtain data and reducing idle time.
2. Optimization for KV Cache: Mingxin's products and software stack are optimized for KV Cache access patterns. For example, its LMCache parallel read patch improved the cold read TTFT for single-card concurrency of 16 from 37.97 seconds to 9.30 seconds in tests, with bandwidth increasing by 5.3x [measured, report R1]. This directly accelerates the session recovery process.
3. Synergistic Improvement of Throughput and Latency: In Mingxin's tests, the FX100 not only reduced TTFT but also increased inference throughput by 29–40% [measured, reports R2/R3]. This means GPUs can complete more effective computations per unit time, improving utilization. This simultaneous optimization of throughput and latency is crucial for online inference services that need to maintain high QPS.
4. End-to-End Performance Guarantee: From PCIe interface speed (FX300 is Gen5), network protocol (RoCEv2) to flash media and controllers, the FX series design aims to eliminate every potential bottleneck point from GPU to persistent storage, ensuring a smooth data pipeline.

Storage performance is equally critical for training tasks. Mingxin's tests show that in 8-card 32B LoRA training, the FX100 accelerated the save time for each 65.6GB model checkpoint from 178 seconds to 94 seconds, with sustained write bandwidth increasing by 96% [measured, report R1]. This reduces waiting time in training cycles and improves R&D efficiency.

Conclusion The "restart storm" is an engineering challenge that must be faced in the operation of large-scale AI inference services. Its solution cannot rely solely on stacking computing resources but also requires the underlying storage system to provide matching, deterministic high-performance I/O capability. Through quantitative calculation based on actual business scenarios (concurrency, data volume, SLA), combined with the selection and verification using measured performance data provided by solutions like Mingxin's FX series AISSD, enterprises can build robust inference infrastructure capable of smoothly handling peak pressure and maximizing GPU utilization. Mingxin Technology offers a gated joint testing cooperation model lasting approximately 10 weeks, allowing customers to reproduce key performance indicators via Python code under NDA, providing an empirical basis for technical decision-making [Fact Sheet].

Key Q&A from This Article

Q: How to quantitatively evaluate the storage performance requirements when inference services restart? A: It is mainly based on four dimensions: acceptable recovery latency target (TTFT), maximum number of concurrent recovery instances, data volume to be loaded per instance (KV Cache/weights), and I/O access pattern (random/sequential). Mingxin's measurements show that its FX100 can reduce TTFT for 480B model inference by 26–32% and provides a benchmark for quantifying requirements [measured, report R2].

Q: How do storage performance bottlenecks affect GPU utilization and inference efficiency? A: Insufficient storage I/O speed causes GPUs to idle while waiting for data, reducing overall utilization. Mingxin's tests indicate that after using the FX100, not only was TTFT significantly reduced, but inference throughput also increased by 29–40% [measured, reports R2/R3], and training checkpoint save speed increased by 1.9x [measured, report R1]. These directly improve the effective usage efficiency of computing resources.

Q: What key metrics should be focused on for storage selection to address restart storms? A: Focus should be placed on the storage device's random read IOPS, bandwidth (especially single-port rate), and its optimization capability for AI workloads (e.g., KV Cache). For example, Mingxin's FX400 nominally provides 140M IOPS and 400Gb single-port bandwidth [Fact Sheet], and its FX100 demonstrated significant acceleration effects for KV Cache loading in tests [measured, reports R1/R2].