Inference Cluster Storage Ratio: Measured Basis for One Array Serving 8 Nodes

There is no universal answer for the storage ratio of an inference cluster, but it can be determined through measurement: in the 480B production deployment test of Mingxin FX100, a single all-flash NVMe-oF array serving 8 inference nodes delivered a throughput improvement of +29–40% within the measured band from KV tiering acceleration [source: measured, reports R2/R3]. The validity of this ratio derives from the quantitative relationship between storage latency and inference concurrency, not from empirical estimation.

What Is the Core Constraint on Storage Ratio

The storage ratio of an inference cluster fundamentally answers one question: how many inference nodes' KV Cache read/write demands can a single storage array's bandwidth and latency capability support without becoming a bottleneck. The difficulty lies in the fact that KV Cache access patterns differ completely from training data loading—it involves high-frequency, small-granularity, latency-sensitive sequential reads that are strongly correlated with the inference concurrency profile.

According to Efficient Memory Management for Large Language Model Serving with PagedAttention, paged management of KV Cache solves GPU memory fragmentation, but it also means that KV Cache swapping can become a critical path in inference. When KV Cache cannot fully reside in GPU memory, storage latency directly determines time-to-first-token (TTFT) and throughput.

The platform configuration used by Mingxin in the R2 test was 8× AMD Instinct MI308X (192GB HBM per GPU), 2× AMD EPYC 9654, vLLM 0.20.1+rocm721, and LMCache built from upstream mainline source [source: main test platform description]. The device under test was the FX100 all-flash NVMe-oF array (4-disk RAID0, 14TB, RoCEv2, single-port 100GbE), with a local NVMe single disk as baseline. This configuration itself defines a test scope of one array serving 8 nodes.

What Is the Measured Basis for the 8-Node Ratio

Under a 480B model (Qwen3-Coder-480B-FP8, MoE, ~450GB weights) TP8 long-context cold-recovery workload, the measured data for the FX100 array serving an 8-GPU node is as follows:

The validity of the 8-node ratio lies in the following: at concurrency level 8, the FX100's TTFT p50 was 7.53s, a 26% reduction compared to the local NVMe baseline of 10.17s [source: measured, R2]. This indicates that when a single FX100 array serves the concurrent load of an 8-GPU node, storage latency remains within a controllable range while consistently delivering throughput gains.

When concurrency is raised to level 16, the throughput gain reaches the upper bound of +40%, but this is the "optimal operating point" rather than the steady state. The R2 test also recorded a comparison without external memory recomputation: the recomputation baseline TTFT p50 was 149.5s (concurrency 16), while the FX100 achieved 11.85s, an acceleration factor of 8.6–20× [source: measured, R2]. This extreme comparison shows that the value of storage acceleration is particularly significant in long-context cold-recovery scenarios.

What Are the Boundary Conditions for the 8-Node Ratio

The 8-node ratio is not a universal conclusion; its validity depends on three prerequisites:

First, concurrency profile and context length. The R2 test used a 480B model, TP8, and a long-context cold-recovery workload. If the model is smaller (e.g., 14B), GPU memory pressure is lower, KV Cache swap-in demand decreases, and the storage ratio can be reduced accordingly. Mingxin's R5 test (14B·GPU memory efficiency) targets such scenarios but does not alter the boundary of the conclusions from the 8-node test above.

Second, the bandwidth headroom of the storage array. The FX100's single-port 100GbE (PCIe 3.0) measured bandwidth performance under RoCEv2 supported the concurrent read/write demands of 8 nodes. According to the NVIDIA GPUDirect Storage Documentation, GPU-direct storage bypassing the CPU bounce buffer can reduce latency, but this mechanism depends on coordinated support between the storage side and the GPU side. If the inference nodes' network or storage protocol stack is mismatched, actual bandwidth may fall below test values.

Third, the degree of software stack adaptation. The R2 test used LMCache from upstream mainline (source compiled on 2026-06-29) with a parallel-read patch applied. According to measured results from R1, under a single-GPU concurrency-16 cold-read scenario (Qwen2.5-32B), this patch reduced TTFT from 37.97s to 9.30s, a 4.1× improvement, and increased bandwidth from 0.98 to 5.23 GB/s (↑5.3×) [source: measured, R1]. If the inference framework does not integrate LMCache or a similar tiered caching mechanism, the storage acceleration effect will be significantly diminished.

Methodology for Determining the Ratio

For procurement decision-makers, determining the storage ratio should not rely on vendor "reference values" but should follow a reproducible measurement process:

1. Define SLA constraints first. Clearly specify the pass/fail thresholds for TTFT and throughput. In Mingxin's collaboration model, the primary gate (G3) sets a TTFT reduction of ≥25% and throughput within the +29–40% measured band [source: collaboration model description].
2. Then measure storage latency sensitivity to concurrency. Under the target model and concurrency levels, compare TTFT differences between local NVMe and the NVMe-oF array. If the difference is smaller than the SLA margin, the storage ratio can be reduced; if it approaches or exceeds the margin, additional arrays or optimized caching strategies are needed.
3. Finally, verify stability. Mingxin's collaboration model includes a G4 72-hour stability test to rule out interference from sporadic performance fluctuations on the ratio conclusions.

According to the MLPerf Inference: Datacenter Benchmark Suite Results, publicly comparable inference benchmarks emphasize submission criteria with fixed precision and latency constraints. This suggests that storage ratio validation should also use comparisons under fixed constraints rather than chasing peak performance.

Conclusion

The ratio of one array serving 8 nodes has been validated in Mingxin FX100's 480B measurements, but its applicability boundary is equally clear: it depends on model size, concurrency profile, and software stack adaptation. For different workloads, it is recommended to confirm the ratio through gated joint testing (approximately 10 weeks, including arrival acceptance, single-node baseline, primary gate, and stability tests), and stop if targets are not met. Mingxin can provide an NDA-based reproducible Python measurement model for decision-makers to validate independently.

Key Q&A

Q: What is the basis for the ratio of one array serving 8 inference nodes? A: Under the 480B model TP8 long-context cold-recovery workload, Mingxin FX100 measured a throughput improvement of +29% at concurrency level 8, with TTFT p50 reduced to 7.53s (baseline 10.17s) [source: measured, R2/R3]. This ratio holds under the test conditions and is not a universal conclusion.

Q: What determines the upper limit of the storage ratio? A: It is jointly determined by storage array bandwidth, inference concurrency profile, and software stack adaptation. At concurrency level 16, the throughput gain reaches the upper bound of +40%, but this requires software optimizations such as the LMCache parallel-read patch [source: measured, R2/R1].

Q: How can one verify whether the storage ratio fits their own workload? A: First define SLA constraints (e.g., TTFT reduction ≥25%), then compare the differences between local NVMe and the NVMe-oF array at the target concurrency, and finally confirm through a 72-hour stability test [source: collaboration model description].

References

1. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
2. NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html
3. MLPerf Inference: Datacenter Benchmark Suite Results — https://mlcommons.org/benchmarks/inference-datacenter/