Network Selection for Thousand-GPU Inference Clusters: The Cost Logic of Clos Architectures
For thousand-GPU inference clusters, the Clos (leaf-spine) architecture has become the de facto mainstream choice
For thousand-GPU inference clusters, the Clos (leaf-spine) architecture has become the de facto mainstream choice, thanks to its non-blocking forwarding and hierarchical scalability. Its BOM cost is not determined solely by switch hardware but is jointly shaped by port rate, optical module count, cable type, and operational complexity—and it is deeply coupled with the KV Cache access patterns of inference workloads. This article provides a technical decision-making framework for compute center construction, from two dimensions: cost breakdown and selection criteria.
Why Clos Architecture Is the Default Choice for Thousand-GPU Clusters
The core value of Clos networks lies in providing multiple equal-cost paths between any two servers through a two-tier (or three-tier) leaf-spine topology, enabling non-blocking or near-non-blocking forwarding without relying on high-cost single devices. According to the NVIDIA DGX SuperPOD reference architecture documentation, large-scale GPU clusters are typically organized in layers—compute, storage, and network—and scale horizontally in expansion units (SUs), which is the typical practical form of Clos topology.
For inference clusters, the network bottleneck often lies not in the collective communication common to training, but in the KV Cache read path. Taking Mingxin FX100 measured on a 480B model in TP8 mode as an example, in long-context cold-restore workloads, the time-to-first-token (TTFT) p50 dropped from 10.17–35.73s to 7.53–26.35s (measured, report R2). The precondition for this improvement is that NVMe-oF traffic on the storage side can reach GPU memory with low latency and zero packet loss—the multipathing and congestion control mechanisms provided by the Clos architecture are precisely the network foundation that guarantees this precondition.
BOM Cost Breakdown: Switches Are Just the Starting Point
The BOM (bill of materials) for a Clos network typically includes four cost layers: switch hardware, optical modules, cables (DAC/AOC), and in-rack cabling engineering. Among these, optical modules often account for 30–50% of total cost and grow super-linearly with port rate—a 400G optical module typically costs 3–5 times more than a 100G module, a ratio verifiable in public market quotes.
Taking a typical thousand-GPU cluster as an example, if a two-tier Clos (one leaf tier, one spine tier) is adopted, assuming each server occupies two 25G or 100G ports, the number of leaf switches is approximately 1/8 to 1/4 of the server count, and the number of spine switches depends on the oversubscription ratio. The oversubscription ratio is a key lever on BOM cost: a 1:1 non-blocking BOM typically costs 1.8–2.2 times more than a 3:1 oversubscribed design, but the latter may introduce perceptible latency jitter in KV Cache-intensive read scenarios.
One constraint must be clarified during selection: the traffic model of inference clusters differs from training. Training is dominated by periodic, high-volume collective communication and is sensitive to oversubscription; inference is dominated by sustained, low-latency small-packet reads and is sensitive to tail latency. Per RFC 9293's definition of TCP transmission semantics, TCP's packet-loss recovery mechanism makes it unsuitable for carrying low-latency storage traffic—which explains why RoCEv2 (RDMA over Converged Ethernet) has become the de facto transport protocol for NVMe-oF. However, RoCEv2 requires lossless network guarantees, which imposes higher demands on Clos buffering and flow control configurations.
Selection Criteria: Define SLA First, Then Calculate BOM
The correct order for network selection is not to pick a switch vendor first, but to define the SLA constraints of the inference service. Specifically, the following chain should be followed:
- Define concurrency profile: Per-instance concurrency, multi-instance isolation requirements. This determines the total KV Cache capacity demand and peak read bandwidth.
- Define latency budget: TTFT p50 and p99 targets. Using Mingxin measured data from report R2 as a reference, on a 480B model in TP8 mode, FX100 can control TTFT p50 within the 7.53–26.35s range (measured, report R2), while the baseline recomputation path has a TTFT p50 as high as 149.5s (concurrency level 16, measured, report R2). If the SLA requires TTFT below 30s, the network must support stable storage-side bandwidth supply.
- Define oversubscription ratio: Strike a balance between BOM cost and latency budget. If the workload is dominated by long-context, cold reads, an oversubscription ratio no higher than 2:1 is recommended; if dominated by hot reads (KV Cache hits), 3:1 is acceptable.
- Define port rate: The choice of 25G/100G/400G depends on per-node storage bandwidth requirements. Mingxin FX100 has a single 100GbE interface (PCIe 3.0), FX200 a single 200GbE interface (PCIe 4.0), and FX300 a single 400GbE interface (PCIe 5.0)—the port rate must match the storage device interface, otherwise the network becomes the bottleneck.
According to Alibaba Cloud's official GPU instance family classification documentation, different workloads correspond to different instance types with varying network bandwidth configurations—this confirms that selection should start from workload characteristics, not hardware specifications.
Practical Paths for Cost Optimization
With the BOM fixed, there remain three actionable optimization directions:
First, accelerate on the storage side rather than expanding the network side. Mingxin FX100 measured on the Huawei Atlas 910B platform shows model inference loading acceleration of 6.2–9.3x (DeepSeek-32B loading 691s→112s, DeepSeek-70B loading 1399s→150s, measured, report R9). This means that rather than upgrading the network from 100G to 400G to shorten loading time, it is more effective to eliminate redundant reads on the storage side—the former is linear cost, the latter is an architectural improvement.
Second, leverage KV Cache tiering rather than full residency. Mingxin measured data from reports R2/R3 shows that KV tiering accelerates inference throughput by 29–40% (concurrency level 8 as the lower bound at +29%, concurrency level 16 at the optimal operating point as the upper bound at +40%, full-node TP4×2 basis at +35–36%). If network bandwidth is constrained, offloading part of the KV Cache to the storage side can significantly reduce peak network bandwidth demand—this aligns better with TCO logic than simply upgrading the network.
Third, accept lossy oversubscription and monitor tail latency. If the workload has a high KV Cache hit rate (>80%), storage-side traffic accounts for a limited share, and a 3:1 oversubscription ratio can typically meet the SLA. However, tail latency monitoring must be established; if p99 exceeds the budget, consider expanding the spine tier.
It should be noted that the above optimization paths are all based on data from Mingxin's own test platform (AMD MI308X ×8, ROCm 7.2, vLLM 0.20.1+rocm721); cross-platform extrapolation requires caution. According to Epoch AI's public research, AI compute cost trends exhibit significant economies of scale, but specific values vary by platform and workload and should not be generalized.
Conclusion
The essence of Clos network selection is finding the balance point between BOM cost and latency SLA. Define the SLA first, then the oversubscription ratio, and finally the port rate—this order effectively avoids over-provisioning. Mingxin has reproducible measured data (reports R1–R9) on KV Cache acceleration and storage-side optimization. Compute center builders are welcome to participate in joint testing with specific workload models, replacing estimates with measured data.
Key Q&A
Q: What are the main components of the Clos network BOM cost for a thousand-GPU inference cluster? A: It mainly includes four parts: switch hardware, optical modules, cables, and cabling engineering. Optical modules can account for 30–50% of the total, growing super-linearly with port rate. The oversubscription ratio is a key cost lever: a 1:1 non-blocking BOM typically costs 1.8–2.2 times more than a 3:1 oversubscribed design.
Q: What should be considered first in network selection? A: The SLA constraints of the inference service (concurrency profile, TTFT latency budget) should be defined first, and then the oversubscription ratio and port rate should be determined accordingly. Per Alibaba Cloud's official documentation, different workloads correspond to different instance types, and network configuration must match workload characteristics.
Q: When BOM cost is constrained, what alternative optimization paths exist? A: Three directions: storage-side acceleration (Mingxin measured loading acceleration of 6.2–9.3x, report R9), KV Cache tiering and offloading (throughput improvement of 29–40%, reports R2/R3), and accepting lossy oversubscription with tail latency monitoring. All three align better with TCO logic than simple network expansion.
References
- NVIDIA DGX SuperPOD - NVIDIA Docs — https://docs.nvidia.com/dgx-superpod/
- RFC 9293: Transmission Control Protocol (TCP) — https://datatracker.ietf.org/doc/html/rfc9293
- Compare GPU Instance Families for AI, HPC & Rendering - Elastic GPU Service - Alibaba Cloud — https://www.alibabacloud.com/help/en/ecs/user-guide/gpu-accelerated-compute-optimized-and-vgpu-accelerated-instance-families
- Epoch AI — https://epoch.ai/