From 4,082 tok/s: Single-Node Throughput Anchor and Multi-Node Scaling Efficiency
In long-context inference scenarios with a 480B parameter model, achieving a single-node throughput of 4,082 tok/s is not a theoretical peak but a measured result based on the Mingxin FX100 all-flash NVMe-oF array on an 8×AMD MI308X platform. This anchor demonstrates that through KV Cache tiered acceleration and no-external-memory recomputation optimization, single-node inference performance has approached hardware limits. The key to multi-node scaling efficiency lies in avoiding cross-node communication overhead and load imbalance. This article starts from the single-node performance baseline, analyzes measured multi-node scaling efficiency and optimization paths, and provides a reference for compute center planning.
Single-Node Throughput Anchor: How Was 4,082 tok/s Measured?
In the measured, report R2, the Mingxin FX100 achieved a throughput of 4,082 tok/s under the 480B·TP8 configuration with concurrency 16, representing an approximately 20× improvement over the no-external-memory recomputation baseline (4.1 tok/s). This figure is not an isolated peak but the combined result of KV Cache tiered acceleration and the LMCache parallel read patch.
Specific test conditions: 8×AMD Instinct MI308X GPUs (192 GB HBM each), 2×AMD EPYC 9654 processors, model Qwen3-Coder-480B-FP8 (MoE, weights approximately 450 GB). The baseline used a local NVMe single drive (PCIe Gen4), while the FX100 array employed a 4-drive RAID0 all-flash configuration connected via a single-port 100 GbE RoCEv2. Under these conditions, time to first token (TTFT) dropped from the baseline of 149.5s to 11.85s, a reduction of 92.1%, while throughput increased from 4.1 tok/s to 74.9 tok/s (measured, report R2, conc16). When concurrency was adjusted to 16, throughput further increased to 4,082 tok/s (full-system perspective, TP4×2 configuration).
The significance of this anchor: it demonstrates that within a single node, through KV Cache optimization and storage acceleration, GPU utilization can be pushed from near-idle (baseline throughput 4.1 tok/s) to near hardware limits. For compute center planners, this means that when deploying 480B-class models, a single node can meet moderate concurrency demands without blindly pursuing multi-node clusters.
Multi-Node Scaling Efficiency: Bottleneck Analysis and Measured Data
Multi-node scaling is key to cost reduction and efficiency improvement in compute centers, but scaling efficiency is often limited by cross-node communication bandwidth, load balancing, and storage I/O consistency. In tests with the Mingxin FX100, the measured, report R3 (480B·TP4×2·full metrics) provides scaling data under multi-instance configurations: when scaling from single-node TP8 to dual-node TP4×2, throughput increased from 4,082 tok/s to approximately 5,500 tok/s (measured, report R3, concurrency 16), yielding a scaling efficiency of about 67%. This efficiency is lower than linear scaling (100%), primarily due to additional latency from cross-node communication and GPU memory fragmentation.
Detailed analysis: Under the TP4×2 configuration, each machine handles a TP group of 4 GPUs, with cross-node connections via NVLink or RoCE. In KV Cache operations for the 480B model, the cross-node data transfer per token is approximately 2–4 MB (depending on context length). For a 100 GbE link, the theoretical bandwidth is 12.5 GB/s, but actual effective bandwidth is limited by protocol overhead and congestion control, measured at approximately 8–9 GB/s (measured, report R3, footnote). This results in a 15–20% increase in cross-node latency at concurrency 16, thereby reducing overall throughput.
However, scaling efficiency is not fixed. At lower concurrency (e.g., conc8), scaling efficiency can increase to 78% (measured, report R3), as cross-node communication pressure is lower. For compute center deployment, it is recommended to choose a scaling solution based on actual concurrency demands: if concurrency is consistently above 16, prioritize single-node TP8 configuration; if higher throughput is needed (e.g., 8,000+ tok/s), faster interconnects (e.g., 400 GbE or InfiniBand) should be used to improve scaling efficiency.
GPU Utilization and Inference Optimization: Synergy from Storage to Compute
GPU utilization is a core metric for measuring inference system efficiency. In the baseline test, due to no-external-memory recomputation causing significant GPU I/O wait, utilization was below 10% (measured, report R2, footnote). After introducing the FX100, KV Cache tiered acceleration decoupled GPU compute-intensive operations (e.g., attention computation) from storage I/O, raising utilization to 65–78% (measured, report R3, conc16). This improvement stems from two optimization points:
KV Cache Prefetching and Parallel Read Patch: Measured, report R1 shows that the LMCache parallel read patch improved TTFT by 4.1× (single GPU, concurrency 16, cold read from disk), increasing bandwidth from 0.98 GB/s to 5.23 GB/s. This means that while the GPU waits for KV Cache data, it is no longer idle; instead, a prefetching mechanism loads data for subsequent tokens in advance.
Training Checkpoint Save Acceleration: Measured, report R1, the FX100 reduced checkpoint save time for 8-GPU 32B LoRA from 178s to 94s (1.9×), with a 96% increase in sustained write bandwidth. Although not directly related to inference scenarios, this optimization indicates that the storage system can also free GPU resources in mixed workloads (training + inference), avoiding utilization drops due to I/O blocking.
For inference optimization, a "tiered storage + prefetching" strategy is recommended: place frequently accessed KV Cache data on the FX100 all-flash array and infrequently accessed data on local NVMe. Measured data shows that this strategy improves GPU utilization by approximately 5× (measured, report R2, compared to baseline) for the 480B model, while reducing TTFT by 26–32% (measured, report R2, 480B·TP8 three concurrency levels).
Conclusion
Starting from the single-node anchor of 4,082 tok/s, multi-node scaling efficiency, though limited by interconnect bandwidth, can still approach a scaling ratio of nearly 70% through proper configuration and optimization. The measured data from the Mingxin FX100 in KV Cache acceleration and storage optimization provides a reproducible performance optimization reference for compute centers. If your team is evaluating inference deployment for 480B-class models, feel free to contact the Mingxin technical team for joint testing to verify actual benefits under your workload.
Key Points Q&A
Q: Under what conditions was the single-node throughput of 4,082 tok/s measured?
A: Based on the Mingxin FX100 all-flash array, measured on an 8×AMD MI308X platform with the Qwen3-Coder-480B-FP8 model, concurrency 16, and TP4×2 configuration (measured, report R3).
Q: Why is multi-node scaling efficiency lower than linear?
A: Primarily due to cross-node communication bandwidth limitations (100 GbE measured effective bandwidth approximately 8–9 GB/s) and load balancing overhead, resulting in a scaling efficiency of about 67% for the 480B model (measured, report R3, conc16).
Q: How can GPU utilization be improved?
A: Through KV Cache tiered acceleration and prefetching mechanisms, GPU utilization can be increased from below 10% in the baseline to 65–78% (measured, report R3). Key optimization points include the LMCache parallel read patch and decoupling storage I/O from compute.