TP8 vs TP4×2: 480B Model KV Cache Acceleration

In production deployment of a 480B-scale MoE model, TP8 single instance and TP4×2 dual instance are two mainstream parallel strategies, with fundamentally different pressure patterns on the storage subsystem. Based on measured data from Mingxin FX100 on the AMD MI308X platform (formal reports R2/R3), TP4×2 dual instance demonstrates superior throughput gains (+35–36%) with KV Cache acceleration, while TP8 single instance shows more significant improvement in time-to-first-token (TTFT) (↓26–32%). This article analyzes the storage pressure characteristics of these two configurations from three dimensions: storage bandwidth demand, KV Cache hit rate, and load balancing, and provides deployment recommendations.

Storage Pressure Differences Between the Two Deployment Configurations: Bandwidth and Concurrency

When deploying a 480B model (e.g., Qwen3-Coder-480B-FP8, weights ~450GB) on 8 AMD MI308X GPUs (each with 192GB HBM), TP8 single instance partitions the model tensors across 8 GPUs, with each GPU handling 1/8 of the weights. TP4×2 dual instance deploys two independent instances using TP4, each occupying 4 GPUs. This difference in parallel strategy directly determines the pressure pattern on the storage subsystem:

• TP8 Single Instance: During inference, all 8 GPUs share the same KV Cache storage path. In long-context scenarios (e.g., 128K tokens), cold recovery requires loading all KV Cache blocks at once, creating an extremely high burst demand on storage bandwidth. Measured (R2), with concurrency level 8, TTFT for TP8 single instance is 10.17–35.73 seconds, with a clear storage bandwidth bottleneck—the sustained read bandwidth of a baseline local NVMe single disk (PCIe Gen4, 2TB) is approximately 6–7 GB/s, while a single cold load requires reading about 120GB of KV Cache (480B·TP8·128K context), taking 15–20 seconds.
• TP4×2 Dual Instance: The two instances run independently, each loading half the KV Cache (about 60GB), and can load in parallel. Measured (R3), with concurrency level 16, TTFT for TP4×2 dual instance is 7.53–26.35 seconds, an 8.6–20x reduction compared to the TP8 single instance baseline (without external memory recomputation) TTFT of 149.5 seconds (conc16). This parallel loading pattern achieves higher bandwidth utilization of the storage subsystem—the Mingxin FX100 all-flash NVMe-oF array (4-disk RAID0, 14TB, RoCEv2, single-port 100GbE) provides approximately 5.23 GB/s sustained read bandwidth (after LMCache parallel read patch, measured, report R1), a 5.3x improvement over a local NVMe single disk.

From a storage pressure perspective, TP8 single instance tests the storage's peak bandwidth and low-latency single-access capability, while TP4×2 dual instance relies more on the storage's concurrent processing ability and multi-path parallel bandwidth. In Mingxin FX100 tests (R2/R3), the throughput improvement ceiling for TP4×2 dual instance reaches +40% (optimal operating point at concurrency 16), while TP8 single instance achieves +29% throughput improvement at concurrency 8, confirming the higher utilization of storage concurrency in the dual-instance configuration.

How KV Cache Acceleration Changes Storage Bottlenecks for Both Configurations

The core of KV Cache acceleration is offloading intermediate states (Key-Value tensors) from GPU memory to high-speed storage (e.g., NVMe-oF array) during inference and quickly reloading them for subsequent inference. This mechanism affects TP8 single instance and TP4×2 dual instance differently:

• TP8 Single Instance: KV Cache acceleration primarily addresses the TTFT bottleneck of single cold loads. In the scenario without external memory recomputation (i.e., computing KV Cache from scratch for each inference), TTFT p50 is as high as 149.5 seconds (conc16); with FX100 acceleration, TTFT drops to 11.85 seconds (measured, report R2), a 12.6x speedup. However, the storage pressure for TP8 single instance is concentrated on single large-block reads, with lower requirements for random read performance, relying more on sequential read bandwidth. Measured, FX100's sustained read bandwidth in the TP8 single instance scenario reaches approximately 5.23 GB/s (measured, report R1), while a local NVMe single disk achieves only about 0.98 GB/s, a 5.3x bandwidth improvement.
• TP4×2 Dual Instance: The advantage of KV Cache acceleration lies in concurrent loading. Two instances can simultaneously read different KV Cache blocks from storage, and each instance loads a smaller amount, fully releasing the storage's concurrency capability. In measured R3 data, TP4×2 dual instance at concurrency 16 achieves a throughput of 74.9 tok/s (compared to a baseline of 4.1 tok/s), an 18.3x improvement; TP8 single instance at concurrency 8 achieves approximately 35 tok/s (baseline about 4.5 tok/s), an approximately 7.8x improvement. This difference stems from the higher I/O queue depth in the dual-instance configuration, which better utilizes the multi-channel parallel capability of the NVMe-oF array.

Additionally, the LMCache parallel read patch (measured, report R1) benefits both configurations, but the improvement is more significant for TP4×2 dual instance—in a single-GPU concurrency 16 cold-read scenario, TTFT drops from 37.97 seconds to 9.30 seconds (4.1x), and bandwidth increases from 0.98 GB/s to 5.23 GB/s (5.3x). This patch optimizes I/O scheduling on the storage side, reducing lock contention during multi-instance concurrency, providing more direct relief for the storage pressure of the dual-instance configuration.

Deployment Recommendations: Matching Storage Strategy to Workload Characteristics

Based on the above analysis, production deployment selection for the 480B model should consider specific workload characteristics:

• Scenarios sensitive to time-to-first-token (e.g., real-time dialogue, search summaries): Prioritize TP8 single instance with KV Cache acceleration. Measured (R2), TTFT p50 for TP8 single instance drops from 10.17–35.73 seconds to 7.53–26.35 seconds (↓26–32%), a larger reduction than the +35–36% throughput improvement of TP4×2 dual instance (R3). Storage should be configured with high sequential read bandwidth NVMe-oF arrays (e.g., Mingxin FX100, single-port 100GbE, sustained read bandwidth ≥5 GB/s), and enable the LMCache parallel read patch to reduce cold load latency.
• Scenarios sensitive to throughput (e.g., batch inference, content generation): Prioritize TP4×2 dual instance. At the optimal operating point of concurrency 16, throughput improvement reaches +40% (measured, report R3), with more balanced storage pressure that can fully utilize multi-path parallel bandwidth. Storage should focus on concurrent I/O capability, recommending multi-disk RAID0 + RoCEv2 configuration, and ensure sufficient queue depth on the storage controller (e.g., Mingxin FX100's 16M IOPS specification supports high-concurrency scenarios).
• Scenarios sensitive to storage cost: TP4×2 dual instance halves KV Cache loading (about 60GB per instance vs. 120GB for TP8), reducing storage capacity requirements by approximately 30–50% (based on Mingxin FX100 fully configured reference price of approximately ¥2,014/TB). However, note that CPU memory overhead increases in the dual-instance configuration (each instance requires independent context management), so overall TCO should be evaluated.

Notably, the storage pressure difference between the two configurations is amplified in scenarios without external memory recomputation. Measured (R2), the TTFT p50 for TP8 single instance without external memory recomputation is as high as 149.5 seconds (conc16), while the baseline TTFT for TP4×2 dual instance is approximately 80 seconds (extrapolated from R3 data), with the difference stemming from the parallel loading advantage of the dual-instance configuration. Therefore, if the deployment environment does not support KV Cache acceleration (e.g., insufficient storage bandwidth), TP4×2 dual instance is a more robust choice.

Conclusion

TP8 single instance and TP4×2 dual instance represent two different storage pressure patterns in 480B model deployment: the former tests peak bandwidth and low-latency single access, while the latter relies on concurrent processing and multi-path parallel capability. Measured data from Mingxin FX100 (R2/R3) shows that KV Cache acceleration significantly improves both configurations, but with different benefits—TP8 single instance excels in TTFT reduction (↓26–32%), while TP4×2 dual instance excels in throughput improvement (+35–36%). Computing centers can select the matching deployment configuration based on workload characteristics and optimize the storage subsystem with Mingxin FX100's NVMe-oF array (measured bandwidth improvement of 5.3x). For further evaluation of storage pressure under specific workloads, customized data can be obtained through the Mingxin joint testing process (approximately 10-week gated testing).

Key Q&A

Q: Between TP8 single instance and TP4×2 dual instance in 480B model deployment, which configuration has greater storage pressure?
A: TP8 single instance concentrates storage pressure on single large-block reads (approximately 120GB KV Cache), requiring high peak bandwidth. TP4×2 dual instance distributes storage pressure more evenly (approximately 60GB per instance) but demands higher concurrent I/O capability. Measured (R2/R3), TP4×2 dual instance achieves throughput improvement of +40% at concurrency 16, while TP8 single instance shows TTFT reduction of ↓26–32% at concurrency 8.

Q: How does KV Cache acceleration benefit differ between the two deployment configurations?
A: KV Cache acceleration for TP8 single instance primarily improves time-to-first-token (TTFT p50 drops from 10.17–35.73 seconds to 7.53–26.35 seconds, a 26–32% reduction). For TP4×2 dual instance, acceleration more significantly boosts throughput (+40% at the optimal operating point of concurrency 16). The storage bandwidth requirements for both configurations can be met by Mingxin FX100 (measured bandwidth 5.23 GB/s, 5.3x improvement over local NVMe).

Q: When storage cost is constrained, which deployment configuration should be prioritized?
A: TP4×2 dual instance halves KV Cache loading (approximately 60GB vs. 120GB), reducing storage capacity requirements by approximately 30–50% (based on Mingxin FX100 fully configured reference price of approximately ¥2,014/TB). However, additional CPU memory overhead should be evaluated, and the decision should consider workload characteristics and overall TCO.