KV Cache Network Architecture: Design Considerations for Hyperscale Data Centers

In hyperscale data centers, the storage and access of KV Cache are becoming a critical bottleneck for inference performance. Network architecture design must balance throughput, latency, and cost. In the 480B production deployment configuration, Mingxin FX100 delivers 29–40% inference throughput improvement through tiered KV acceleration (measured, reports R2/R3), providing a quantitative reference for network architecture selection. This article examines three dimensions: network plane partitioning, transport protocol selection, and storage deployment.

Why Does KV Cache Require a Dedicated Network Plane?

The access pattern of KV Cache is fundamentally different from training data flows. According to the analysis in Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP '23), KV Cache exhibits memory fragmentation and dynamic growth characteristics, which dictate that its network requirements are low-latency random reads rather than high-bandwidth sequential writes. In hyperscale data centers, mixing KV Cache traffic with training traffic causes mutual interference; therefore, the design should allocate a dedicated network plane.

Measured data from Mingxin report R2 shows that under 480B·TP8 with three concurrency levels, TTFT p50 dropped from 10.17–35.73s to 7.53–26.35s, a reduction of 26–32%. This result is predicated on the absence of network congestion contention on the KV Cache access path. If the network were shared with training traffic, latency jitter would directly propagate to inference SLAs.

The granularity of network plane partitioning should align with the expansion unit. According to the reference architecture described in NVIDIA DGX SuperPOD - NVIDIA Docs, large-scale GPU clusters are organized in compute/storage/network layers, with the expansion unit as the basic scaling increment. The KV Cache network can follow this layered approach, but the specific number of planes and bandwidth ratios must be calculated based on actual workload models—the DGX SuperPOD documentation does not define a specific KV Cache network design.

Transport Protocol: RDMA or TCP?

The core tension in KV Cache networking is the need for both low latency (microsecond-level) and high throughput (hundreds of Gbps). According to RFC 5040: A Remote Direct Memory Access Protocol Specification, RDMA defines kernel bypass and zero-copy semantics, suitable for low-latency scenarios. Meanwhile, RFC 9293: Transmission Control Protocol (TCP) provides reliable byte streams but with significant protocol stack overhead.

In Mingxin report R1 measurements, the LMCache parallel read patch under single-GPU·concurrency-16·cold disk read (Qwen2.5-32B) reduced TTFT from 37.97s to 9.30s (4.1×), and bandwidth increased from 0.98 to 5.23 GB/s (↑5.3×). This test was conducted on a RoCEv2 network with a single 100 GbE port, demonstrating the significant advantage of the RDMA path in cold-read scenarios.

It should be clarified that RoCEv2 is not defined by the InfiniBand specification—according to the InfiniBand Specification Frequently Asked Questions, IBTA is responsible for the InfiniBand specification, while the RoCE encapsulation format is advanced through collaboration between IBTA and IEEE; the two standards have different ownership. Design selection should verify protocol versions and switch support matrices separately.

How Does the Storage Layer Coordinate with the Network?

The choice of KV Cache storage medium directly influences network load characteristics. Mingxin FX100 employs an NVMe-oF all-flash array with 4-drive RAID0 (14 TB, XFS), connected via a single 100 GbE RoCEv2 port. R2 measurements show that under 480B long-context cold-recovery workloads, throughput improves 29% at concurrency level 8 (lower bound), and 40% at the optimal operating point of concurrency 16 (upper bound); at the full-machine level with TP4×2, the improvement is 35–36%.

Key design points for the storage layer: first, RAID strategy must match network bandwidth—when individual drive performance exceeds demand and the network becomes the bottleneck, RAID0's parallel read capability cannot be fully utilized; second, the metadata operation path of the file system format (e.g., XFS) affects cold-start latency; third, cache eviction policies should integrate with network topology awareness to reduce cross-switch access.

According to Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving, the KVCache-centric disaggregated architecture reduces redundant computation through prefix cache reuse and cross-node KV pooling. This design direction aligns with Mingxin's measured results, but the Mooncake paper does not provide quantifiable network performance data; specific benefits must be validated on the target cluster.

Cost-Performance Trade-offs

Network architecture design must make explicit trade-offs between performance and cost. RDMA networks require lossless Ethernet configurations (PFC, ECN), with switch costs higher than standard Ethernet; while the TCP path is simpler to deploy, its latency does not meet the SLA requirements of the KV Cache read path. According to SGLang: Efficient Execution of Structured Language Model Programs, RadixAttention improves multi-turn conversation hit rates through prefix tree reuse—this suggests that network design should prioritize latency for high-hit-rate scenarios rather than pursuing peak bandwidth indiscriminately.

Mingxin report R9 measurements (Ascend platform) show model inference loading acceleration (vs. NFS) of 6.2–9.3×: DeepSeek-32B service loading from 691s → 112s, DeepSeek-70B from 1399s → 150s. This result demonstrates that dedicated acceleration devices provide significant advantages over general-purpose protocols like NFS on the storage access path. However, note that this test was conducted on the Huawei Atlas 910B platform; cross-platform comparisons require caution.

Key Q&A Summary

Q: How should network planes be partitioned for KV Cache architecture? A: It is recommended to separate KV Cache traffic from training traffic by allocating a dedicated network plane. The specific number of planes and bandwidth ratios should be calculated based on workload models, following the layered approach of DGX SuperPOD as a reference—but that document does not define a specific KV Cache network design.

Q: How to choose between RDMA and TCP? A: For the KV Cache read path, RDMA (e.g., RoCEv2) is recommended; Mingxin report R1 measurements show a 4.1× TTFT improvement in cold-read scenarios. TCP can be used for the control plane and backup paths. Note that RoCE and InfiniBand have different standards ownership.

Q: How can the storage layer and network be jointly optimized? A: Storage medium, RAID strategy, and network bandwidth must be designed to match. Mingxin report R2 measurements show a 40% throughput improvement at concurrency 16 under 480B workloads, but specific benefits must be validated on the target cluster.

References

1. SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
2. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
3. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
4. InfiniBand Specification Frequently Asked Questions — https://www.infinibandta.org/ibta-specification/
5. RFC 9293: Transmission Control Protocol (TCP) — https://datatracker.ietf.org/doc/html/rfc9293
6. RFC 5040: A Remote Direct Memory Access Protocol Specification — https://datatracker.ietf.org/doc/html/rfc5040
7. NVIDIA DGX SuperPOD - NVIDIA Docs — https://docs.nvidia.com/dgx-superpod/