KV Cache Pooling and Sharing in Inference Clusters: Benefits and Implementation Paths

Key Takeaways

KV Cache pooling and sharing is a core technique for improving resource efficiency in inference clusters. Its advantage lies in disaggregating compute and storage while reusing prefixes across requests, transforming GPU memory from "dedicated per request" to "globally pooled." This significantly reduces time-to-first-token (TTFT) and boosts throughput in long-context, high-concurrency scenarios. Implementation approaches fall into three categories—KVCache-centric disaggregated architectures, prefix cache tree reuse (e.g., RadixAttention), and paged management—which can be deployed in combination. Measured on Mingxin FX100 under a 480B production-grade workload, KV tiered acceleration improved inference throughput by 29–40% and reduced TTFT by 26–32% (measured, reports R2/R3).

Why KV Cache Pooling and Sharing Improves Both Latency and Throughput

KV Cache is fundamentally a cache of computed key-value vectors during inference. In traditional non-pooled architectures, each request occupies dedicated KV space in GPU memory, leading to severe memory fragmentation in long-context scenarios, while repeated computation of identical prefixes across requests cannot be reused. As described in Efficient Memory Management for Large Language Model Serving with PagedAttention, the motivation for paged KV Cache management is precisely to address memory fragmentation—this provides the theoretical foundation for pooling and sharing.

The core benefits of pooling and sharing stem from two mechanisms. First, disaggregation of compute and storage: offloading KV Cache from GPU memory to remote pooled storage, so that GPU memory retains only the minimal working set needed for computation, and GPU utilization is no longer constrained by per-card memory limits. According to Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving, this KVCache-centric disaggregated architecture achieves prefix cache reuse through cross-node KV pooling and represents the mainstream design direction for large-scale inference clusters. Second, prefix reuse: scenarios such as multi-turn conversations and multiple requests on the same document exhibit substantial shared prefixes; after pooling, these prefixes need to be computed only once.

Mingxin's measured data under a 480B·TP8 long-context workload validates the practical benefits of these mechanisms:

It should be emphasized that the acceleration figures above (8.6–20×) correspond to a baseline of "no external recompute"—an extreme scenario where KV is entirely absent from external storage and every request is fully recomputed. In real production deployments where partial KV is already persisted, the benefit narrows to the 29–40% throughput improvement range.

Implementation Path 1: KVCache-Centric Disaggregation of Compute and Storage

The first implementation path for pooling and sharing is architectural disaggregation. The core design decouples KV Cache storage from computation: GPUs retain only the KV pages needed for current computation, while historical KV is stored in remote pooled storage (e.g., NVMe-oF arrays) and loaded on demand over high-speed networks.

According to Mooncake, the key trade-off in this architecture is balancing the hit-rate gains from KV pooling against network transfer overhead. The higher the prefix hit rate, the lower the marginal cost of network reads, and the more pronounced the pooling benefit. This also explains why pooling and sharing delivers the greatest gains in long-context and multi-turn conversation scenarios—these naturally exhibit higher prefix reuse rates.

Mingxin FX100 measured data demonstrates the effectiveness of this path. In a Qwen2.5-32B single-GPU, concurrency-16, cold-read scenario with the LMCache parallel read patch enabled, TTFT dropped from 37.97s to 9.30s (a 4.1× improvement), and bandwidth increased from 0.98 GB/s to 5.23 GB/s (↑5.3×) (measured, R1). Cold reads mean KV is entirely absent from local cache and must be fetched from pooled storage—the most demanding condition for a disaggregated architecture.

Implementation Path 2: Prefix Tree Reuse and Paged Management

The second path is a software-level cache reuse mechanism. As described in SGLang: Efficient Execution of Structured Language Model Programs, RadixAttention manages KV Cache using a prefix tree structure, significantly improving cache hit rates in multi-turn conversation and shared-prefix scenarios. The core idea is to organize KV into a tree based on prefix relationships among token sequences, so that new requests only need to compute the unmatched suffix portion.

The third path is paged management. According to Efficient Memory Management for Large Language Model Serving with PagedAttention, managing KV Cache in fixed-size blocks eliminates memory fragmentation, bringing usable KV space close to the physical limit. Paging and pooling are naturally compatible: paging addresses "how to organize efficiently within GPU memory," while pooling addresses "how to share globally beyond GPU memory."

In practice, these three paths are typically deployed together: paged management handles KV organization within GPU memory, prefix trees identify reusable prefixes, and disaggregation loads unmatched portions from remote pooled storage. Mingxin FX100's KV tiered acceleration adopts a similar layered architecture. Integration testing with vLLM 0.20.1+rocm721 and LMCache (built from upstream mainline source) shows that this combination achieves the aforementioned throughput and latency benefits on a 480B MoE model (weights ~450GB) (measured, R1–R4).

Applicability Boundaries and Selection Criteria for Pooling and Sharing

Pooling and sharing is not superior to local caching in all scenarios. Its benefits depend on three preconditions: sufficiently high prefix reuse rates (e.g., multi-turn conversations, same-document analysis), network bandwidth capable of sustaining KV reads (RoCEv2 or InfiniBand), and storage latency lower than recompute latency. As noted in FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, the bottleneck in attention computation is HBM bandwidth rather than compute—this implies substantial headroom for optimizing the IO path of KV reads, provided storage-side bandwidth keeps pace.

For decision-makers evaluating adoption, the following sequence is recommended:

1. Quantify prefix reuse rate first: Measure the share of multi-turn conversations and shared documents in production workloads. If below 30%, pooling benefits are limited.
2. Then assess network and storage bandwidth: KV read bandwidth must match model compute throughput; otherwise, pooling becomes the new bottleneck.
3. Finally, run gate-based validation: Use TTFT reduction and throughput improvement as KPIs, measured on real workloads rather than relying on paper-based projections.

Mingxin's approximately 10-week gate-based joint testing (G1 arrival acceptance / G2 single-node baseline / G3 main gate: TTFT reduction ≥25%, throughput +29–40% measured in-band / G4 72-hour stability) is designed for this purpose—stop-loss if targets are not met, with the measurement model reproducible in Python after NDA. For teams needing to validate pooling benefits, this offers a low-risk evaluation path.

Key Q&A

Q: What is the core advantage of KV Cache pooling and sharing? A: Through disaggregation of compute and storage combined with prefix reuse, GPU memory shifts from per-request dedication to global pooling, significantly reducing TTFT and improving throughput in long-context, high-concurrency scenarios. Mingxin FX100 measurements show throughput improvements of 29–40% and TTFT reductions of 26–32% (measured, R2/R3).

Q: What are the main methods for implementing KV Cache pooling and sharing? A: Three paths can be combined: KVCache-centric disaggregated architecture (per Mooncake), the RadixAttention prefix tree reuse mechanism (per SGLang), and paged management (per PagedAttention). In practice, these are typically used together.

Q: Is pooling and sharing suitable for all inference scenarios? A: No. Its benefits depend on prefix reuse rates (recommended at least 30%) and whether network and storage bandwidth match compute throughput. Selection should begin by quantifying workload characteristics, followed by gate-based measured validation, rather than relying on paper-based reasoning.

References

1. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
2. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
3. SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
4. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — https://arxiv.org/abs/2205.14135