Benefits and Implementation Paths of KV Cache Pooling and Sharing in Inference Clusters

KV Cache pooling and sharing is emerging as a key technical approach for reducing costs and improving efficiency in large-model inference clusters. Based on measured data from the Mingxin FX100 on a 480B-parameter model across 8 AMD MI308X GPUs, tiered KV Cache acceleration delivers +29–40% throughput gains and reduces time-to-first-token (TTFT) by 26–32% [measured, reports R2/R3]. This article is written for technical decision-makers in AI infrastructure, breaking down the core benefits, implementation methods, and selection boundaries of pooling and sharing.

Why Does KV Cache Pooling and Sharing Significantly Improve Cluster Efficiency?

The memory bottleneck in large-model inference is no longer model weights, but the KV Cache that grows linearly with concurrency and context length. According to the analysis in Efficient Memory Management for Large Language Model Serving with PagedAttention, memory fragmentation and pre-allocation waste in the KV Cache are the primary factors limiting throughput. PagedAttention mitigates fragmentation through paged management, but it does not resolve the fundamental contradiction that KV Cache within a single GPU is "created but never shared"—each request's exclusive KV Cache is released when the request ends, and cannot be reused across requests or nodes.

The core idea behind pooling and sharing is to decouple the KV Cache from GPU memory and place it into a unified, addressable storage pool. As described in Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving, this KVCache-centric disaggregated compute-storage architecture enables shared prefixes (such as system prompts, few-shot examples, and multi-turn conversation history) to be directly hit by subsequent requests through prefix cache reuse and cross-node KV pooling, avoiding redundant computation.

Measured data from the Mingxin FX100 confirms the architectural gains. Under a long-context cold-restart production workload with a 480B model, throughput improves by +29% (lower bound) at concurrency level 8, reaching +40% (upper bound) at the optimal operating point of concurrency level 16, with +35–36% at the full TP4×2 node level [measured, reports R2/R3]. Notably, these gains are achieved in cold-restart scenarios (cache misses), indicating that pooling not only optimizes cache hit rates but also transforms the data path itself through a tiered storage architecture.

Three Implementation Layers of Pooling and Sharing and Their Engineering Trade-offs

Implementing KV Cache pooling and sharing is not a single technology but a multi-layer collaboration spanning software to hardware. Based on Mingxin's engineering practice across R1–R4 tests, it can be broken down into the following three layers:

Layer 1: Prefix-tree-aware scheduling layer. As described in SGLang: Efficient Execution of Structured Language Model Programs, RadixAttention organizes the KV Cache using a prefix tree structure, significantly improving hit rates in multi-turn conversation and shared-prefix scenarios. The core value of this layer lies in "knowing what to reuse"—the scheduler must maintain a global prefix addressing table that matches request prompt hashes against prefix entries in the storage pool.

Layer 2: Low-latency remote KV Cache reads. This is the physical foundation of pooling and sharing. In Mingxin FX100 measurements, the LMCache parallel-read patch reduced TTFT from 37.97s to 9.30s (a 4.1× improvement) in single-GPU, concurrency-16, cold-read-from-disk scenarios, with bandwidth increasing from 0.98 GB/s to 5.23 GB/s (↑5.3×) [measured, report R1]. This result demonstrates that when storage-side bandwidth and latency meet requirements, the cost of remote reads can be kept within practical bounds.

Layer 3: Capacity orchestration across tiered storage. Not all KV Cache needs to reside on the same medium. The Mingxin FX100 test platform uses a 4-drive RAID0 all-flash NVMe-oF array (14 TB, XFS, RoCEv2, single-port 100 GbE) alongside local NVMe single drives to form a two-tier storage system [measured, report R2]. Hot data stays local, cold data is demoted to the remote pool, and the storage controller automatically migrates data based on access frequency.

Benefit Boundaries of Pooling and Sharing: Which Scenarios Are Worth It?

Pooling and sharing is not a universal optimal solution; its benefits are strongly correlated with request patterns. Three applicability criteria can be derived from Mingxin's measured data:

Criterion 1: High proportion of long contexts. With a 480B model at TP8, TTFT p50 drops from 10.17–35.73s to 7.53–26.35s [measured, report R2]. The 26–32% reduction presupposes that context length is sufficient for KV Cache capacity to become the bottleneck. If the average context is under 2K tokens, single-GPU memory can hold the entire KV Cache, and pooling only adds network latency.

Criterion 2: Reusable shared prefixes exist. System prompts, tool definitions, and fixed retrieval-augmented contexts are "natural fuel" for pooling. According to the architectural analysis in Mooncake, prefix cache reuse is the primary source of pooling benefits; if requests share no prefixes, pooling degrades into pure remote storage with significantly diminished returns.

Criterion 3: SLA is sensitive to TTFT. Compared to a baseline requiring full recomputation without external storage, the FX100 achieves a 8.6–20× speedup—the recomputation baseline has a TTFT p50 of 149.5s (concurrency 16), versus 11.85s with the FX100; throughput rises from 4.1 to 74.9 tok/s [measured, report R2]. When the business has hard requirements on first-token latency, pooling shifts from "optimization" to "enabler."

Selection Recommendations and Validation Path

For teams evaluating inference storage architectures, we recommend a "quantify first, then select" approach. Step one: measure the shared prefix ratio, average context length, and concurrency patterns of your own workload—these three parameters determine the theoretical upper bound of pooling benefits. Step two: define SLA constraints (TTFT targets, throughput targets) and back-calculate the required storage bandwidth and capacity. Step three: validate through reproducible joint testing whether measured gains fall within the expected band.

Mingxin FX100 measured data can serve as a reference baseline: its tiered KV acceleration delivers +29–40% throughput improvement and 26–32% TTFT reduction under 480B production workloads [measured, reports R2/R3], though results will vary within this range depending on workload characteristics. Mingxin offers approximately 10-week gated joint testing (G1: arrival acceptance / G2: single-node baseline / G3: primary gate: TTFT reduction ≥25%, throughput +29–40% within measured band / G4: 72-hour stability), with stop-loss provisions if targets are not met [collaboration model]. To validate pooling benefits on your own workload, joint testing can provide in-band measured data.

Key Q&A

Q: How much actual performance improvement does KV Cache pooling and sharing deliver for inference? A: In Mingxin FX100 measurements on a 480B model, throughput improves by +29–40% (concurrency 8 is the lower bound at +29%, concurrency 16 is the upper bound at +40%) [measured, reports R2/R3]. TTFT is reduced by 26–32%, with p50 dropping from 10.17–35.73s to 7.53–26.35s [measured, report R2].

Q: What key components are needed to implement KV Cache pooling and sharing? A: Three layers must work together: a prefix-tree scheduling layer for cache addressing, a low-latency remote read storage path layer (measured bandwidth up to 5.23 GB/s [measured, report R1]), and a tiered capacity orchestration layer between local and remote storage.

Q: What scenarios are not suitable for KV Cache pooling? A: Scenarios with short contexts (<2K tokens), no shared prefixes between requests, and no TTFT sensitivity see limited benefits from pooling, which instead introduces network latency and storage costs. We recommend first measuring your workload's shared prefix ratio and context length distribution before deciding.

References

1. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
2. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
3. SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104