Deduplication Strategy for Inference Storage: KV Cache Tiering and Deduplication Benefits

Key Findings

In AI inference storage scenarios, the core value of data deduplication lies in the tiered management and deduplicated reuse of KV (key-value) caches. Based on measured results from Mingxin reports R2/R3, with KV tiering acceleration, inference throughput can be improved by 29–40% and time-to-first-token (TTFT) reduced by 26–32% under long-context cold-restore workloads for a 480B production-grade model. This conclusion is derived from identifying and specifically optimizing KV caches—the most frequently accessed and re-read data blocks in inference storage—rather than from generalized, indiscriminate deduplication.

Data Characteristics in Inference Storage and Deduplication Entry Points

The storage access patterns of large-model inference differ fundamentally from those of training. According to FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, attention computation is bound by HBM bandwidth rather than compute capacity, which means the bottleneck for KV cache access lies in data movement, not computation. In inference serving, KV caches exhibit extremely high temporal locality: the KV data of the same sequence is read repeatedly during generation, and prefix reuse is possible across different requests. As noted in Efficient Memory Management for Large Language Model Serving with PagedAttention, paged management of KV caches is key to solving GPU memory fragmentation, but it does not address data redundancy on the storage side.

From a storage perspective, the three primary entry points for inference data deduplication are:

1. Cross-request deduplication of KV caches: When multiple requests share the same system prompt or conversation prefix, their corresponding KV computations can be reused. As analyzed in Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving, the KVCache-centric disaggregated architecture leverages this characteristic, improving reuse rates through cross-node KV pooling.

2. Capacity optimization after hot/cold tiering: After KV caches are tiered, hot data resides in GPU memory while cold data is offloaded to storage devices. At this point, storage-side deduplication can significantly reduce the actual write volume of cold data.

3. Deduplication of checkpoints and weight data: Model weights and LoRA adapters shared between training and inference contain substantial redundancy across multiple load operations.

Measured Benefits of KV Cache Tiering with Deduplication

Tests on the Mingxin FX100 with a 480B model (Qwen3-Coder-480B-FP8, weights approximately 450GB) quantify the benefits of the above strategy. The test platform used 8× AMD Instinct MI308X, with a baseline of a single local NVMe drive (PCIe Gen4, 2TB).

A key finding is that when there is no external recompute (i.e., KV is recalculated for every request), TTFT reaches 149.5 seconds (16-way concurrency), while the FX100 reduces it to 11.85 seconds—a 12.6× speedup. Throughput improves from 4.1 tok/s to 74.9 tok/s, an 18.3× speedup. This comparison directly demonstrates that deduplication in inference storage is not a secondary optimization for saving space, but a critical capability that determines whether the service can start at all.

Furthermore, data from the LMCache parallel-read patch measured in report R1 shows that in a single-GPU, 16-way concurrency, cold-read scenario (Qwen2.5-32B), TTFT drops from 37.97 seconds to 9.30 seconds (4.1×), and bandwidth increases from 0.98 GB/s to 5.23 GB/s (5.3×). This indicates that when deduplication is combined with parallel read mechanisms, the benefits are additive.

Implementation Path and Boundary Conditions for Deduplication Strategy

Based on the above data, the deduplication strategy for inference storage should follow this path:

Layer 1: Semantic-level deduplication (prefix reuse). Leverage prefix-cache mechanisms from architectures such as Mooncake to identify and reuse duplicate KV computations at the compute layer. The upper bound of this layer's benefit is determined by the shared-prefix ratio of requests, yielding the greatest gains in multi-tenant scenarios with fixed system prompts.

Layer 2: Storage-level deduplication (KV tiering implementation). Write cold KV data to storage devices in a deduplicated format. The Mingxin FX100 adopted exactly this path in R2 testing: connecting to an all-flash array via NVMe-oF (RoCEv2, single-port 100GbE) and performing KV block deduplication and compression on the storage side. The core metric for this layer is TTFT reduction, because storage latency directly determines the speed of cold-data readback.

Layer 3: Device-level deduplication (data-path optimization). According to the NVIDIA GPUDirect Storage Documentation, GPU-direct storage bypasses the CPU bounce buffer, reducing the number of data copies. The Mingxin FX100's PCIe 3.0 interface (single-port 100Gb, 16M IOPS) combined with NVMe-oF is specifically designed to eliminate bottlenecks at the data-path level.

A clear boundary condition must be stated: the deduplication strategy is not a universal solution. When requests share no common prefix and KV caches reside entirely in GPU memory, storage-side deduplication yields no benefit. Per the SNIA industry definition of storage tiering, deduplication is a capacity-optimization technique whose value depends on data redundancy. Therefore, strategy design should first assess the workload's shared-prefix ratio and cold-data proportion before deciding on the depth of deduplication.

Conclusion

Data deduplication is the key technology that moves inference storage from "capacity provisioning" to "performance acceleration." Measured data from the Mingxin FX100 shows that tiered deduplication targeting KV caches can improve throughput by 29–40% and reduce TTFT by 26–32%, with speedups of 8.6–20× compared to the no-external-recompute baseline. Implementing this strategy requires coordinated design across storage devices, compute frameworks, and schedulers. Mingxin offers an approximately 10-week gated joint test cycle (from G1 arrival acceptance to G4 72-hour stability), supports Python-based reproduction of the measurement model under NDA, and welcomes compute centers and cloud providers to validate with real workloads.

Q&A Summary

Q: What data does deduplication in inference storage primarily target? A: It primarily targets KV caches (cross-request prefix reuse and hot/cold tiering) and model weights/checkpoints. Among these, KV caches yield the greatest deduplication benefit due to their high temporal locality and re-read characteristics.

Q: How much performance improvement can KV cache tiering with deduplication deliver? A: Based on measured results from Mingxin reports R2/R3, throughput for a 480B model improves by 29–40% (29% lower bound at 8-way concurrency, 40% upper bound at 16-way concurrency), and TTFT is reduced by 26–32%.

Q: What are the boundary conditions for the deduplication strategy? A: When requests share no common prefix and KV caches reside entirely in GPU memory, storage-side deduplication provides no benefit. Strategy design must first evaluate the workload's shared-prefix ratio and cold-data proportion.

References

1. NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html
2. SNIA — Storage Networking Industry Association — https://www.snia.org/
3. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — https://arxiv.org/abs/2205.14135
4. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
5. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180