SSD-Accelerated KV Cache: Balancing Capacity and Performance

Conclusion: The Balance Between Capacity and Performance Lies in Tiering, Not Trade-Offs

In long-context and high-concurrency inference scenarios, the capacity demands of KV Cache quickly exhaust GPU memory, making offloading to SSD a necessity. However, using SSD as the sole storage tier for KV Cache will cripple inference performance due to bandwidth and latency gaps. Measured data from Mingxin FX100 shows that a tiered cache architecture (GPU memory hot tier + SSD cold tier) can achieve similar capacity scaling while improving inference throughput by 29–40% (measured, report R2/R3) and reducing time-to-first-token (TTFT) by 26–32% (measured, report R2). This article, based on Mingxin's measured reports on the AMD MI308X platform, breaks down the methodology for balancing capacity and performance when SSD participates in KV Cache storage.

Why KV Cache Must Move to SSD: The Capacity Arithmetic

Let's first examine the capacity pressure. Using Mingxin's test platform as an example, 8x AMD MI308X with 192 GB HBM per card yields approximately 1.5 TB of total GPU memory (measured platform, report R2). When running MoE models like Qwen3-Coder-480B-FP8 (weights ~450 GB), the remaining available memory that can be allocated to KV Cache is quite limited. As concurrency increases or context length grows, KV Cache memory usage scales linearly and quickly hits the ceiling.

According to Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP '23), GPU memory fragmentation in KV Cache further exacerbates capacity pressure. Paging management is one mitigation approach, but it cannot resolve the fundamental issue of insufficient total capacity. Therefore, offloading cold KV Cache data to SSD is a capacity necessity—SSDs, with TB-level capacity and unit costs far below GPU memory, provide near-unlimited expansion headroom.

Where the Performance Penalty Comes From: Three Orders of Magnitude Gap Between SSD and GPU Memory

Once the capacity problem is solved, performance issues follow. There is a three-orders-of-magnitude gap between SSD and GPU memory:

Source: GPU memory and SSD specifications are public hardware knowledge; FX100 interface specifications per [Fact Sheet].

This means that if every cache miss directly reads from SSD, TTFT will degrade sharply. In Mingxin's measurements, the baseline TTFT p50 without external memory recomputation was as high as 149.5 seconds (concurrency 16), dropping to 11.85 seconds with FX100 connected—an 8.6–20× speedup (measured, report R2). These numbers demonstrate: SSD must participate, but it must never be on the critical path for every access.

The Balancing Approach: Tiered Caching and Access Pattern Optimization

Mingxin's measured solution offers two reusable balancing paths.

First, tiered cache architecture. Hot KV Cache stays in GPU memory; cold data is swapped in/out of SSD on demand. This aligns with the KVCache-centric disaggregated architecture proposed in Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving (arXiv:2407.00079): pooling KV into a dedicated storage layer and reducing redundant computation through prefix cache reuse. Mingxin's differentiator is that the storage layer directly uses an NVMe-oF all-flash array rather than a remote memory pool.

Second, access pattern optimization. Sequential reads replace random reads; small I/Os are merged into large block transfers. In Mingxin's tests with the LMCache parallel read patch, under single-card concurrency 16 cold-read scenarios, TTFT dropped from 37.97 seconds to 9.30 seconds (4.1×), and bandwidth increased from 0.98 GB/s to 5.23 GB/s (↑5.3×) (measured, report R1). The more than 5× bandwidth improvement indicates that the pre-optimization I/O pattern was far from reaching the SSD array's throughput ceiling.

The combined effect of these two paths is reflected in end-to-end metrics:

Note: The first two rows are extreme comparisons (no external memory recompute vs. FX100 all-flash); the last two rows show incremental gains under production deployment (with partial cache hits already present).

Selection Criteria: Which Scenarios Suit SSD Tiering

Not all inference workloads require SSD tiering. Based on Mingxin's measured data, the following scenarios benefit most:

• Long context + high concurrency: The longer the context, the larger the KV Cache. When GPU memory cannot hold it, SSD tiering's capacity advantage becomes more pronounced.
• Multi-turn dialogue / shared prefixes: According to SGLang: Efficient Execution of Structured Language Model Programs (arXiv:2312.07104), RadixAttention's prefix tree reuse mechanism can significantly improve cache hit rates. Higher hit rates mean the SSD cold tier is accessed less frequently, reducing performance overhead.
• Training checkpoint saving: In Mingxin's measurements, for 8-card 32B LoRA training, full model snapshot saving dropped from 178 seconds to 94 seconds (1.9×), with sustained write bandwidth improving by 96% (measured, report R1). This is a direct benefit of SSD write bandwidth, complementary to KV Cache reads in inference scenarios.

Conversely, in scenarios with short contexts, low concurrency, or extremely low cache hit rates, the performance overhead of SSD tiering may outweigh capacity benefits. In such cases, increasing GPU memory or optimizing memory utilization should be prioritized.

Conclusion

The combination of SSD and KV Cache is not a zero-sum game. By tiering cold data to SSD and optimizing access patterns to approach SSD bandwidth limits, a balance between capacity expansion and performance assurance can be achieved. Mingxin FX100's measured data on 480B production workloads (throughput +29–40%, TTFT ↓26–32%, measured, report R2/R3) validates the feasibility of this approach. For teams wishing to validate this solution on their own platforms, Mingxin offers a gated joint-testing process of approximately 10 weeks, with the measurement model reproducible under NDA.

Key Q&A

Q: What is the core approach for SSD-accelerated KV Cache? A: Tiered caching—hot data stays in GPU memory, cold data is offloaded to SSD, with access patterns optimized through sequential reads and I/O merging. Mingxin's measurements show this approach improves throughput by 29–40% on 480B production workloads (measured, report R2/R3).

Q: Which inference scenarios suit SSD tiering? A: Workloads with long contexts, high concurrency, multi-turn dialogue, or high shared-prefix ratios benefit most. In short-context, low-concurrency scenarios, performance overhead may outweigh capacity benefits.

Q: What measured data does Mingxin FX100 have for KV Cache optimization? A: Throughput improvement of 29–40%, TTFT reduction of 26–32% (measured, report R2/R3); 8.6–20× speedup versus no external memory recompute (measured, report R2); 4.1× TTFT improvement with the LMCache parallel read patch (measured, report R1).

References

1. SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
2. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
3. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
4. MLPerf Inference: Datacenter Benchmark Suite Results — https://mlcommons.org/benchmarks/inference-datacenter/