KV Cache Storage Configuration Guide for Prefill and Decode Phases

KV Cache storage configuration should not be a one-size-fits-all approach: the prefill and decode phases have fundamentally different storage requirements. The former demands high bandwidth to rapidly write large volumes of KV data, while the latter requires low latency to ensure responsive token-by-token generation. Measured on the Mingxin FX100 under a 480B production deployment with long-context cold-recovery workloads, tiered acceleration improved inference throughput by 29–40% (measured, reports R2/R3). The key lies in optimizing the storage path separately for each phase. Based on Mingxin measured data and public architectural research, this article provides a phase-specific storage configuration methodology.

Prefill Phase: Why Bandwidth Matters More Than Latency

The prefill phase processes input prompts, computing and writing a large volume of KV Cache in a single pass. This phase represents the peak of storage write activity, and its core constraint is bandwidth rather than latency. According to Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP '23), KV Cache memory management directly limits throughput ceilings, and the data volume in the prefill phase determines the peak of memory pressure.

In Mingxin's measured report R2, with a 480B model (MoE, weights approximately 450GB) under TP8 at three concurrency levels, the baseline TTFT p50 without external memory recomputation reached as high as 149.5s (at concurrency level 16). With the FX100 connected, this dropped to 11.85s—an order-of-magnitude difference driven by relieving the prefill-phase storage bandwidth bottleneck (measured, report R2). Configuration recommendations:

• Storage medium: Prioritize NVMe-oF all-flash arrays; per-drive bandwidth must cover the model's peak concurrent write demand.
• Queue depth: In the prefill phase, increase the concurrent write queue depth as appropriate, trading per-request latency for overall bandwidth utilization.
• Caching strategy: Enable prefix-tree reuse for shared prefixes (e.g., system prompts, few-shot examples). According to SGLang: Efficient Execution of Structured Language Model Programs (arXiv:2312.07104), the RadixAttention mechanism can significantly improve hit rates in multi-turn dialogue and shared-prefix scenarios, reducing redundant writes.

Decode Phase: Latency-Sensitive, Bandwidth Demand Drops Sharply

The decode phase generates output token by token, reading only the latest KV Cache increment each step. Storage access in this phase is characterized by high-frequency, small-data reads, and latency directly determines token generation speed after TTFT. Mingxin's measured report R2 shows that first-token latency (TTFT) decreased by 26–32% after optimization, with p50 dropping from 10.17–35.73s to 7.53–26.35s (measured, report R2)—this lays the foundation for stable output during the decode phase.

The core principle for decode-phase storage configuration is avoid paying for high bandwidth:

• Capacity over bandwidth: The decode phase needs sufficient KV Cache capacity to support long contexts, not extreme read/write bandwidth.
• Localized reads: Keep hot KV data resident in GPU memory or local NVMe as much as possible to reduce cross-node access. According to Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving (arXiv:2407.00079), a KVCache-centric disaggregated architecture balances capacity and latency through prefix cache reuse and cross-node KV pooling.
• Prefetch mechanism: Leverage the sequential access pattern of the decode phase to prefetch the next batch of required KV data from remote storage to local storage in advance.

Tiered Storage: One Configuration Serving Both Phases

A single storage configuration cannot simultaneously satisfy the high bandwidth of prefill and the low latency of decode; a tiered approach is the more pragmatic choice. Mingxin FX100's KV tiered acceleration design follows this approach: a hot tier (GPU memory/local NVMe) serves the decode phase, while a cold tier (NVMe-oF array) handles prefill writes and long-context overflow.

In Mingxin's measured report R1, with the LMCache parallel-read patch under single-GPU concurrency 16 cold-read scenarios (Qwen2.5-32B), TTFT dropped from 37.97s to 9.30s, and bandwidth increased from 0.98 GB/s to 5.23 GB/s (↑5.3×) (measured, report R1)—this data demonstrates that even in cold-recovery scenarios, parallel read optimization on the storage side can yield order-of-magnitude improvements.

Configuration Practice: Deriving Storage Parameters from SLAs

Storage configuration should not start from hardware specifications but from service-level agreements (SLAs). The steps are:

1. Determine concurrency levels and context length: These define the total KV Cache capacity requirement and peak write bandwidth.
2. Set TTFT and throughput targets: Mingxin's joint-testing gate requires TTFT reduction ≥25% and throughput improvement of 29–40% within the measured band (approximately 10 weeks of gate-based joint testing, G3 main gate), which can serve as a reference baseline.
3. Split the storage budget by phase: Configure a high-bandwidth cold tier for the prefill phase and a low-latency hot tier for the decode phase, linked through caching policies.
4. Validate the cold-recovery path: In long-context scenarios, the storage read efficiency of cold recovery (e.g., service restart, instance migration) is often overlooked. In Mingxin's measured report R2, full-machine throughput at TP4×2 improved by 35–36% (measured, report R3), with part of the gain attributable to cold-recovery path optimization.

It should be noted that the above configuration methodology assumes the storage system supports remote direct access protocols such as NVMe-oF. According to NVIDIA GPUDirect Storage documentation, GPU-direct storage can bypass the CPU bounce buffer to shorten the data path, but applicability depends on the specific implementation of the GPU and storage system (NVIDIA official documentation). For environments without direct-connect support, the additional latency introduced by CPU relay must be evaluated.

Key Q&A

Q: What is the core metric for prefill-phase storage configuration? A: Bandwidth takes priority over latency. The prefill phase writes a large volume of KV Cache in one pass, and storage bandwidth determines the TTFT floor. Mingxin's measured report R2 shows TTFT p50 dropped from 149.5s to 11.85s after optimization (concurrency level 16).

Q: How should decode-phase storage configuration be balanced? A: It is latency-sensitive with low bandwidth demand. Configure a low-latency local hot tier (GPU memory/local NVMe) and avoid over-provisioning for high bandwidth. Combined with prefix cache reuse mechanisms, redundant reads can be reduced.

Q: How can storage configuration compliance be verified? A: Derive from SLAs—set TTFT and throughput targets, then validate by phase. Mingxin's joint-testing gate uses TTFT reduction ≥25% and throughput improvement of 29–40% as the measured in-band standard, with the cold-recovery path requiring separate testing.

Mingxin Technology provides FX100/FX200/FX300 series storage acceleration products, supporting approximately 10 weeks of gate-based joint testing (G1 arrival acceptance / G2 single-node baseline / G3 main gate / G4 72-hour stability), with stop-loss measures if targets are not met. To validate phase-specific storage configurations under a particular workload, measurements can be taken directly during joint testing.

References

1. SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
2. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
3. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
4. NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html