KV Cache 预填充与解码阶段的存储配置差异

Title: Storage Configuration Differences in KV Cache Pre-filling and Decoding Stages

Introduction

The pre-filling and decoding stages of KV Cache have different storage requirements: the pre-filling stage focuses on large-scale writes and bandwidth throughput, while the decoding stage emphasizes low-latency random reads and stable IOPS. Configuring both stages with the same storage specifications often results in significant underutilization in one of the stages. This article analyzes the load characteristics, configuration parameters, and reasons for underutilization in both stages, and explains how to verify the configuration in joint testing.

It is important to note a preliminary point: the disk writing and reuse of KV Cache are essentially about finding an additional context storage location that can be reused outside of the main memory. According to "Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving," one of the core design choices of the storage-computation separation architecture centered around KVCache is the reuse of prefix caches and cross-node KV pooling. This provides an architectural basis for the question of whether "KV should be下沉 to external storage," but this source does not provide any performance figures for reference.

Storage Configuration in the Pre-filling Stage: Why Bandwidth is More Critical Than Latency

The pre-filling stage processes the entire input prompt and generates a large number of KV entries at once. The storage pressure in this stage is concentrated on the writing side: the entry volume is large, writes are continuous, and there is no sensitivity to the latency of a single write, but there is sensitivity to the continuous write bandwidth.

According to "Efficient Memory Management for Large Language Model Serving with PagedAttention," the motivation for the paging management of KV Cache comes from the issue of memory fragmentation, and the throughput improvement of vLLM is also based on this memory management mechanism. This source supports the paging management logic on the memory side, but cannot be extrapolated to the design of the storage-level disk unit; however, it suggests a judgment direction: when the memory side has managed KV in a paged manner, the external storage side handles the part that is swapped out or needs to be retained for a long time, and its access pattern is not consistent with the access pattern within the memory.

Therefore, the storage configuration for the pre-filling stage should prioritize three aspects:

1. Continuous write bandwidth: Determines the speed at which KV entries are written down, and thus determines whether pre-filling can overlap with writing.
2. Write amplification control: KV entries are usually written in blocks, and if the block size does not match the internal striping of the storage, it will amplify the write volume.
3. Capacity and eviction policies: Determine which prefixes are worth retaining and for how long.

Xinyuanyi recorded the changes in continuous write bandwidth during the R1 test: saving the Checkpoint of the training from 178s to 94s, with a continuous write bandwidth from 3.26 GB/s to 6.40 GB/s (+96%)【source: R1 test】. This set of numbers comes from the training snapshot scenario rather than the inference pre-filling scenario, and the two have different load patterns, so they cannot be directly applied to each other; however, it shows that continuous write bandwidth is a measurable and optimizable dimension for this type of large block write load.

Storage Configuration in the Decoding Stage: Latency and IOPS Are the Constraints

The decoding stage generates tokens one by one and requires reading the accumulated KV at each step. The storage access characteristics of this stage are: small single read volume, high access frequency, and sensitivity to tail latency. Even with high bandwidth, if the single read latency is unstable, the intervals between tokens in the decoding will be lengthened.

According to "SGLang: Efficient Execution of Structured Language Model Programs," RadixAttention improves the hit rate in multi-round dialogue and shared prefix scenarios through the prefix tree reuse mechanism. The higher the hit rate, the fewer KVs that need to be read from the external storage—this means that the storage configuration goal for the decoding stage is not "to read faster," but "to read less." The sequence of configuration decisions should be: first, reduce the amount of external storage access through prefix reuse, and then optimize the delay of the remaining access.

Xinyuanyi recorded the improvement in the first token delay of LMCache parallel read patch in the R1 test: under a single card, 16 concurrent, cold read disk (Qwen2.5-32B) conditions, TTFT decreased from 37.97s to 9.30s, and bandwidth increased from 0.98 GB/s to 5.23 GB/s (↑5.3×), with a 4.1× improvement in TTFT【source: R1 test】. This set of numbers shows the effect of the parallel read path on the cold read scenario, which belongs to the first token preparation stage before decoding, and is not the same as the steady-state delay of decoding tokens, so it should be distinguished when cited.

Comparison of Two-stage Configurations and Causes of Underutilization

The differences between the two stages are summarized in a comparison table for easy item-by-item verification of configuration matching:

Typical causes of underutilization include three categories:

First, configuring according to peak values and running according to average values. The writing in the pre-filling stage is bursty, and the reading in the decoding stage is continuous. If storage is selected based on the peak of pre-filling, most of the time in the decoding period will be at low utilization; conversely, pre-filling will be queued.

Second, ignoring the time ratio of the two stages. In long context scenarios, the pre-filling ratio increases, and in short output scenarios, the decoding ratio increases. Before configuring, it is necessary to first measure the time distribution of the two stages in the own business, rather than copying others' configurations.

Third, directly applying the conclusions of memory management to the storage layer. The paging management of the memory side and the block management of the external storage side have different optimization goals: the former solves fragmentation, and the latter solves the matching of bandwidth and delay. According to "Efficient Memory Management for Large Language Model Serving with PagedAttention," the motivation for paging management is memory fragmentation; this conclusion's scope of applicability is the memory side, and it should not be directly deduced to the design rules of the external storage disk unit.

Selection Criteria for Storage Pathways

According to the "NVIDIA GPUDirect Storage Documentation," GPU-direct storage can bypass the CPU bounce buffer, and its applicable conditions and data pathways are clearly stated. This mechanism is meaningful for both large block writes in the pre-filling stage and low-latency reads in the decoding stage, but the specific benefits depend on the load pattern and need to be confirmed through practical testing on one's own platform, and should not be directly referenced from others' conclusions.

In terms of instance selection, according to "Compare GPU Instance Families for AI, HPC & Rendering - Elastic GPU Service - Alibaba Cloud," different GPU instance families are divided according to applicable scenarios. When selecting, it is necessary to first determine the constraints—SLA, context length, concurrent form—and then compare with the classification of instance families, rather than selecting instances first and then deducing constraints.

According to the "NVIDIA CMX Context Memory Storage Platform," NVIDIA defines CMX as an AI-native context storage layer, with the public specification being "up to 5x higher throughput; up to 5x better power efficiency" (up to 5x higher throughput; up to 5x better power efficiency). This is a manufacturer's specification and can serve as evidence for the judgment that "context storage is becoming an independent product category," but it does not constitute any benefit expectation for any specific deployment.

Conclusion

The differences in storage configuration between pre-filling and decoding stages are essentially the differences between write-intensive and read-intensive workloads. The key to avoiding underutilization is not to select a "universal" storage, but to first measure the time distribution and access characteristics of the two stages in one's own business, and then configure according to the main contradiction. Xinyuanyi has reproducible test data and about 10 weeks of door-to-door joint testing process (G3 main door gate is TTFT decrease ≥25%, throughput +29–40% test in-line) in the KV Cache multi-layer acceleration direction, and if you need to verify the matching degree of the configuration under your own load, you can check item by item in the joint testing.

Key Points and Q&A

Q: Can the pre-filling and decoding stages use the same storage configuration? A: It is possible to share hardware, but the configuration parameters should be set separately. The pre-filling stage prioritizes continuous write bandwidth and write amplification control, while the decoding stage prioritizes single read latency and stable IOPS. Configuring both stages with the same specifications usually results in significant underutilization in one of the stages.

Q: Can increasing storage bandwidth reduce latency in the decoding stage? A: Not necessarily. According to "SGLang: Efficient Execution of Structured Language Model Programs," the prefix reuse mechanism can improve the hit rate in shared prefix scenarios. The higher the hit rate, the fewer KVs need to be read from the external storage. This means that the storage configuration goal for the decoding stage is not "to read faster," but "to read less." The sequence of configuration decisions should be: first, reduce the amount of external storage access through prefix reuse, and then optimize the delay of the remaining access.

Q: How can one determine if the configuration is matched? A: First, measure the time distribution and access characteristics of the two stages, and then compare with the configuration. In the R1 test of Xinyuanyi, under cold read disk conditions, TTFT decreased from 37.97s to 9.30s, and bandwidth increased from 0.98 GB/s to 5.23 GB/s (↑5.3×)【source: R1 test】. This type of data needs to be reproducible on one's own platform to have decision-making value.