LMCache Parallel Read: TTFT Reduction to 9.3 Seconds
In large model inference scenarios, the first token latency (TTFT) during cold start is a critical bottleneck affecting user experience
In large model inference scenarios, the first token latency (TTFT) during cold start is a critical bottleneck affecting user experience. By introducing the LMCache parallel read patch paired with a high-performance storage acceleration solution, measured results show TTFT can be reduced from 37.97 seconds to 9.30 seconds, a 4.1x improvement. The core of this breakthrough lies in: parallelizing the originally serial KV Cache read process, eliminating the blocking of the compute pipeline by storage I/O wait. This article will analyze from three dimensions: technical principles, measured validation, and deployment considerations, to help technical decision-makers understand the feasibility and applicable scenarios of this optimization path.
Technical Principle: Why LMCache Needs a Parallel Read Patch
LMCache is a widely used KV Cache management framework within the current vLLM ecosystem. Its core function is to cache computed KV tensors to external storage (such as NVMe SSD or distributed storage) to avoid redundant computation during long-context inference or conversation continuation. However, LMCache's default implementation has a key performance flaw—serial reading.
In the standard process, when LMCache reads KV Cache from external storage, it initiates multiple I/O requests sequentially, starting the next only after the previous one completes. For large models (e.g., Qwen3-Coder-480B-FP8), the amount of KV Cache data that needs to be loaded for a single inference can reach several GB. Serial I/O leads to storage bandwidth utilization below 20%. Taking a single-GPU, concurrency 16, cold read scenario as an example, measurements show that the default LMCache's TTFT is as high as 37.97 seconds, while the storage bandwidth is only 0.98 GB/s, far below the 6–8 GB/s theoretical bandwidth of modern NVMe SSDs.
The core idea of the parallel read patch is: splitting KV Cache read requests into multiple independent subtasks, initiating them simultaneously via multithreading or asynchronous I/O, leveraging the storage device's parallel capability to maximize bandwidth. In specific implementation, the patch treats the KV tensors of each Transformer layer as independent data blocks. A scheduler issues read requests for all blocks at once, and then aggregates results via a callback mechanism. This design changes I/O wait time from "serial accumulation" to "parallel overlap," theoretically increasing storage bandwidth utilization to over 80%.
Measured Validation: Quantitative Basis for the 4.1x TTFT Improvement
Measured tests conducted on the Mingxin FX100 all-flash NVMe-oF array (report ID R1) validate the effectiveness of the parallel read patch. The test environment was 8 × AMD Instinct MI308X GPUs, model Qwen2.5-32B, concurrency 16, cold read scenario (i.e., first load with no cache). The baseline was a local single NVMe drive (PCIe Gen4, 2 TB), and the comparison group was the FX100 array (4-drive RAID0, 14 TB, RoCEv2 100 GbE).
Key data are as follows:
- TTFT Reduction: From 37.97 seconds to 9.30 seconds, a 4.1x improvement.
- Storage Bandwidth Increase: From 0.98 GB/s to 5.23 GB/s, a 5.3x improvement.
- Bandwidth Utilization: The theoretical bandwidth of the FX100 array is approximately 8 GB/s (NVMe bandwidth of 4-drive RAID0), the measured 5.23 GB/s corresponds to about 65% utilization, significantly higher than the 12% of serial mode.
The root of this improvement is: the parallel read patch changes the I/O pattern from a "sequential queue" to a "concurrent flood," fully exploiting the parallel capability of the NVMe-oF array. It is worth noting that the TTFT improvement factor (4.1x) is slightly lower than the bandwidth improvement factor (5.3x), due to fixed overheads in data aggregation and tensor reassembly on the GPU side. However, the overall effect is significantly better than the serial solution.
Deployment Considerations: Applicable Conditions and Performance Boundaries of the Parallel Read Patch
The parallel read patch is not a universal optimization; its effectiveness is highly dependent on the parallel capability of the storage subsystem. The following are three key factors technical decision-makers need to evaluate:
- Storage Device Parallelism: Patch performance is positively correlated with the storage device's IOPS and bandwidth. In the measured test, the FX100 array's 4-drive RAID0 configuration provided a 5.3x bandwidth improvement; if using a single NVMe drive, the bandwidth improvement might drop to 2–3x. For distributed storage or HDD solutions, due to higher I/O latency, the benefits of the parallel read patch might be offset by network or disk seek overhead.
- Model Scale and Concurrency: Larger models (e.g., 480B parameters) have larger KV Cache data volumes, making the benefits of parallel reading more pronounced. At lower concurrency (e.g., conc1), the difference between serial and parallel reading may shrink because the single I/O volume is insufficient to saturate storage bandwidth.
- Compatibility with vLLM: The current parallel read patch has been integrated into the LMCache upstream mainline (compiled from source 2026-06-29), but vLLM version matching must be ensured (tests based on vLLM 0.20.1+rocm721). If using older vLLM or custom branches, compatibility of the patch interface needs verification.
Furthermore, in actual deployment, it is recommended to combine it with a KV Cache tiered acceleration strategy. Mingxin FX100's measured tests on a 480B model indicate that when the parallel read patch is used together with KV tiered acceleration (caching hot data in GPU memory, cold data in external storage), TTFT can be further reduced by 26–32% (data from report R2), and throughput increased by 29–40% (data from report R3). This combined solution performs particularly well in long-context cold recovery workloads.
Conclusion
The LMCache parallel read patch achieves a 4.1x TTFT improvement by converting serial I/O to concurrent I/O, given support from storage hardware. For data center decision-makers, the feasibility of this optimization path has been validated through measured tests on the Mingxin FX100. If you are evaluating the impact of storage acceleration solutions on inference performance, please contact the Mingxin technical team for joint testing support. We can provide an approximately 10-week gated validation process to ensure the promised TTFT reduction of ≥25% can be reproduced in measured tests.
Key Q&A for This Article
Q: How does the LMCache parallel read patch achieve TTFT reduction? A: By changing KV Cache reading from serial to multi-threaded parallel I/O, fully utilizing storage device bandwidth. In measured tests, bandwidth increased from 0.98 GB/s to 5.23 GB/s, TTFT decreased from 37.97 seconds to 9.30 seconds, a 4.1x improvement (source: measured, report R1).
Q: What hardware conditions are required to deploy the parallel read patch? A: A storage subsystem supporting high IOPS and bandwidth is required, such as an NVMe-oF array. The benefit with a single NVMe drive is limited (~2–3x). Benefits may be offset by latency in distributed storage or HDD solutions.
Q: What is the relationship between the parallel read patch and KV tiered acceleration? A: They can be used together. The parallel read patch optimizes cold data loading, while tiered acceleration keeps hot data in memory. The combined solution, measured on a 480B model, shows TTFT reduction of 26–32% and throughput increase of 29–40% (source: measured, reports R2/R3).