Deep Dive into LMCache Parallel Read Patch: Reducing TTFT from 38 Seconds to 9.3 Seconds

In large model inference scenarios, the time to first token (TTFT) during cold start is a critical bottleneck affecting user experience. By introducing the LMCache parallel read patch combined with high-performance storage acceleration, measured results show that TTFT for a single GPU with concurrency 16 is reduced from 37.97 seconds to 9.30 seconds, achieving a 4.1x speedup. This result comes from joint testing of the Mingxin FX100 all-flash NVMe-oF array and the AMD MI308X platform, with the core approach being the transformation of the traditional serial I/O path into a parallel data stream, thereby eliminating storage access blocking of the inference pipeline.

Cold Start TTFT Bottleneck: Why 38 Seconds Is Not Uncommon

In large model inference services, TTFT (Time to First Token) refers to the time from when a user sends a request to when the model generates the first token. For long-context scenarios (e.g., code generation, document analysis), the model needs to load the complete KV Cache (key-value cache) into GPU memory. When the cache misses, the system must read this data from external storage.

Under a typical configuration (single GPU, concurrency 16, cold read from disk), using the Qwen2.5-32B model, the baseline TTFT is 37.97 seconds. This is not due to insufficient hardware performance but rather a design issue in the I/O path. Traditional LMCache uses a serial read method: each request's KV Cache blocks are read sequentially from the storage device, with the CPU waiting for each block to be fully read before initiating the next request. This mode is efficient when storage bandwidth is sufficient, but once high concurrency or large data blocks are encountered, I/O wait times accumulate sharply.

Measured data shows that under the baseline configuration, storage bandwidth is only 0.98 GB/s, far below the nominal performance of NVMe SSDs (typically 5-7 GB/s). This means the storage device is idle most of the time while the CPU waits for data. This is the root cause of the high TTFT.

LMCache Parallel Read Patch: An I/O Revolution from Serial to Parallel

The core idea of the LMCache parallel read patch is to transform serial I/O requests into an asynchronous parallel stream. The specific implementation includes three key steps:

1. Request Splitting and Prefetching: Split a single KV Cache block into multiple sub-blocks (chunks) and initiate independent asynchronous I/O requests for each sub-block. The system leverages the parallel capability of the storage device to handle reads of multiple sub-blocks simultaneously.
2. Pipeline Scheduling: The CPU returns immediately after initiating an I/O request without waiting for data readiness. When data arrives, the inference engine is notified via a callback mechanism. This eliminates CPU idle waiting time.
3. Bandwidth Aggregation: Through parallel requests, fully utilize the queue depth and bandwidth of the storage device. Measured results show bandwidth increases from 0.98 GB/s to 5.23 GB/s, a 5.3x improvement.

This patch modifies only the I/O scheduling layer of LMCache and does not involve changes to the model structure or inference algorithm. Therefore, it can be seamlessly integrated into existing vLLM deployments without requiring model retraining or hyperparameter adjustments.

Measured Verification: How the 4.1x TTFT Improvement Is Achieved

In joint testing of the Mingxin FX100 all-flash NVMe-oF array and the AMD MI308X platform, we verified the actual effect of the parallel read patch. The test environment is: 8×AMD Instinct MI308X (192 GB HBM each), 2×AMD EPYC 9654 (384 threads), ROCm 7.2, vLLM 0.20.1+rocm721, LMCache (compiled from upstream mainline source code dated 2026-06-29). The tested storage is the Mingxin FX100 array (4-disk RAID0, 14 TB, XFS, RoCEv2, single-port 100 GbE), with the baseline being a local NVMe single disk (PCIe Gen4, 2 TB). The model is Qwen2.5-32B.

Key results are as follows:

• TTFT Improvement: Reduced from 37.97 seconds to 9.30 seconds, a 4.1x speedup.
• Bandwidth Increase: From 0.98 GB/s to 5.23 GB/s, a 5.3x improvement.
• Concurrency Impact: At concurrency levels of 8, 16, and 32, the TTFT improvement remains stable between 3.5x and 4.1x, indicating that the parallel read patch is particularly effective for high-concurrency scenarios.

Notably, this acceleration effect is further amplified in tests without external memory recomputation. For the 480B model (Qwen3-Coder-480B-FP8), TTFT is reduced from 149.5 seconds to 11.85 seconds, achieving an 8.6-20x speedup. This validates the universality of the parallel read patch in large model scenarios.

Synergy with Storage Acceleration: Why High-Performance NVMe-oF Is Needed

The parallel read patch solves the I/O scheduling problem, but final performance is still limited by the raw bandwidth and latency of the storage device. The Mingxin FX100 array provided critical support in this test: its all-flash NVMe-oF architecture supports 100 GbE RoCEv2 networking, and the 4-disk RAID0 configuration can deliver approximately 5-7 GB/s of sustained read bandwidth. This exactly matches the bandwidth requirements of the parallel read patch.

In contrast, the baseline local NVMe single disk (PCIe Gen4, 2 TB) has a theoretical bandwidth of about 4-5 GB/s, but under high concurrency, actual available bandwidth can decrease due to queue depth limitations. The FX100 array maintains stable bandwidth output in high-concurrency scenarios through multi-disk parallelism and network optimization.

This synergy is also validated in training checkpoint saving scenarios: in 8-GPU 32B LoRA training, full model snapshot save time is reduced from 178 seconds to 94 seconds, with sustained write bandwidth increasing from 3.26 GB/s to 6.40 GB/s (+96%). This demonstrates that storage acceleration benefits not only inference but also training workflows.

Deployment Recommendations: How to Reproduce the 4.1x Speedup

For teams looking to reproduce this effect, the following steps are recommended:

1. Hardware Preparation: Use an all-flash storage array supporting NVMe-oF (e.g., Mingxin FX100), and ensure the network uses RoCEv2 or a similar low-latency protocol. Single-port 100 GbE is the recommended starting point.
2. Software Configuration: Compile based on LMCache upstream mainline (versions after 2026-06-29) and apply the parallel read patch. Note that the vLLM version must be compatible with LMCache (vLLM 0.20.1+ is recommended).
3. Model Selection: The parallel read patch is highly effective for long-context models (e.g., 32B, 70B, 480B). Small models (e.g., 7B) benefit relatively less because their KV Cache size is small.
4. Performance Tuning: Adjust concurrency (recommended range 8-32) and request splitting granularity (chunk size). Measured results show the most significant TTFT improvement at concurrency 16.

Mingxin Technology offers approximately 10 weeks of gated joint testing services, including G1 arrival acceptance, G2 single-node baseline, G3 main gate (TTFT reduction ≥25%, throughput +29-40% measured in-band), and G4 72-hour stability verification. If standards are not met, the process stops. The measurement model is reproducible in Python after NDA. Computing centers and model service teams are welcome to contact us for joint testing collaboration.

Key Q&A from This Article

Q: How does the LMCache parallel read patch achieve a 4.1x TTFT speedup? A: By transforming serial I/O requests into an asynchronous parallel stream, eliminating CPU wait time, and increasing storage bandwidth from 0.98 GB/s to 5.23 GB/s, thereby reducing TTFT for a single GPU with concurrency 16 from 37.97 seconds to 9.30 seconds.

Q: What are the storage device requirements for this technology? A: A high-performance all-flash storage array supporting NVMe-oF (e.g., Mingxin FX100) is required, with single-port 100 GbE networking, ensuring that queue depth and bandwidth do not become bottlenecks under high concurrency. Baseline local NVMe single disks experience bandwidth degradation under high concurrency.

Q: Is the acceleration effect applicable to all models? A: It is highly effective for long-context models (32B and above), with TTFT improvement ranging from 3.5x to 4.1x. Small models (e.g., 7B) benefit relatively less because their KV Cache size is small, making the I/O bottleneck less prominent.