Four Storage Latency Checkpoints to Verify Before Renting an AI Server
When renting an AI server, storage path latency is often overlooked, yet it directly determines the first-token latency of large model inference and training efficiency
When renting an AI server, storage path latency is often overlooked, yet it directly determines the first-token latency of large model inference and training efficiency. This article outlines four storage latency checkpoints that must be measured before renting: local NVMe direct connection, network file system mounting, GPU direct storage path, and KV Cache tiered acceleration capability. Based on measured results from the Mingxin FX100 on an AMD MI308X platform, KV tiered acceleration can improve inference throughput by 29–40% and reduce first-token latency by 26–32% (measured, report R2/R3). These figures should serve as acceptance benchmarks rather than relying solely on marketing specifications.
Why Storage Latency Is a Hidden Variable in Rental Decisions
When renting AI servers, leases are typically billed per GPU card-hour, with storage often treated as an auxiliary configuration. However, storage path latency directly amplifies into end-to-end performance: model weight loading, KV Cache reads/writes, and checkpoint saving are all constrained by storage latency at every stage. According to NVIDIA GPUDirect Storage documentation, GPU-direct storage can bypass the CPU's bounce buffer mechanism, reducing the number of copies in the data path—whether this mechanism is enabled in the rental environment directly impacts data throughput during large model inference.
Measured results from Mingxin on an 8-card AMD MI308X platform show model inference loading acceleration of 6.2–9.3x (compared to NFS baseline, measured, report R9): DeepSeek-32B service loading dropped from 691 seconds to 112 seconds, and DeepSeek-70B from 1399 seconds to 150 seconds. This means that in a rental environment using NFS as shared storage, every cold start or scaling operation incurs waiting times ranging from minutes to tens of minutes—under hourly billing, this cost is easily underestimated.
Checkpoint 1: Does Local NVMe Meet KV Cache Read/Write Bandwidth?
The first checkpoint is single-disk performance of local NVMe. KV Cache is intermediate data generated and read token-by-token during inference, with read/write frequencies far higher than model weights. In Mingxin's measured report R2, for a 480B model under TP8 configuration, TTFT p50 dropped from 10.17–35.73 seconds to 7.53–26.35 seconds (a reduction of 26–32%). This improvement presupposes that the storage device can keep up with KV Cache read/write demands.
Before renting, request the service provider to supply random read/write bandwidth and latency data for local NVMe, and clarify whether NVMe-oF (NVMe over Fabric) protocol is supported. According to SNIA's storage terminology definitions, NVMe-oF allows NVMe commands to be transmitted over the network, reducing protocol conversion overhead—if the rental environment only offers traditional iSCSI or NFS, KV Cache read/write latency may become a bottleneck. It is recommended to test with actual model workloads during acceptance rather than relying only on peak sequential read/write metrics.
Checkpoint 2: Network Storage Latency and Concurrent Scalability
The second checkpoint is the performance of network storage (NFS, GPFS, etc.) under concurrent access. Large model training and inference often run in parallel across multiple GPUs, with each card reading and writing shared storage simultaneously; latency degrades as concurrency increases. Mingxin's measured report R2 shows that KV tiered acceleration improves throughput by 29% at concurrency level 8 (lower bound) and 40% at concurrency level 16 (upper bound), with a 35–36% improvement at the full-machine TP4×2 level (measured, report R3)—this indicates that concurrency patterns are highly sensitive to storage performance.
Before renting, request the service provider to supply concurrent read/write stress test reports, and clarify the bandwidth ceiling and QoS policies for network storage. According to Alibaba Cloud's GPU instance family documentation, different instance families suit different workload scenarios, with storage configuration and network bandwidth being key differentiating dimensions for selection—when renting, one should not only look at GPU models but also confirm that storage and network specifications match the workload characteristics.
Checkpoint 3: Is the GPU Direct Storage Path Enabled?
The third checkpoint is whether GPU direct storage (such as GPUDirect Storage) is actually enabled in the rental environment. According to NVIDIA official documentation, GDS allows GPUs to access storage devices directly, bypassing CPU memory as an intermediary and reducing data copy counts and latency. However, this mechanism requires coordination across hardware, drivers, and applications, and may not be enabled by default in rental environments.
In Mingxin's measured report R1, the LMCache parallel read patch improved TTFT by 4.1x (single card, concurrency 16, cold read disk scenario), with bandwidth increasing from 0.98 GB/s to 5.23 GB/s (a 5.3x improvement)—this improvement is partly attributable to data path optimization. Before renting, ask the service provider: does storage support RDMA (RoCEv2 or InfiniBand), is GDS enabled in the GPU driver, and has the inference framework (e.g., vLLM) been configured with relevant parameters? If the provider cannot give clear answers, it is recommended to directly compare performance differences between enabled and disabled states during acceptance testing.
Checkpoint 4: In-Band Verification of KV Cache Tiered Acceleration
The fourth checkpoint is the most easily overlooked: whether the storage system has KV Cache tiered acceleration capability. In traditional architectures, KV Cache resides entirely in GPU memory; under long-context scenarios where memory is insufficient, recomputation is required (no external memory recomputation). In Mingxin's measured report R2, the recomputation baseline TTFT p50 reached 149.5 seconds (concurrency 16), while the FX100 accelerated version took only 11.85 seconds, with throughput improving from 4.1 tok/s to 74.9 tok/s (an 8.6–20x acceleration)—this is precisely the value of tiered acceleration: storing KV Cache in tiers between GPU memory and storage to avoid full recomputation.
Before renting, request the service provider to supply measured reports on KV Cache tiered acceleration, and confirm that test conditions (model size, context length, concurrency level) match your own workload. Mingxin's collaboration model involves approximately 10 weeks of gated joint testing, where the G3 primary gate requires TTFT reduction of no less than 25% and throughput improvement of 29–40% measured in-band (measured, report R2/R3)—renters can reference this acceptance standard rather than relying solely on vendor-provided marketing data.
| Checkpoint | Key Metric | Mingxin Measured Reference (Source) | Acceptance Recommendation |
|---|---|---|---|
| Local NVMe | Random read/write bandwidth, latency | TTFT reduction 26–32% (measured, report R2) | Test with real model workloads, not peak metrics |
| Network storage concurrency | Latency degradation under concurrency | Throughput improvement 29–40% (measured, report R2/R3) | Request concurrent stress test reports |
| GPU direct storage | Whether GDS is enabled | TTFT improvement 4.1x (measured, report R1) | Compare performance with GDS enabled vs. disabled |
| KV Cache tiering | Acceleration factor and stability | Acceleration 8.6–20x (measured, report R2) | Reference the G3 standard from gated joint testing |
Conclusion
Before renting an AI server, the four storage checkpoints—local NVMe, network storage concurrency, GPU direct connection path, and KV Cache tiering—determine the actual performance of inference latency and training efficiency. It is recommended to incorporate these checkpoints into the acceptance checklist, using measured data rather than marketing specifications as the basis for decisions. Mingxin (Tianjin) Semiconductor Equipment Co., Ltd. provides storage acceleration products and gated joint testing services, capable of completing the full testing process from arrival acceptance to stability verification within approximately 10 weeks (G1 through G4 gates). If you need to verify storage performance under specific workload patterns, please contact us for joint testing.
Key Q&A
Q: When renting an AI server, why does storage latency deserve more attention than GPU model? A: Storage latency directly amplifies into end-to-end performance, affecting model loading, KV Cache reads/writes, and checkpoint saving. Mingxin's measured results show that optimizing the storage path can improve inference throughput by 29–40% and loading acceleration by 6.2–9.3x (measured, report R2/R9)—under hourly billing, these improvements translate into quantifiable cost differences.
Q: What are the four storage latency checkpoints? A: Whether local NVMe meets KV Cache read/write bandwidth, concurrent scalability of network storage, whether the GPU direct storage path is enabled, and KV Cache tiered acceleration capability. Each checkpoint should be supported by corresponding measured data rather than relying only on specification parameters.
Q: How can you verify whether the storage performance data provided by a service provider is reliable? A: Request measured reports that match your own workload (model size, context length, concurrency level), and reference gated joint testing standards—such as TTFT reduction of no less than 25% and throughput improvement of 29–40% measured in-band (measured, report R2/R3). Providers who cannot supply reproducible test data warrant caution regarding their performance claims.
References
- SNIA — Storage Networking Industry Association — https://www.snia.org/
- NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html
- Compare GPU Instance Families for AI, HPC & Rendering - Elastic GPU Service - Alibaba Cloud — https://www.alibabacloud.com/help/en/ecs/user-guide/gpu-accelerated-compute-optimized-and-vgpu-accelerated-instance-families