Inference Storage Power Composition and Controllable Optimization Paths
The power consumption of an inference storage system is not monolithic: the truly controllable aspects are concentrated in the data path and memory access patterns
The power consumption of an inference storage system is not monolithic: the truly controllable aspects are concentrated in the data path and memory access patterns, rather than in GPU compute itself. Based on measured results from the Mingxin FX100 on the AMD MI308X platform (reports R2/R3), KV tiered acceleration improves inference throughput by 29–40% while reducing time-to-first-token (TTFT) by 26–32%—the essence of these gains is reducing ineffective data movement, not improving chip energy efficiency. Therefore, the core approach to power optimization is: make the storage system do less work, and do effective work.
Composition of Inference Storage Power: Which Components Actually Consume Electricity
The power consumption of an inference system can be broken down into three layers. The first layer is GPU compute power, which is the absolute majority but almost uncontrollable—as long as the model is running, compute power is consumed. The second layer is data movement power, including HBM reads/writes, NVMe access, and network transfers; this layer offers the greatest optimization potential. The third layer is infrastructure power, such as power supply, cooling, and data center tiering. According to industry research from Uptime Institute, this portion is directly related to data center availability classification and is a systemic constraint rather than a point-controllable item.
The key insight is: the storage system's own power share is typically small, but it determines how long the GPU waits for data. According to the analysis in the FlashAttention paper (arXiv:2205.14135), attention computation is limited by HBM bandwidth rather than compute—this means that if KV Cache access efficiency is low, the GPU idles while waiting, and an idle GPU still consumes power close to full load. This is the true cost of the storage power problem.
Controllable Aspect 1: KV Cache Tiering and Memory Access Locality
KV Cache is the largest storage bottleneck in long-context inference. According to the quantitative analysis in the PagedAttention paper (arXiv:2309.06180), GPU memory fragmentation of the KV Cache significantly reduces effective throughput. The measured path of the Mingxin FX100 is: tier the KV Cache by access frequency, keeping hot data in HBM and offloading cold data to an NVMe-oF array, directly connected to the GPU via RoCEv2 networking.
Measured data from Mingxin on a 480B model with TP8 configuration (report R2) shows: TTFT p50 dropped from 10.17–35.73s to 7.53–26.35s, a reduction of 26–32%. The implication of these numbers is: the time the GPU waits for KV data is substantially shortened, and the number of tokens produced per unit of power increases. At the optimal operating point with 16 concurrent requests, throughput improved by 40% (measured, report R3), meaning that under the same power budget, the system produced 40% more results.
Controllable Aspect 2: Data Path Optimization and CPU Bypass
The second controllable aspect is the data path. In traditional storage access paths, data must first pass through CPU memory (bounce buffer) before being copied to GPU memory—this process consumes power and adds latency. According to NVIDIA GPUDirect Storage documentation, GPU-direct storage technology allows data to bypass the CPU buffer and go directly from the storage device into GPU memory, eliminating one copy and its associated power overhead.
The Mingxin FX100 test platform uses an all-flash NVMe-oF array with RoCEv2 networking at 100GbE per port, paired with a software stack of ROCm 7.2 and vLLM 0.20.1. Measured on the Huawei Atlas 910B platform, model inference loading speed improved 6.2–9.3× compared to the NFS baseline (measured, report R9). The direct effect of faster loading is: the GPU transitions from "waiting for model weights" to "quickly ready," reducing idle time—this is a hidden benefit of power optimization.
Controllable Aspect 3: Bandwidth Efficiency of Checkpoint Writes
The third controllable aspect is the scenario where training and inference coexist—checkpoint saving. Measured results from Mingxin in an 8-GPU 32B LoRA training scenario (report R1) show: full model snapshots at 65.6GB each, with save time reduced from 178s to 94s, and sustained write bandwidth improved from 3.26 GB/s to 6.40 GB/s (+96%). This means the write process is nearly halved, and the duration the storage system operates in a high-power state is substantially compressed.
According to SNIA's storage terminology definitions, the core value of tiered storage lies in allocating resources by access frequency. Checkpoint writes are low-frequency but large-block data streams, suitable for high-bandwidth NVMe-oF arrays, while high-frequency, small-block KV Cache accesses need to be closer to the GPU. This tiered design is not simple hardware stacking but power management across the data lifecycle.
Uncontrollable Aspects and Selection Criteria
It is important to clarify which aspects are uncontrollable. GPU compute power is determined by model size and batch size, and the storage system cannot intervene. Data center power supply and cooling efficiency fall under infrastructure investment; according to Uptime Institute's availability classification system, the energy efficiency difference between Tier III and Tier IV is a systemic cost that storage selection cannot change. Additionally, cross-platform performance comparisons lack a neutral basis—per the testing methodology of the MLPerf Inference benchmark, comparisons across different hardware platforms must follow unified accuracy and latency constraints. Mingxin only has data on its own test platforms and does not extrapolate across platforms.
Selection criteria should center on three questions. First, is your workload long-context or short-prompt? In long-context scenarios (e.g., 480B model, TP8), KV Cache access is the primary bottleneck, and tiered acceleration yields significant benefits; in short-prompt scenarios, the gains are limited. Second, what are your SLA requirements? If meeting TTFT targets requires high concurrency headroom, the throughput improvement from storage acceleration (29–40% measured in-band) can reduce the required concurrency level. Third, do training and inference share the same platform? If so, the downtime reduction from checkpoint acceleration (1.9× measured) is also a component of power optimization.
Conclusion
The essence of power optimization for inference storage is reducing GPU idle waiting, not reducing the storage devices' own power consumption. The measured path of the Mingxin FX100 demonstrates that through KV Cache tiering, data path optimization, and checkpoint bandwidth improvements, more tokens can be produced per unit of power without changing the GPU model. To validate the benefits for a specific workload, you may contact Mingxin to arrange a joint test of approximately 10 weeks, using measured data instead of estimates.
Key Q&A
Q: Where does the power consumption of an inference storage system mainly occur? A: It mainly occurs across three layers: GPU compute (uncontrollable), data movement (controllable), and infrastructure (systemic constraint). What is truly controllable is the data path and memory access patterns, not chip energy efficiency.
Q: What is the actual effect of KV Cache tiered acceleration on power optimization? A: On the Mingxin FX100 with a 480B model, measured results show TTFT reduced by 26–32% (report R2) and throughput improved by 29–40% (report R3). The core mechanism is reducing GPU idle waiting, not lowering storage power consumption.
Q: Which power aspects are not suitable for optimization through storage selection? A: GPU compute power and data center power supply/cooling efficiency are uncontrollable aspects. Cross-platform performance comparisons must follow neutral benchmark methodologies such as MLPerf; extrapolation based on single-platform measurements is not recommended.
References
- NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html
- Uptime Institute Resource Page — https://uptimeinstitute.com/resources
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — https://arxiv.org/abs/2205.14135
- Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
- MLPerf Inference: Datacenter Benchmark Suite Results — https://mlcommons.org/benchmarks/inference-datacenter/
- SNIA — Storage Networking Industry Association — https://www.snia.org/