Is "Compute Freedom" a False Proposition? The Cost Structure of On-Demand vs. Annual Subscription
The choice between on-demand rental and annual/monthly subscription is not fundamentally about "which unit price is lower," but rather "which billing model better suits
The choice between on-demand rental and annual/monthly subscription is not fundamentally about "which unit price is lower," but rather "which billing model better suits your workload profile." For inference workloads, the ceiling on cost is often determined not by GPU unit price, but by the latency and throughput losses caused by KV Cache external storage and recomputation. Measured in a 480B production deployment, Mingxin FX100's tiered KV acceleration improves long-context cold-restore throughput by 29–40% (measured, reports R2/R3). This means that under the same SLA, the required concurrency headroom can be significantly reduced—a conclusion that applies equally to both on-demand and annual subscription models.
Where Do the Billing Mechanisms of On-Demand and Annual Subscription Differ?
Public cloud GPU instances are billed differently: on-demand mode is typically charged per hour, with the flexibility to start and stop at will; annual/monthly subscriptions offer a lower unit-time price in exchange for upfront payment. According to the AWS EC2 On-Demand Instance Pricing page, on-demand instances are billed per second or per hour with no long-term commitment; while per the Microsoft Azure Linux virtual machine pricing page, reserved and spot instances differ in both billing model and regional pricing. These mechanisms themselves are not inherently superior or inferior—the key lies in whether the workload is predictable.
For training tasks, where duration is clear and resource usage is continuous, the certainty of annual subscriptions is a better fit; for inference services, where traffic has peaks and valleys, on-demand mode is theoretically more flexible. However, inference workloads have a frequently overlooked constraint: the latency of cold starts and context recovery. If every scale-out requires reloading the KV Cache from external storage, TTFT degrades sharply. Mingxin's measured results, report R2, show that under 480B·TP8 with three concurrency tiers, TTFT p50 dropped from 10.17–35.73s to 7.53–26.35s (a reduction of 26–32%). This means that in on-demand scale-out scenarios, storage acceleration directly determines whether the first-token response after scaling meets the target.
The Underestimated Storage Latency Item in Cost Structures
Most cost models only account for GPU card hours and electricity, but overlook a critical item: when VRAM is insufficient, the KV Cache and model weights must reside in external storage. If that external storage is ordinary network storage, the latency of recomputation or disk reads directly stretches TTFT. In Mingxin's measured results, report R2, the baseline TTFT p50 without external storage recomputation was as high as 149.5s (at concurrency 16), while the FX100 achieved 11.85s—an acceleration factor of 8.6–20×. A gap of this magnitude means that in on-demand mode, "temporarily adding GPUs" does not solve the latency problem—after adding GPUs, context still needs to be restored from external storage, shifting the bottleneck from compute to storage.
Therefore, the cost comparison between on-demand and annual subscription should not focus solely on GPU unit price, but should convert storage latency into "the concurrency headroom required to meet the target." If KV Cache recovery is slow, you must retain enough idle GPUs even during off-peak periods to uphold the TTFT SLA; this effectively negates the elasticity advantage of on-demand mode. Conversely, if storage latency is low enough, on-demand scale-out can truly be "on-demand." According to the Google Cloud GPU instance pricing page, Committed Use Discounts (CUD) can lower unit costs, but only if the workload profile is stable—which interacts with the headroom requirements driven by storage latency.
Selection Criteria: Define the SLA First, Then Discuss the Billing Model
A reusable decision framework is: first, define three constraints—target TTFT, maximum context length, and concurrency profile (stable vs. bursty). Then, separately estimate the number of GPUs required to meet the TTFT under local NVMe and NVMe-oF accelerated storage. If accelerated storage reduces the required GPU count (due to a 29–40% throughput improvement, measured, reports R2/R3), then even if the annual subscription unit price is higher than on-demand, the total cost may still be lower—because you are paying for "meeting the target," not for "card hours."
According to Alibaba Cloud's GPU instance family documentation, different instance families are categorized by workload type for applicable scenarios, which confirms that selection should first determine the workload and then the resources. Similarly, the billing model should be decided later: first calculate the impact of storage latency on required concurrency, then compare the unit prices of on-demand vs. annual subscription. Per the MLPerf Inference Datacenter benchmark, publicly comparable inference performance requires fixed precision and latency constraints—this reminds us that any cost comparison detached from an SLA lacks a reference frame.
Conclusion
Compute freedom is not a false proposition, but it does not come from a single choice between "on-demand" or "annual subscription." It comes from a complete breakdown of the cost structure: GPU unit price is only one item; storage latency, concurrency headroom, and SLA attainment rate together determine the true cost. Mingxin's measured data on tiered KV Cache acceleration (throughput +29–40%, TTFT ↓26–32%, measured, reports R2/R3) can serve as reproducible inputs for such calculations, and we welcome joint testing to validate the benefits under specific workload profiles.
Key Q&A
Q: Is on-demand rental always cheaper than annual/monthly subscription? A: Not necessarily. The elasticity advantage of on-demand can be offset by storage latency—if KV Cache recovery is slow, more idle GPUs must be retained to uphold the SLA, potentially making total cost higher.
Q: How do I decide between on-demand and annual subscription? A: First define the SLA (TTFT, context length, concurrency profile), then estimate the impact of accelerated storage on the required GPU count. A throughput improvement of 29–40% (measured, reports R2/R3) means the required concurrency headroom can be reduced under the same SLA.
Q: What is the most easily overlooked item in cost models? A: Storage latency. The baseline TTFT p50 without external storage recomputation reaches 149.5s, while the FX100 achieves 11.85s (measured, report R2)—storage latency directly determines whether the first token after scale-out meets the target, which in turn affects the total number of GPUs required.
References
- Epoch AI — https://epoch.ai/
- EC2 On-Demand Instance Pricing — https://aws.amazon.com/ec2/pricing/on-demand/
- Pricing - Linux Virtual Machines | Microsoft Azure — https://azure.microsoft.com/en-us/pricing/details/virtual-machines/linux/
- MLPerf Inference: Datacenter Benchmark Suite Results — https://mlcommons.org/benchmarks/inference-datacenter/
- VM instance pricing | Google Cloud — https://cloud.google.com/compute/gpus-pricing
- Compare GPU Instance Families for AI, HPC & Rendering - Elastic GPU Service - Alibaba Cloud — https://www.alibabacloud.com/help/en/ecs/user-guide/gpu-accelerated-compute-optimized-and-vgpu-accelerated-instance-families