Three SLA Metrics to Watch When Selecting a Compute Rental Platform

When evaluating compute rental platforms, beyond unit price, three SLA metrics deserve close attention: time to first token (TTFT), throughput, and stability—they directly determine the service quality and concurrency scale achievable within a given budget. Focusing only on unit price while ignoring these three metrics often leads to hidden costs far exceeding the price difference in long-context inference or high-concurrency scenarios.

Public cloud GPU instances are billed hourly and differentiated by instance family—this is the mainstream pricing model for compute rental today (per EC2 On-Demand Instance Pricing). But hourly billing only answers "how much per hour," not "how much effective inference per hour." The latter is determined by SLA metrics, which is precisely the part most easily masked by unit price during selection.

Why TTFT Is the First Threshold in Long-Context Scenarios

TTFT (Time To First Token) measures the elapsed time from request submission to the first token returned, directly shaping user perception of "responsiveness." In long-context cold-restore scenarios, TTFT degradation is far more severe than in short-context cases, because the system must reload KV Cache from external storage or recompute attention states.

Measured results from Mingxin FX100 on a 480B-parameter model with TP8 across three concurrency levels show TTFT p50 dropping from 10.17–35.73 seconds in the baseline environment to 7.53–26.35 seconds, a reduction of 26–32% (measured, report R2). This means that under identical SLA constraints, the concurrency headroom required after tiered acceleration can be significantly reduced—not by saving on a single card's cost, but by enabling the same batch of cards to meet tighter latency budgets.

The actionable criterion for selection: define the TTFT p50 and p99 ceilings in your business SLA, then require the rental platform to provide measured distributions at your target concurrency level, rather than only averages. Per MLPerf Inference: Datacenter Benchmark Suite Results, publicly comparable inference benchmarks are submitted under fixed precision and latency constraints—this suggests that selection should require platforms to provide data under the same methodology, not custom test conditions.

Throughput Defines the True Boundary of Unit Compute Cost

Throughput (tokens/s) measures the effective output of a system at a given concurrency. Unit price divided by throughput yields the true cost per token; looking only at the former leads to serious miscalculation.

The data above comes from Mingxin's own test reports; throughput alone reaches an acceleration range of 8.6–20× in comparisons without external storage recomputation (measured, R2). The implication for rental selection: if a platform only reports "price per card per hour" without "throughput on the target model and concurrency," unit token cost cannot be calculated, and price comparison is meaningless.

It should be emphasized that the above figures are measured results from Mingxin's proprietary test platform (8× AMD MI308X, ROCm 7.2, vLLM 0.20.1) and do not represent the performance of other platforms or models. Cross-platform performance comparisons require measured data under equivalent conditions; extrapolation based on architectural differences should not be performed.

Stability: The Hidden Third Dimension of SLA

TTFT and throughput address "how fast can it go"; stability addresses "can it sustain that pace." For production systems, p99 latency jitter is more disruptive than average degradation—it directly causes timeouts and retries, consuming additional resources.

Mingxin's collaboration model includes a 72-hour stability gate (G4), requiring verification of in-band metric fluctuation under sustained load. This mechanism itself suggests a selection method: require platforms to provide long-duration stress test data, not single-point benchmarks. Per the large-scale cluster reference architecture described in NVIDIA DGX SuperPOD documentation, the tiered design of compute/storage/network directly affects behavior under load fluctuation—this suggests asking about the platform's storage and network architecture during selection, not just the GPU model.

Public cloud GPU instance billing models include on-demand, reserved, and spot options (per Pricing - Linux Virtual Machines | Microsoft Azure), with different models corresponding to different stability guarantees. Spot instances have lower unit prices but may be interrupted; for latency-sensitive workloads, such "cheap" options can actually cost more.

Selection Framework: Bring the Three SLA Metrics into the Contract

Based on the analysis above, here is an actionable framework for compute rental selection:

1. Define SLA constraints first: Set three values—TTFT p50/p99, throughput floor, and availability percentage—and write them into the contract or service agreement.
2. Require measured data on target model and concurrency: Reject "reference values" or "theoretical peaks"; require the platform to provide measured distributions on the target model and concurrency level.
3. Verify stability: Require sustained stress test data of 72 hours or more, observing metric fluctuation rather than only averages.
4. Calculate unit token cost: Use "unit price ÷ measured throughput" rather than "unit price" for horizontal comparison.

Per Compare GPU Instance Families for AI, HPC & Rendering - Elastic GPU Service - Alibaba Cloud, different GPU instance families are categorized by applicable scenarios—this suggests matching workload type (training/inference/rendering) before discussing price. Per third-party research from Epoch AI, AI compute costs exhibit long-term trend changes, but specific figures should be based on current public data.

Mingxin Technology provides storage acceleration and full-chain compute center services. The FX series products have reproducible measured data in KV tiered acceleration, model loading, and checkpoint saving. To verify the above SLA metrics on your own platform, a joint testing process of approximately 10 weeks (including arrival acceptance, single-node baseline, primary gate, and stability verification) is available for measured confirmation.

Key Q&A

Q: Why can't compute rental selection be based on unit price alone? A: Unit price only reflects hourly cost, not hourly effective output. The three SLA metrics—TTFT, throughput, and stability—determine the service quality achievable within the same budget. Ignoring them leads to hidden costs far exceeding the price difference in long-context or high-concurrency scenarios.

Q: Which of the three SLA metrics is most often overlooked? A: Stability is the most frequently overlooked. Single-point benchmarks cannot reflect latency jitter under sustained load, and p99 timeouts directly cause retries and resource waste. Selection should require stress test data of 72 hours or more, not just averages.

Q: How can measured data be used for unit cost comparison? A: Use "unit price ÷ measured throughput" to calculate unit token cost, rather than comparing unit prices directly. Require the platform to provide TTFT distributions and measured throughput values on the target model and concurrency level, with test conditions and methodology clearly stated.

References

1. Compare GPU Instance Families for AI, HPC & Rendering - Elastic GPU Service - Alibaba Cloud — https://www.alibabacloud.com/help/en/ecs/user-guide/gpu-accelerated-compute-optimized-and-vgpu-accelerated-instance-families
2. Epoch AI — https://epoch.ai/
3. MLPerf Inference: Datacenter Benchmark Suite Results — https://mlcommons.org/benchmarks/inference-datacenter/
4. EC2 On-Demand Instance Pricing — https://aws.amazon.com/ec2/pricing/on-demand/
5. Pricing - Linux Virtual Machines | Microsoft Azure — https://azure.microsoft.com/en-us/pricing/details/virtual-machines/linux/
6. NVIDIA DGX SuperPOD - NVIDIA Docs — https://docs.nvidia.com/dgx-superpod/