How to Calculate $/M token for AI Compute Centers: A Reproducible Cost Model Methodology

How to Calculate $/M token for AI Compute Centers: A Reproducible Cost Model Methodology

In the planning and operation of AI compute centers, $/M token (cost per million tokens for inference) is a core metric for economic evaluation, but its calculation is not simply hardware price divided by throughput. A reproducible cost model requires decomposing Total Cost of Ownership (TCO) into Capital Expenditure (CapEx) and Operational Expenditure (OpEx), then dynamically adjusting based on measured token output and workload characteristics (e.g., KV cache hit rate). This article provides a methodological framework to help technical decision-makers build their own $/M token model from scratch and explains how to calibrate assumptions using measured data (e.g., throughput improvements from KV tiered acceleration).

Decomposing TCO for AI Compute Centers: From Hardware to Operations

The denominator of $/M token is token output, and the numerator is the TCO of the compute center over that time period. TCO typically includes the following four layers:

1. Hardware CapEx: GPUs/accelerators (e.g., AMD MI308X, NVIDIA H100), servers, networking (e.g., RoCEv2 switches), storage (e.g., Mingxin FX100 all-flash array). According to Mingxin's price list, the fully configured reference price for FX100 is ¥371,200 (approximately ¥2,014/TB), and for FX200 it is ¥331,200 (approximately ¥1,797/TB). The hardware depreciation period is typically 3–5 years (tax depreciation or technical lifespan).
2. Facility CapEx: Data center civil construction, power (UPS, cooling), racks. IDC data shows that facility costs for typical hyperscale data centers account for 30–50% of hardware costs.
3. OpEx: Power (GPU full-load power consumption + cooling + networking), operations personnel, software licenses (e.g., ROCm or CUDA ecosystem), network bandwidth. Power costs typically account for 60–70% of OpEx, estimated at $0.08–0.15/kWh (depending on region).
4. Other Costs: Training data storage, model version management, business interruption risks (e.g., latency jitter caused by cold recovery of KV cache).

Reproducible Formula: $$TCO_{monthly} = \frac{CapEx_{total}}{\text{Depreciation Months}} + OpEx_{monthly}$$ Where $CapEx_{total}$ includes one-time investments in hardware and facilities. For example, an 8-card MI308X node (including FX100 array) has a hardware CapEx of approximately $200,000 (estimated at 2026 market prices), plus 40% for facility costs, resulting in $CapEx_{total} ≈ $280,000. Depreciation over 36 months gives a monthly cost of $7,778. OpEx, based on 8 kW power consumption (GPU + cooling), $0.10/kWh, and 30 days, is approximately $576. Monthly TCO ≈ $8,354.

From Hardware Throughput to Token Output: Workload Characteristics and Measured Adjustments

Dividing TCO by monthly token output yields $/M token, but token output depends on inference workload type and system optimization. Key variables include:

• Concurrency: In production deployments, concurrency of 8–16 is common. Measured, report R2 from Mingxin shows that for a 480B model at 8 concurrency, KV tiered acceleration improves throughput by +29%, and at 16 concurrency, by +40%. This means if baseline throughput is 10 tok/s, FX100 can increase it to 12.9–14 tok/s.
• Context Length: Long contexts (e.g., 128K tokens) have large KV cache usage, and time-to-first-token (TTFT) is a key bottleneck. In measured, report R2, FX100 reduces TTFT p50 from 10.17–35.73s to 7.53–26.35s (a reduction of 26–32%), directly lowering user wait time, but throughput is limited by GPU memory.
• Cache Hit Rate: KV cache reuse (e.g., LMCache) can significantly reduce recomputation. Measured, report R1 shows that LMCache's parallel read patch improves TTFT by 4.1× (37.97s → 9.30s) and bandwidth by 5.3×. If 50% of requests hit the cache in a business scenario, effective throughput can double.

Calculation Example: Assume an 8×MI308X node running a 480B model with baseline throughput (no KV acceleration) of 10 tok/s and concurrency of 16. Monthly token output = 10 tok/s × 3600 s/h × 24 h × 30 d = 25.92M tokens. With FX100's +40% acceleration, this becomes 36.29M tokens. $/M token = $8,354 / 36.29 ≈ $230. If cache hit rate is 50%, throughput doubles, reducing $/M token to $115.

Reproducibility and Boundary Conditions of the Cost Model

Any cost model must clearly define assumption boundaries; otherwise, results are not comparable. Here are three key steps to building a reproducible model:

1. Hardware Selection and Depreciation Assumptions: Specify GPU model (e.g., MI308X vs H100), network bandwidth (e.g., 100 GbE vs 400 GbE), and storage solution (e.g., bandwidth differences between local NVMe and all-flash arrays). Mingxin's FX series offers different PCIe generations and IOPS specifications (FX100: 16M IOPS, FX200: 32M IOPS), and selection should be based on actual workload.
2. Parameterization of Workload Characteristics: Define concurrency, context length, cache hit rate, and model parameter count. For example, the KV cache for a 480B MoE model occupies approximately 450 GB (weights), requiring 8×192 GB of GPU memory for full caching. If GPU memory is insufficient, external storage (e.g., FX100's NVMe-oF array) is needed, requiring measured TTFT and throughput data (e.g., the 8.6–20× acceleration factor without external memory recomputation from report R2).
3. Measured Calibration: Throughput and latency data in the model should come from formal test reports, not vendor claims. Mingxin provides an approximately 10-week gated joint testing process (G1–G4), including in-band verification of TTFT reduction ≥25% and throughput improvement +29–40%, ensuring assumptions are reproducible.

Common Pitfalls:

• Ignoring power costs: GPU full-load power (e.g., MI308X at approximately 750W) plus cooling coefficient (PUE 1.2–1.5) can make power costs account for 70% of OpEx.
• Assuming linear scaling: Throughput does not scale linearly with concurrency; use measured data (e.g., differences between 8 and 16 concurrency in report R2).
• Neglecting cache hit rate: In long-context scenarios, KV cache reuse rates can reach 60–80%. If the model assumes 0%, it overestimates $/M token by several times.

Conclusion

Calculating $/M token is not a static arithmetic problem but a dynamic model that combines TCO decomposition, workload characteristics, and measured data. Mingxin Technology provides quantifiable performance improvements in storage acceleration (e.g., FX100's KV tiered acceleration: throughput +29–40%, TTFT reduction 26–32%), which can be directly embedded into cost models to adjust baseline assumptions. For technical decision-makers in AI compute centers, it is recommended to use the framework in this article, combined with measured data from their own workloads (e.g., obtained through Mingxin's joint testing process), to build a reproducible $/M token model for more accurate economic evaluation of compute infrastructure.

Key Q&A

Q: What are the core components of TCO in the calculation of $/M token?
A: TCO includes hardware CapEx (e.g., GPUs, storage arrays), facility CapEx (data center civil construction and power), and OpEx (power, operations, networking). The depreciation period is typically 3–5 years, and power costs account for approximately 60–70% of OpEx.

Q: How can measured data be used to adjust throughput assumptions in the cost model?
A: Taking Mingxin FX100 as an example, measured, report R2 shows that KV tiered acceleration improves throughput by +29–40% (at concurrency levels of 8–16) and reduces TTFT by 26–32% for a 480B model. These data can directly replace baseline throughput values in the model, avoiding misjudgment from linear scaling.

Q: What is the impact of cache hit rate on $/M token?
A: If cache hit rate increases from 0% to 50%, effective throughput can double, theoretically halving $/M token. In long-context scenarios, solutions like LMCache can improve TTFT by 4.1× (as shown in measured, report R1), significantly reducing latency costs.