How to Define Acceptance Criteria for Computing Power Collaboration: Embedding Measured Metrics into Contracts

In today's era where AI applications are transitioning from experimentation to production—whether deploying agent systems, video generation services, or building private computing infrastructure—the collaboration between buyers and suppliers often stalls at one critical juncture: acceptance criteria.

Metrics like "peak computing power" or "theoretical IOPS" look impressive on marketing materials, but under real-world workloads—such as long-context inference, model loading, or training checkpoint saving—they often diverge significantly from actual user experience. Where does the problem lie? The measurement standards for computing performance are inconsistent, and there is a lack of reproducible testing protocols.

This article uses measured data from Mingxin Technology's FX100 all-flash NVMe-oF array as an example to explore how to embed measured metrics into contracts, shifting computing power collaboration from "looking at parameters" to "looking at results."

Why "Peak Parameters" Cannot Serve as Acceptance Criteria

In computing equipment procurement, suppliers often provide "theoretical maximum IOPS" or "peak bandwidth," but these numbers are nearly impossible to replicate under real workloads. The reasons are:

• Workload Characteristic Differences: KV Cache access in AI inference involves random small I/O (typically 4KB-64KB), while training checkpoints involve sequential large I/O (GB-level). The performance of the same SSD can differ by an order of magnitude between these two scenarios.
• System Bottleneck Shifts: In distributed inference, network latency, CPU scheduling, and GPU memory bandwidth can all become bottlenecks. High single-drive performance may not translate into throughput gains.
• Opaque Measurement Standards: Some vendors use "empty drive tests" or "short burst" data, while in actual deployment, devices operate under steady-state loads.

Therefore, truly effective acceptance criteria should be based on workload models consistent with the production environment, with agreed-upon parameters such as testing tools, concurrency levels, data sizes, and duration.

Four Key Dimensions of Measured Metrics

Taking Mingxin FX100 tests on the AMD MI308X platform as an example (report numbers R2/R3), the acceptance logic can be broken down into four dimensions, each with clear measurement methods and boundary conditions.

1. Throughput Improvement: From "Theoretical IOPS" to "Inference Throughput"

Contracts should specify: under a designated model (e.g., Qwen3-Coder-480B-FP8), specified concurrency (e.g., 16 levels), and specified context length, the throughput improvement (tok/s) percentage.

• Mingxin FX100 measured: 480B model · TP4×2 full-machine configuration, throughput improvement +35–36% (measured, report R3). This value comes from vLLM server-side statistics, not disk-level IOPS conversion.
• Suggested contract wording: "Under [model name] · [concurrency level] · [context length] conditions, compared to the local NVMe baseline, inference throughput improvement shall not be less than [specific percentage]."

2. Latency Improvement: p50/p99 Time to First Token (TTFT)

For latency-sensitive scenarios like agents and video generation, TTFT directly impacts user experience.

• Mingxin FX100 measured: 480B · TP8 three concurrency levels, TTFT p50 reduced from 10.17–35.73s to 7.53–26.35s, a reduction of 26–32% (measured, report R2).
• Contracts must clarify: Is p50 or p99 measured? Does it include network transmission time? Is it conducted under cold start (no prefilling) conditions?

3. Acceleration Factor: For "No External Cache Recalculation" Scenarios

When insufficient GPU memory forces the model to recalculate KV Cache, performance drops sharply. In this scenario, Mingxin FX100 measured acceleration of 8.6–20× (measured, report R2).

• Contract value: This directly validates "GPU memory savings" capability, especially suitable for customers with limited GPU memory in private deployments.
• Note: The baseline for recalculation must be defined (e.g., "using a local NVMe single drive without any KV Cache acceleration").

4. Loading and Saving: Model Deployment and Training Efficiency

• Model loading acceleration (vs NFS): On Huawei Atlas 910B platform, DeepSeek-70B loading time reduced from 1399s to 150s (9.3×, measured, report R9).
• Training checkpoint saving acceleration: In 8-card 32B LoRA scenario, sustained write bandwidth increased from 3.26 to 6.40 GB/s (+96%, measured, report R1).
• Contract suggestion: Specify file system type (e.g., XFS), network protocol (e.g., RoCEv2), and data sharding method.

How to Embed Measured Protocols into Contracts

An executable acceptance contract should include the following elements:

Fixed Test Environment

• Hardware: Specify server model, GPU model, and network configuration. Mingxin tests used 8× AMD MI308X, 2× AMD EPYC 9654, ROCm 7.2, vLLM 0.20.1+rocm721.
• Software: Specify inference framework version, model format (e.g., FP8), operating system, and file system.
• Baseline: Clearly define the comparison object (e.g., "local NVMe single drive" or "NFS"), and ensure the baseline device is consistent with the production environment.

Standardized Test Workload

• Model: Specify name, parameter count, and quantization precision. For example, "Qwen3-Coder-480B-FP8 (MoE, weights ≈ 450GB)."
• Workload: Agree on concurrency level, context length, and request pattern (e.g., "cold recovery" or "continuous inference").
• Measurement duration: At least 30 minutes to exclude short burst effects.

Reproducibility Assurance

• Toolchain: Specify test scripts (e.g., vLLM benchmark) and data collection methods (e.g., Prometheus + Grafana).
• Code openness: In Mingxin's collaboration model, measurement scripts are Python-reproducible after NDA. Contracts can stipulate that "the supplier must provide independently runnable test scripts."
• Third-party witness: If possible, introduce a mutually agreed-upon third-party testing organization.

Acceptance Thresholds and Stop-Loss Mechanisms

• Pass threshold: For example, "TTFT reduction ≥ 25%" or "throughput improvement ≥ 29%" (Mingxin G3 gate standard).
• Non-compliance handling: Agree on retesting, corrective action within a specified period, or contract termination (e.g., Mingxin's "approximately 10-week gate-level joint testing, stop loss if not met").

Conclusion

Computing power procurement is shifting from "buying hardware" to "buying results." For AI applications, agents, video generation, and especially private deployments, "measured metrics" are far more meaningful than "theoretical parameters."

Mingxin Technology, through its FX100 product, demonstrates a reference paradigm for the industry by presenting metrics such as KV acceleration, model loading, and checkpoint saving via reproducible testing protocols. If you are planning a computing center build or optimizing the performance of an existing inference system, feel free to contact Mingxin for joint testing collaboration to verify measured metrics in your production environment.