How to Make a POC Meaningful: A Methodology for One-Week Real-Traffic Testing

Introduction: When POC Becomes a "Showmatch"

In AI compute procurement, Proof of Concept (POC) should be the most critical step for technical decision-making. However, in reality, many POCs degenerate into "showmatches": vendors run standard benchmarks to produce impressive numbers, users test throughput with simple scripts, and both parties sign off after a week—only to find that, upon deployment, first-token latency spikes and concurrency causes frequent stuttering.

Where does the problem lie? The POC fails to simulate real production workloads. AI applications have evolved from single-model inference to complex scenarios like multi-agent collaboration, video generation, and private deployment. The traffic characteristics of these scenarios—long contexts, cold starts, concurrency jitter—cannot be reproduced by any offline benchmark. This article proposes a methodology: use real traffic, run for one week, and perform end-to-end validation, so that the POC truly answers the question, "Will it work when we buy it?"

Why "Benchmark Scores" Cannot Replace a POC

A common pitfall in compute procurement is relying on standard benchmarks. For example, using MLPerf Inference or vLLM's built-in performance tests to measure throughput and latency under a few batch sizes, and assuming that covers the requirements. But in real production, AI applications face dynamic traffic:

• AI Applications and Agents: Multi-turn dialogues, tool calls, and long-context memory cause fluctuations in KV Cache hit rates. In cold-recovery scenarios, first-token latency can spike from milliseconds to seconds.
• Video Generation: For example, ComfyUI + LTX-Video 2.3, where model loading and frame sequence processing impose extremely high burst demands on storage IO. Under NFS mounting, loading times can stretch to several minutes.
• Private Deployment: Limited enterprise intranet bandwidth, multi-user concurrency, and frequent model updates—these factors are completely ignored in offline tests.

Mingxin's FX100 demonstrated this difference in measured, report R2: For a 480B model in a "no external memory recomputation" scenario, the baseline TTFT p50 was as high as 149.5s, while FX100 reduced it to 11.85s, achieving an acceleration ratio of 8.6–20×. However, this data can only be reproduced under real workloads with "cold recovery + long context + concurrency level 16"—single-batch benchmarks cannot capture it at all.

Methodology: A Three-Step Framework for One-Week Real-Traffic Testing

Step 1: Define the Baseline of "Real Traffic"

The first step of a POC is not to run benchmarks, but to collect production traffic characteristics. If the user does not yet have a production environment, construct simulated traffic based on business scenarios. Key parameters include:

• Concurrency Model: Peak concurrency, average concurrency, and concurrency fluctuation patterns (e.g., step increases, burst spikes).
• Context Length Distribution: Proportion of short dialogues (<4K), long documents (32K+), and ultra-long contexts (128K+). In the R2 test, for a 480B model under TP8 and concurrency level 16, TTFT improvement reached 26–32%, and this improvement is strongly correlated with context length.
• Cold-to-Hot Ratio: Ratio of new requests (cold start) to requests reusing KV Cache (hot requests). In measured, report R1, with the LMCache parallel read patch, cold-read TTFT dropped from 37.97s to 9.30s (4.1×), but this benefit is only meaningful when cold requests account for more than 30% of traffic.
• Model Loading Frequency: In private deployments, models may be loaded frequently due to version updates or resource scheduling. In measured, report R9, DeepSeek-70B loading time on the Huawei 910B platform dropped from 1399s to 150s (9.3×), which is critical for multi-model switching scenarios.

Recommendation: Before the POC, record at least one week of production traffic logs, or construct a "traffic script" based on business documents, including at least three typical scenarios (e.g., peak concurrency, cold-start storm, model hot-update).

Step 2: One-Week End-to-End Testing

A POC cannot be limited to a single day, because AI system performance exhibits time-dimensional fluctuations:

• Storage Cache Warm-Up: The cache hit rate of an NVMe-oF array gradually improves under sustained load. In the R1 test for FX100, checkpoint save time for 8-card 32B LoRA training dropped from 178s to 94s (1.9×). This improvement was not obvious during the first save and required multiple consecutive runs to stabilize.
• Network Jitter and Retransmission: RoCEv2 networks may experience micro-burst packet loss during prolonged operation, causing throughput degradation. A one-week test can expose such issues.
• GPU Memory Fragmentation in Model Inference: vLLM's memory management may produce fragmentation after 72 hours of continuous operation, affecting the concurrency ceiling. In the R4 test, FX100 ran stably for 72 hours in a 480B multi-instance configuration, with throughput fluctuation controlled within ±5%.

Execution points: Divide the week into "warm-up period (1 day) → steady-state period (3 days) → stress period (2 days) → recovery period (1 day)." During the stress period, inject peak traffic (e.g., double the concurrency) and observe the degradation curves of TTFT, throughput, and bandwidth. In the R2 test, when FX100 went from concurrency level 8 to level 16, throughput improvement increased from 29% to 40%, indicating that the system performed better under higher pressure—such nonlinear behavior can only be discovered through long-cycle testing.

Step 3: Make Decisions with Reproducible Quantitative Metrics

The final output of a POC should not be "feels good" or "it works," but rather a set of reproducible quantitative metrics. It is recommended to include at least:

• TTFT (Time to First Token): p50, p95, p99, distinguishing between cold starts and hot requests. In the R2 test for FX100, TTFT p50 dropped from 10.17–35.73s to 7.53–26.35s, a reduction of 26–32%, with even more significant improvement at p95.
• Throughput (tokens/s): At the full-system level (TP4×2), FX100 achieved a 35–36% improvement [measured, report R3].
• Loading Time: End-to-end time for model loading and checkpoint saving. In the R9 test, DeepSeek-32B loading dropped from 691s to 112s (6.2×).
• Stability: Over 72 hours of continuous operation, the coefficient of variation (CV) for performance metrics should be less than 10%.

These metrics must be accompanied by reproducible test scripts (e.g., Python code) so that the user can verify them in their own environment. In Mingxin's collaboration model, after signing an NDA, we provide Python-reproducible calculation models precisely to eliminate the "black box" concern.

Conclusion: POC Is Not the End, but the Beginning

Testing with real traffic for one week essentially transforms the POC from a "vendor demonstration" into a "user experiment." It requires more time investment from both parties, but in return, it provides certainty after deployment—especially for scenarios sensitive to latency and stability, such as AI applications, agents, video generation, and private deployment. A failed POC can mean millions in wasted investment.

In the R2–R9 tests for Mingxin's FX100, from KV Cache acceleration to model loading and training checkpoint saving, we provided quantifiable measured data (e.g., TTFT↓26–32%, throughput +29–40%). We welcome users to bring their real traffic scripts and conduct a one-week gated joint test (approximately 10-week process, including G1 arrival acceptance to G4 72-hour stability). If targets are not met, stop the loss. Decisions on compute procurement should be based on real data, not marketing rhetoric.