Deployment Path and Measured Performance of Domestic AI Inference Acceleration Cards in Real-Time Database Queries

The query performance bottleneck in real-time databases is shifting from disk I/O to KV Cache access and model loading within the AI inference pipeline. By optimizing storage access paths, domestic AI inference acceleration cards can reduce first-token latency by 26%–32% without altering query logic, and cut model loading time to 1/6–1/9 of the original level [measured, reports R2/R9]. This conclusion is drawn from multiple controlled tests of the Mingxin FX100 on AMD MI308X and Huawei Ascend 910B platforms. The following sections detail the working mechanism, quantitative results, and selection recommendations.

Why Real-Time Database Queries Are Bottlenecked at the Storage Layer, Not the Compute Layer

A typical query path in real-time databases (e.g., time-series databases, streaming feature stores) involves three steps: SQL parsing, model/index loading, and inference execution. As MoE large models (e.g., Qwen3-Coder-480B-FP8) are embedded into real-time risk control and quantitative trading scenarios, the time spent reading model weights and KV Cache now far exceeds compute time. For a 480B model, weights are approximately 450GB; loading via NFS takes 1,399 seconds for cold start (measured on DeepSeek-70B) [measured, report R9]. Meanwhile, KV Cache requires frequent swapping in long-context queries; without external memory recomputation, TTFT p50 reaches 149.5 seconds at concurrency 16 [measured, report R2].

Traditional solutions rely on local NVMe SSDs or NFS, but suffer from two structural flaws: first, single-disk bandwidth is limited (PCIe Gen4 single-disk is about 2–3GB/s), which cannot match multi-card parallel loading demands; second, the NFS protocol stack incurs high overhead, with high latency on small-file random reads. The Mingxin FX100, as an all-flash NVMe-oF array, connects directly to GPU nodes via RoCEv2 networking, compressing storage access latency to microsecond levels, thereby pulling the query bottleneck back from I/O to compute itself.

Measured Data: Quantitative Gains from KV Cache Acceleration and Model Loading

On an 8×AMD MI308X platform (192GB HBM per card), using vLLM 0.20.1 + ROCm 7.2, three test groups were run on Qwen3-Coder-480B-FP8 [measured, reports R2/R3]:

• First-token latency (TTFT): Under TP8 with three concurrency levels, TTFT p50 dropped from 10.17–35.73 seconds to 7.53–26.35 seconds, a reduction of 26%–32%. At the optimal operating point of concurrency 16, throughput improved by 40% (lower bound: 29% at concurrency 8).
• Without external memory recomputation: The recomputation baseline TTFT p50 was 149.5 seconds (concurrency 16); with FX100, it dropped to 11.85 seconds, a 12.6× speedup; throughput rose from 4.1 tok/s to 74.9 tok/s, an 18.3× improvement [measured, report R2].
• LMCache parallel read patch: In a single-card, concurrency-16 cold-read scenario (Qwen2.5-32B), TTFT dropped from 37.97 seconds to 9.30 seconds (4.1× improvement), and bandwidth rose from 0.98GB/s to 5.23GB/s (5.3×) [measured, report R1].

On the Huawei Ascend 910B platform (measured, report R9), model loading acceleration was even more pronounced: DeepSeek-32B service loading dropped from 691 seconds to 112 seconds (6.2×), and DeepSeek-70B from 1,399 seconds to 150 seconds (9.3×). On the training side, checkpoint saving also dropped from 178 seconds to 94 seconds (1.9×), with sustained write bandwidth improving by 96% [measured, report R1].

These figures show that FX100's acceleration is not dependent on a specific GPU vendor—it holds on both AMD and Ascend platforms, and the speedup factor increases with model size. For real-time databases, this means query cold-start time can be compressed to seconds, providing storage-side assurance for millisecond-level SLAs.

Selection Assessment: When Is It Worth Introducing a Domestic Acceleration Card as a Database Acceleration Layer

Not every real-time database needs a dedicated acceleration card. The following three scenarios offer the clearest benefits:

1. High proportion of long-context queries: If queries involve hundreds of dialogue turns or long-document retrieval, KV Cache swapping is frequent. FX100 reduces TTFT by 26%–32% on the 480B model, with greater gains at higher concurrency (concurrency 16 is the optimal operating point) [measured, report R2].
2. Multi-instance shared models: TP4×2 full-machine throughput improves by 35%–36% [measured, report R3], making it suitable for multi-tenant database services. The FX200 (PCIe 4.0, 32M IOPS) can replace FX100 in cost-sensitive scenarios, with a full-configuration reference price of ¥331,200, approximately ¥1,797/TB.
3. Mixed training and inference workloads: Checkpoint saving accelerates by 1.9× [measured, report R1], shortening training interruption time. The FX300 (PCIe 5.0, 60M IOPS) targets future PCIe 5.0 platforms, with a reference price of ¥924,000.

Note that FX100's acceleration depends on middleware such as LMCache, and requires a network environment supporting RoCEv2. If the existing database does not use KV Cache or a model loading path, the rework effort must be assessed first. Mingxin offers a gate-based joint test of approximately 10 weeks (G1 arrival acceptance / G2 single-machine baseline / G3 main gate: TTFT reduction ≥25%, throughput +29–40% measured in-band / G4 72-hour stability), with stop-loss if targets are not met. The measurement model can be reproduced in Python after NDA.

Conclusion

The value of domestic AI inference acceleration cards in real-time database scenarios has been validated through reproducible measured data: storage-layer optimization can reduce query latency to 1/4–1/3 of the original, without dependence on a specific GPU vendor. For financial risk control and real-time recommendation systems that demand millisecond-level query responses, we recommend prioritizing evaluation of KV Cache path acceleration gains. The Mingxin FX series supports smooth evolution from PCIe 3.0 to 6.0 and integrates quickly with mainstream inference frameworks (vLLM, LMCache). For full test reports or joint validation, contact the Mingxin technical team for R1–R9 raw data and reproduction scripts.

Key Q&A from This Article

Q: How much does a domestic acceleration card improve real-time database query latency? A: In the 480B model long-context scenario, first-token latency drops by 26%–32% (concurrency 8–16) [measured, report R2]; for the no-external-recomputation scenario, TTFT accelerates 12.6× (149.5s → 11.85s) [measured, report R2].

Q: Does the acceleration depend on a specific GPU vendor? A: No. Positive gains were measured on both AMD MI308X (R1–R4) and Huawei Ascend 910B (R9) platforms, with model loading acceleration of 6.2–9.3× [measured, report R9].

Q: Which database scenarios are best suited for such acceleration cards? A: The clearest benefits are in three scenarios: high long-context query ratios, multi-instance shared models, and mixed training/inference workloads. We recommend validating via gate-based joint testing (G3 main gate: TTFT reduction ≥25%) before large-scale deployment.