Inference Deployment in Xinchuang Scenarios: Architecture Essentials for Data Not Leaving the Domain

In Xinchuang scenarios, the core requirement for inference deployment is that data does not leave the domain—meaning model weights, user data, and intermediate states (such as KV Cache) during training and inference must remain within local or controlled networks, without relying on external public networks or untrusted storage. This requires the architecture to leverage localized storage acceleration and tiered KV Cache management on domestic computing platforms (such as AMD MI308X based on ROCm or Ascend 910B), ensuring data security while maintaining inference performance. Based on measured data from Mingxin's FX100, this article provides three actionable architecture essentials: replacing NFS with a local NVMe-oF array, tiered KV Cache acceleration to reduce time-to-first-token (TTFT), and optimization without external recomputation to improve throughput, along with how to validate effects through gated joint testing.

Why Does Data Not Leaving the Domain Require Localized Storage Acceleration?

In Xinchuang scenarios, data not leaving the domain typically means that inference clusters cannot rely on public networks or cross-domain storage services (e.g., public cloud object storage) and must use local or private network storage devices. Traditionally, NFS (Network File System) is a common choice, but on domestic computing platforms (such as Ascend 910B), the latency and bandwidth bottlenecks of NFS significantly slow down model loading and inference startup.

Measured data from Mingxin FX100 on the Ascend 910B platform (report R9) shows that compared to the NFS baseline, model inference loading acceleration reaches 6.2–9.3 times: DeepSeek-32B service loading drops from 691 seconds to 112 seconds, and DeepSeek-70B from 1399 seconds to 150 seconds. This improvement stems from the localized deployment of the FX100 all-flash NVMe-oF array (4-disk RAID0, 14 TB, 100 GbE RoCEv2), avoiding NFS protocol stack overhead and network congestion. On the ROCm platform (AMD MI308X ×8), measured data (report R1) shows that training checkpoint saving also achieves a 1.9x acceleration (from 178 seconds to 94 seconds, with sustained write bandwidth increasing from 3.26 to 6.40 GB/s).

Architecture essential: Under the constraint of data not leaving the domain, local NVMe-oF or similar low-latency storage protocols (such as RoCEv2) should be adopted to replace traditional NFS, matching the loading speed of domestic GPU memory.

How Does Tiered KV Cache Acceleration Reduce Time-to-First-Token?

In inference deployment, the time-to-first-token (TTFT) in long-context scenarios is a key bottleneck, especially for MoE models (such as Qwen3-Coder-480B-FP8), where the read/write latency of KV Cache directly impacts user experience. Data not leaving the domain requires that KV Cache cannot rely on external caching services, so a tiered acceleration mechanism must be established between local storage and GPU memory.

Measured data from Mingxin FX100 on the ROCm platform (report R2) shows that by tiering KV Cache storage between the local NVMe-oF array and GPU memory, the TTFT p50 for the 480B model (TP8, three concurrent levels) drops from 10.17–35.73 seconds to 7.53–26.35 seconds, a reduction of 26–32%. This optimization is achieved through the LMCache parallel read patch (report R1), where in a cold read scenario with single-card concurrency of 16 (Qwen2.5-32B), TTFT drops from 37.97 seconds to 9.30 seconds (4.1x improvement), and bandwidth increases from 0.98 to 5.23 GB/s (5.3x improvement).

Architecture essential: On domestic computing platforms (ROCm or Ascend), a storage system supporting tiered KV Cache acceleration (such as Mingxin FX100's NVMe-oF array) should be deployed. Through localized caching and parallel read patches, TTFT can be controlled within an acceptable range (e.g., under 10 seconds) while meeting the data-not-leaving-domain requirement.

How Does Optimization Without External Recomputation Improve Throughput and Stability?

Data not leaving the domain also means that inference cannot rely on external recomputation resources (such as cloud GPU clusters), so all computations must be completed within local computing power. Optimization without external recomputation (i.e., avoiding reloading already computed KV Cache from external storage) can significantly improve throughput and reduce system jitter.

Measured data from Mingxin FX100 on the ROCm platform (report R2) shows that in a scenario without external recomputation, the TTFT p50 for the 480B model (concurrency 16) drops from 149.5 seconds (recomputation baseline) to 11.85 seconds, an acceleration of 8.6–20 times; throughput increases from 4.1 tok/s to 74.9 tok/s. Tiered KV Cache acceleration improves inference throughput by 29–40% (reports R2/R3), reaching 40% at the optimal operating point (concurrency 16), and 35–36% at the full-machine level (TP4×2).

Architecture essential: In Xinchuang deployments, a storage solution supporting no external recomputation (such as Mingxin FX100's local all-flash array) should be selected. Through KV Cache persistence and fast reading, redundant computation is avoided, maintaining high throughput and low latency under the data-not-leaving-domain constraint.

Conclusion

Inference deployment in Xinchuang scenarios does not mean compromising performance for data not leaving the domain—through localized storage acceleration (replacing NFS with NVMe-oF), tiered KV Cache acceleration (reducing TTFT by 26–32%), and optimization without external recomputation (improving throughput by 29–40%), a balance between security and performance can be achieved on domestic computing platforms (ROCm, Ascend). Mingxin Technology offers approximately 10 weeks of gated joint testing services (including measured in-band TTFT reduction ≥25% and throughput improvement of +29–40%), supporting validation in customer environments. For further understanding of architecture adaptation or joint testing processes, please contact the technical team.

Key Q&A

Q: How can inference performance be ensured while keeping data within the domain in Xinchuang scenarios?
A: Through localized storage acceleration (e.g., Mingxin FX100's NVMe-oF array replacing NFS), model loading acceleration of 6.2–9.3 times is achieved on the Ascend 910B platform (report R9); combined with tiered KV Cache acceleration, TTFT for the 480B model is reduced by 26–32% (report R2).

Q: What specific impact does optimization without external recomputation have on inference throughput?
A: On the ROCm platform (AMD MI308X ×8), in a scenario without external recomputation, throughput for the 480B model increases from 4.1 tok/s to 74.9 tok/s (report R2), and tiered KV Cache acceleration improves inference throughput by 29–40% (reports R2/R3).

Q: Does data not leaving the domain necessarily require specific storage hardware?
A: Not necessarily, but it is recommended to use low-latency local storage protocols (such as NVMe-oF or RoCEv2) to match the loading speed of domestic GPUs. Measured data from Mingxin FX100 on ROCm and Ascend platforms can serve as a reference, but actual effects should be validated through gated joint testing (e.g., TTFT reduction ≥25%).