Inference Deployment in Xinchuang Scenarios: Architecture Essentials for Data Not Leaving the Domain
In Xinchuang scenarios, one of the core challenges of inference deployment is meeting the compliance requirement of "data not leaving the domain"—meaning training data, model weights, and inference intermediate states (such as KV Cache) must always remain within controlled domestic computing infrastructure and must not rely on overseas cloud storage or public networks. Based on measured data from Mingxin Technology on AMD ROCm and Huawei Ascend platforms, this article presents the following conclusions: by deploying an all-flash NVMe-oF storage array (such as the Mingxin FX100) combined with KV cache tiered acceleration, inference throughput can be improved by 29–40% and first-token latency reduced by 26–32% in domestic GPU clusters, while ensuring data never leaves the domain throughout its lifecycle. The following discussion covers three dimensions: storage architecture, KV cache acceleration, and model loading efficiency.
Storage Architecture for Data Not Leaving the Domain: Why NVMe-oF Is Key
The core meaning of data not leaving the domain is that the generation, storage, and transmission of training data, model weights, and inference intermediate states (such as KV Cache) must occur entirely within controlled domestic computing infrastructure. In traditional approaches, model weights are often loaded from external storage via NFS (Network File System), which relies on the TCP/IP protocol stack and may introduce additional latency and security risks on domestic GPU platforms (e.g., Huawei Ascend 910B). Measured results show that on the Huawei Atlas 910B platform, loading the DeepSeek-70B model via the Mingxin FX100 all-flash NVMe-oF array reduced service loading time from 1399 seconds to 150 seconds (a 9.3x acceleration, measured, report R9), meaning model weights can be fully loaded from a storage system within the domain without relying on external networks.
NVMe-oF (NVMe over Fabrics) achieves remote direct memory access (RDMA) via the RoCEv2 protocol, reducing storage latency to microseconds while supporting data encryption and access control. In Xinchuang scenarios, this architecture allows storage nodes to be deployed in the same data center or availability zone, forming a "compute-storage closed loop." The Mingxin FX100, as an all-flash NVMe-oF array, offers a single interface bandwidth of 100Gb (PCIe 3.0). Measured on the AMD MI308X platform, its KV cache read bandwidth reaches 5.23 GB/s (measured, report R1), providing a high-performance, low-latency storage foundation for data not leaving the domain.
Tiered KV Cache Acceleration: Engineering Path to Reduce Latency and Improve Throughput
In long-context inference (e.g., 480B parameter MoE models), KV Cache storage and reading are bottlenecks. Traditional approaches store all KV Cache in GPU memory, but memory capacity is limited (e.g., 192GB HBM per card on MI308X). Once the context length exceeds memory capacity, part of the KV Cache must be offloaded to external storage (e.g., local NVMe SSD), causing a significant increase in time-to-first-token (TTFT)—measured results show that without external storage recomputation, TTFT p50 can reach 149.5 seconds (measured, report R2). The Mingxin FX100 addresses this through tiered acceleration (GPU memory → NVMe-oF storage): cold KV Cache is offloaded to the FX100 array, leveraging its high bandwidth (measured 5.23 GB/s, report R1) and low latency. Under 16 concurrent streams, TTFT dropped from 149.5 seconds to 11.85 seconds (12.6x acceleration, measured, report R2), while throughput increased from 4.1 tok/s to 74.9 tok/s (18x acceleration).
The key to this architecture is "tiering": hot KV Cache remains in GPU memory, while cold KV Cache is offloaded to NVMe-oF storage as needed. The Mingxin FX100's LMCache parallel read patch further optimizes read efficiency: on the Qwen2.5-32B model, with single-card 16 concurrent cold reads, TTFT dropped from 37.97 seconds to 9.30 seconds (4.1x improvement, measured, report R1). For Xinchuang scenarios, this means inference services can support longer contexts (e.g., 128K context for 480B models) without increasing GPU memory, while meeting data-not-leaving-domain requirements—since all KV Cache is stored on the in-domain NVMe-oF array.
Model Loading and Training Checkpoint Acceleration: Practical Gains on Domestic Computing Platforms
Data not leaving the domain involves not only inference but also model deployment and training. On domestic computing platforms (e.g., Huawei Ascend 910B), model weight loading speed directly affects service availability. Traditional NFS loading of DeepSeek-70B requires 1399 seconds (about 23 minutes), while the Mingxin FX100 compresses this to 150 seconds (2.5 minutes, measured, report R9), a 9.3x acceleration. This means in Xinchuang scenarios, models can be quickly loaded from in-domain storage to GPUs, reducing service downtime.
On the training side, checkpoint saving is another critical step. In an 8-card 32B LoRA training scenario, the save time for each full model snapshot (65.6GB) dropped from 178 seconds to 94 seconds (1.9x acceleration, sustained write bandwidth increased from 3.26 GB/s to 6.40 GB/s, measured, report R1). For training tasks on domestic computing platforms (e.g., Huawei Ascend or AMD ROCm) that require frequent checkpoint saves, this acceleration can significantly reduce training wait times while ensuring data always flows within in-domain storage.
Conclusion
Inference deployment for data not leaving the domain in Xinchuang scenarios requires deep collaboration between compute and storage. The Mingxin FX100 series all-flash NVMe-oF array, through measured KV cache tiered acceleration (throughput +29–40%, TTFT -26–32%) and model loading acceleration (6.2–9.3x), provides a compliant, high-performance storage foundation for domestic computing platforms (AMD ROCm, Huawei Ascend). Computing centers and AI enterprises are welcome to contact us for joint testing to verify the practical benefits of the FX100 in their own Xinchuang environments.
Key Q&A
Q: In Xinchuang inference scenarios where data must not leave the domain, what are the core requirements for storage architecture?
A: Storage must support the NVMe-oF protocol (e.g., RoCEv2) to achieve a compute-storage closed loop, ensuring model weights and KV Cache always remain within the domain without relying on external networks. The Mingxin FX100, measured on the Huawei Ascend platform, accelerates DeepSeek-70B loading by 9.3x (measured, report R9), meeting this requirement.
Q: How does the Mingxin FX100 improve latency and throughput for long-context inference?
A: Through KV cache tiered acceleration (GPU memory → NVMe-oF storage), under 16 concurrent streams for a 480B model, TTFT dropped from 149.5 seconds to 11.85 seconds (12.6x acceleration), and throughput increased from 4.1 tok/s to 74.9 tok/s (measured, report R2). The LMCache parallel read patch further improves cold-read TTFT by 4.1x (measured, report R1).
Q: On domestic GPU platforms (e.g., Ascend), what practical benefits does the FX100 offer for model loading?
A: On the Huawei Atlas 910B platform, the FX100 reduces DeepSeek-70B service loading time from 1399 seconds to 150 seconds (9.3x acceleration), and DeepSeek-32B from 691 seconds to 112 seconds (6.2x acceleration), both from measured report R9.