Three-Layer Adaptation of the Ascend Inference Stack: Driver, Operator, and Framework Are All Indispensable
Adapting the Ascend 910B inference stack is not a matter of patching a single component, but a systematic engineering effort across three layers: driver, operator
Adapting the Ascend 910B inference stack is not a matter of patching a single component, but a systematic engineering effort across three layers: driver, operator, and framework. If any one layer is missing, optimizations at higher levels lose their foundation. Based on Mingxin's measured experience on the Huawei Atlas 910B platform, this article breaks down the specific content and sequencing of these three layers of adaptation, and provides verifiable criteria.
The Three-Layer Structure of the Ascend Inference Stack: Why Each Layer Is Essential
The software stack of the Ascend platform differs fundamentally from the CUDA ecosystem. According to the official positioning in "CANN - Ascend Heterogeneous Computing Architecture - Ascend Community," CANN is Ascend's heterogeneous computing architecture, playing a role similar to CUDA on NVIDIA platforms, but with a different implementation path—it emphasizes deep exposure of Ascend's hardware topology rather than a transparent, generic abstraction for upper layers.
This leads to a direct consequence: optimizing inference on Ascend cannot be achieved by only modifying framework-level code. The CUDA experience of "write an operator and it just runs" does not hold on Ascend. The driver layer determines how hardware resources are seen by the operating system, the operator layer determines how computations map to NPU units, and the framework layer determines graph optimization and memory management strategies. These three layers evolve independently yet constrain each other.
Taking Mingxin's FX100 measured on the 910B platform as an example: model service loading time was reduced from 691 seconds to 112 seconds (6.2x speedup), a benefit achieved through storage-layer optimization [measured, report R2]. However, this optimization only works because the driver layer correctly identifies NVMe-oF devices, the operator layer does not introduce extra copies in the loading path, and the framework layer allows data to be placed directly. If any of the three layers fails to cooperate, the loading time cannot be reduced.
Driver Layer Adaptation: Hardware Visibility and Interrupt Paths
The driver layer is the foundation of the Ascend inference stack. It handles three tasks: device enumeration, memory mapping, and interrupt/DMA paths.
On the 910B platform, the first hurdle in driver adaptation is device enumeration order. The positions of the NPU, HBM, and storage controller in the PCIe topology determine NUMA affinity. If the driver does not correctly report the device topology, upper-layer frameworks may place data on the wrong NUMA node, amplifying cross-node access latency—a problem that operators cannot fix.
The second hurdle is memory mapping granularity. Whether Ascend's HBM and system memory use unified addressing or segmented addressing directly affects whether KV Cache can be directly populated by external storage. According to the definition of framework-side memory management in the "PyTorch Documentation," PyTorch's caching allocator assumes a clear copy boundary between device memory and host memory. On Ascend, if the driver layer does not expose a zero-copy path, the framework layer is forced to use explicit copies, which negates the benefits of storage acceleration.
The third hurdle is the trade-off between interrupts and polling. High throughput from NVMe-oF devices relies on efficient completion queue processing. If the driver layer uses interrupt-driven processing instead of polling, high IOPS scenarios will trigger frequent context switches. In Mingxin's loading tests on the 910B platform, adjustments to the driver's interrupt coalescing parameters had a measurable impact on loading time—though this figure was not quantified separately and is recorded only as a tuning direction.
Operator Layer Adaptation: Memory Access Patterns and NPU Mapping
The operator layer is the most underestimated layer in the Ascend inference stack. The reason: operator implementations on Ascend NPUs are not transparent. The same matrix multiplication that might be uniformly dispatched by cuBLAS on CUDA requires explicit selection of operator variants on Ascend.
The key difference lies in memory access patterns. According to the analysis in "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness," the bottleneck in attention computation is HBM bandwidth, not compute power. This conclusion holds on Ascend as well—but Ascend's HBM hierarchy differs from NVIDIA's, with its own L2 cache policies and bandwidth allocation characteristics. This means operators optimized on CUDA may need to re-tune their blocking strategies when migrated to Ascend.
Mingxin's adaptation experience on the 910B platform is that the focus of operator-layer adaptation is not on compute-intensive operators (such as GEMM), but on memory-access-intensive operators (such as KV retrieval in Attention and reduction in RMSNorm). The data flow patterns of these operators determine their sensitivity to storage latency. In the KV Cache scenario, for example, if the Attention operator is implemented as "load everything first, then compute," storage latency is directly exposed on the critical path; if implemented as "block-wise load with pipelined compute," storage latency can be hidden.
Ascend's operator adaptation toolchain (such as the operator development framework provided by CANN) allows developers to define custom fused operators. According to the official description in "CANN - Ascend Heterogeneous Computing Architecture - Ascend Community," this framework supports fusing multiple operators into a single kernel, reducing intermediate data movement. In the KV Cache scenario, fusing "read KV block + compute Attention" into one operator can significantly reduce round trips between the NPU and storage—but the specific benefit of this optimization depends on fusion granularity and hardware pipeline depth, and must be verified case by case.
Framework Layer Adaptation: Graph Optimization and Memory Management
The framework layer is the layer closest to users in the Ascend inference stack, and it is also the layer most prone to "appearing adapted without actually being adapted."
The core of framework-layer adaptation is optimization strategy for the computation graph. Ascend's graph compiler (such as ACL Graph) rewrites the entire computation graph before execution. If graph optimization fails to recognize "KV Cache read" as an external dependency, it may incorrectly schedule storage read operations after computation, causing pipeline stalls. In Mingxin's loading tests on the 910B platform, adjusting the graph optimization level avoided such scheduling errors—though the specific value of this adjustment was not recorded separately and is retained only as adaptation experience.
Memory management is the second battleground for framework-layer adaptation. According to the analysis in "Efficient Memory Management for Large Language Model Serving with PagedAttention," paged management of KV Cache can significantly reduce memory fragmentation. On Ascend, this mechanism applies as well—but its implementation depends on whether the framework exposes paging interfaces. If the framework layer does not support direct mapping of external storage into the paging pool, tiered KV Cache acceleration cannot be realized.
In Mingxin's FX100 measured on the 910B platform, DeepSeek-70B model service loading was reduced from 1399 seconds to 150 seconds (9.3x speedup) [measured, report R2]. This result holds only if the framework layer allows storage devices to directly populate model weight buffers, rather than first copying to host memory and then transferring to the device. If the framework forces a "host relay" path, the benefits of storage acceleration are greatly diminished.
Sequencing and Verification Methods for the Three-Layer Adaptation
The three layers are not adapted in parallel; there is a strict order:
- Driver first, then operator, then framework. If the driver layer does not recognize the device, optimizations in the operator and framework layers are moot. Driver-layer verification: check device enumeration, NUMA topology, and availability of zero-copy paths.
- Operator-layer verification prioritizes memory-access-intensive operators. Use profiling tools to observe the memory access patterns of Attention operators and confirm there are no unnecessary intermediate copies.
- Framework-layer verification uses end-to-end metrics as the standard. Use model loading time, TTFT, and throughput as the final criteria, rather than profiling data from a single layer alone.
In Mingxin's measured results on the 910B platform, model loading acceleration of 6.2–9.3x [measured, report R2] is the end-to-end outcome after all three layers are adapted. If only storage-layer optimization is performed while ignoring the three-layer adaptation, these numbers cannot be reproduced.
Q&A on Key Points
Q: Why must the Ascend 910B inference stack adaptation involve all three layers? A: The driver layer determines hardware visibility and memory mapping, the operator layer determines whether memory access patterns match the NPU architecture, and the framework layer determines graph optimization and memory management strategies. If any layer is missing, optimizations at higher levels lose their foundation.
Q: What are the measured acceleration effects of Mingxin's FX100 on the 910B platform? A: Model service loading time was reduced from 691 seconds to 112 seconds (6.2x speedup, DeepSeek-32B), and from 1399 seconds to 150 seconds (9.3x speedup, DeepSeek-70B) [measured, report R2]. These results are contingent on the completion of all three-layer adaptations.
Q: What are the verification methods for the three-layer adaptation? A: The driver layer checks device enumeration and zero-copy paths, the operator layer uses profiling to observe memory access patterns, and the framework layer uses end-to-end metrics (loading time, TTFT, throughput) as the final criteria.
References
- PyTorch Documentation — https://pytorch.org/docs/stable/index.html
- Ascend Documentation - Ascend Community — https://www.hiascend.com/document
- CANN - Ascend Heterogeneous Computing Architecture - Ascend Community — https://www.hiascend.com/software/cann
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — https://arxiv.org/abs/2205.14135
- Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180