Operator Compatibility Checklist Before Running vLLM on Muxi GPUs

When running vLLM inference on Muxi GPUs, the core conclusion is: operator compatibility checks should focus on three layers—the completeness of PyTorch operator mapping to Muxi's underlying operators, whether the memory management mechanisms vLLM relies on (e.g., PagedAttention) have corresponding implementations in the Muxi ecosystem, and the adaptation status of collective communication libraries and storage paths such as NVMe-oF. Once these three items are confirmed, inference services for most mainstream models can proceed to the testing phase.

As a representative of domestic compute power, Muxi GPUs (e.g., the Xiyun series) differ from the CUDA ecosystem in their software stack, which constitutes the primary source of adaptation cost when deploying vLLM. This difference is not simply a matter of "works or doesn't work," but manifests across three technical layers: operator coverage, memory management strategies, and communication primitives. Below, we break down the verification path for each.

Operator Mapping Completeness: From PyTorch Operators to Muxi's Underlying Implementation

The forward computation graph of vLLM ultimately resolves into a set of PyTorch operators. On the Muxi platform, these operators need to be mapped to the GPU's underlying instructions via its software stack (similar to Huawei's CANN heterogeneous computing architecture; according to the positioning description in "CANN-昇腾异构计算架构-昇腾社区," domestic GPUs generally adopt proprietary heterogeneous architectures to replace the CUDA ecosystem).

During verification, focus on three types of operators:

1. Attention-related operators: vLLM's core optimization lies in the paged management of KV Cache, whose motivation and memory fragmentation issues are systematically described in "Efficient Memory Management for Large Language Model Serving with PagedAttention." On the Muxi platform, confirm whether it provides an implementation equivalent to IO-aware optimizations like FlashAttention—according to the analysis in "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness," attention computation is limited by HBM bandwidth rather than compute power, meaning that if the operator implementation is not optimized for memory access paths, performance will degrade significantly.

2. MoE (Mixture of Experts) operators: Most current mainstream large models adopt MoE architectures, whose routing and expert parallelism logic involve substantial sparse computation. Verify one by one whether Muxi's operator library covers all MoE kernels invoked in vLLM's code paths.

3. Custom extension operators: Some of vLLM's high-performance paths are implemented via custom CUDA extensions. On the Muxi platform, these extensions either need equivalent implementations or must fall back to native PyTorch operators (which may reduce performance). The verification method is to run vLLM's built-in operator coverage tests and observe whether warnings such as "fallback to native" appear in the logs.

Memory Management and KV Cache: PagedAttention Implementation on Muxi

vLLM's memory efficiency heavily depends on the PagedAttention mechanism, which manages KV Cache in pages to reduce fragmentation. According to the analysis of disaggregated storage-compute architectures in "Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving," KV Cache management strategies directly impact service throughput and latency.

On the Muxi platform, two points need confirmation:

• Availability of paging primitives: PagedAttention relies on GPU-side atomic operations and memory management interfaces. If the Muxi driver does not fully expose these primitives, vLLM may degrade to non-paged mode, reducing memory utilization.
• Memory allocation strategy: According to the official definition of framework-side memory management behavior in the "PyTorch Documentation," PyTorch's caching allocator behavior may be inconsistent across different backends. If Muxi has not adapted PyTorch's memory caching mechanism, frequent memory allocation and release may occur, impacting performance.

Mingxin's KV Cache measurements on its own test platform (8×AMD MI308X) show that optimizing the storage path can improve inference throughput by 29–40% (measured, report R2/R3). This improvement presupposes that the GPU-side memory management mechanism functions correctly—if compatibility gaps exist at the operator layer, the benefits of storage-side optimization cannot be realized.

Communication and Storage Paths: Hidden Bottlenecks in Multi-GPU Inference

Multi-GPU inference relies on collective communication libraries (e.g., alternative implementations to NCCL). On the Muxi platform, confirm that its collective communication library is compatible with vLLM's tensor parallelism (TP) and pipeline parallelism (PP) strategies. According to the "NVIDIA GPUDirect Storage Documentation" on GPU-direct storage mechanisms, bypassing the CPU's bounce buffer can significantly reduce data transfer latency—whether the Muxi platform supports a similar data path directly affects the efficiency of exchanging intermediate results such as KV Cache.

The storage path is equally critical. In vLLM inference, the swapping in and out of KV Cache involves data flow between the GPU and the storage system. Mingxin's measurements on the Ascend 910B platform show that optimizing the storage loading path can accelerate model service loading by 6.2–9.3 times (measured, report R9). If the Muxi platform plans to adopt a similar tiered storage strategy, the data path between its GPU and NVMe-oF arrays should be validated in advance.

Executable Verification Path

It is recommended to proceed with adaptation work in the following order:

1. Run official examples: First, run vLLM's built-in simple model inference examples to confirm the basic operator pipeline is functional.
2. Run operator coverage tests: Use vLLM's test suite to identify the list of fallback operators and assess their performance impact.
3. Stress-test memory management: Use long-context workloads to verify whether PagedAttention is effective, observing the memory usage curve.
4. Multi-GPU communication verification: Run TP inference in an environment with 2 or more GPUs to confirm the stability of the collective communication library.
5. Storage path integration testing: If external storage is planned to extend KV Cache capacity, verify the data path between the GPU and storage.

If operator gaps are encountered during the above checks on the Muxi platform, two paths can be evaluated: first, wait for the Muxi software stack to iterate and fill the gaps; second, adjust the model structure to avoid specific operators. Mingxin has accumulated approximately 10 weeks of joint testing methodology for domestic compute adaptation (covering four phases: arrival acceptance, single-machine baseline, main gate, and stability testing). If end-to-end vLLM inference performance needs to be validated on Muxi or other domestic platforms, verification plans can be co-designed during joint testing.

Key Q&A

Q: Before running vLLM on Muxi GPUs, what is the most critical operator compatibility risk? A: The coverage of attention operators (e.g., FlashAttention-equivalent implementations) and MoE operators, as well as the availability of paging primitives that PagedAttention depends on. These two items directly determine whether inference performance meets expectations.

Q: What executable steps are available for operator compatibility checks? A: First, run official examples to confirm the basic pipeline; then run vLLM's operator coverage tests to identify the fallback list; subsequently, use long-context workloads to validate the memory management mechanism; finally, verify collective communication and storage paths in a multi-GPU environment.

Q: Will operator compatibility issues cause vLLM to be completely non-functional? A: In most cases, it will not be completely unusable; rather, it will fall back to non-optimized paths, resulting in performance degradation. It is recommended to check the logs for fallback warnings, quantify the performance loss, and then decide whether to wait for software stack iterations or adjust the model structure.

References

1. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
2. PyTorch Documentation — https://pytorch.org/docs/stable/index.html
3. CANN-昇腾异构计算架构-昇腾社区 — https://www.hiascend.com/software/cann
4. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — https://arxiv.org/abs/2205.14135
5. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
6. NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html