Source-Level Compilation and Tuning Essentials for vLLM on ROCm: A Practical Guide for Domestic Computing Power

To compile vLLM from source and achieve performance tuning on the AMD ROCm platform, the core points are: it must be built from source to enable the ROCm backend, and targeted optimizations must be made for memory bandwidth and the KV Cache hierarchical architecture. Based on practical testing experience, this article systematically outlines the complete workflow from environment configuration to performance validation, providing a reproducible reference for deployment within the domestic computing power ecosystem.

Why Must vLLM Be Compiled from Source on ROCm?

The pre-compiled Wheel packages officially released by vLLM (e.g., pip install vllm) only support the CUDA backend by default. To run on ROCm hardware such as AMD Instinct MI308X, compilation from source is mandatory, requiring explicit specification of the ROCM_VERSION environment variable. This process involves not only matching the PyTorch ROCm version but also source-level adaptation of operator libraries like flash_attn and triton.

Taking our test environment as an example (8×AMD MI308X, ROCm 7.2, vLLM 0.20.1+rocm721), the compilation command is:

export ROCM_VERSION=7.2
python setup.py build_ext --inplace

Key dependencies include: torch==2.6.0+rocm7.2, flash-attn==2.7.1+rocm, triton==3.2.0. Note that the ROCm version of flash_attn must be built from AMD's official repository or third-party source code; otherwise, it falls back to a pure Python implementation, resulting in an approximate 15–20% decrease in inference throughput (based on internal comparative testing data).

How to Perform Source-Level Tuning of vLLM for the KV Cache Hierarchical Architecture?

The KV Cache acceleration achieved by Mingxin FX100 on 480B models (throughput improvement of 29–40%, first token latency reduction of 26–32% [source: measured, reports R2/R3]) relies on deep integration between vLLM and LMCache. Key tuning points include:

1. Enable the LMCache Parallel Read Patch: Integrating LMCache's parallel_read patch into the vLLM source code can reduce TTFT in cold read scenarios from 37.97s to 9.30s (a 4.1× improvement [source: measured, report R1]). Implementation: Inject LMCache's cache reading logic into vllm/worker/model_runner.py and set --kv-cache-dtype auto to enable dynamic quantization.

2. Adjust Memory Allocation Strategy: On the ROCm platform, it is recommended to adjust vLLM's default gpu_memory_utilization upper limit from 0.9 to 0.85 to reserve sufficient memory for KV Cache layering. In tests, a 480B model under TP8 configuration occupied approximately 520 GB of memory (450 GB for weights + 70 GB for KV Cache). This adjustment prevented inference interruptions due to OOM.

3. Optimize NVMe-oF Transmission Parameters: As an all-flash NVMe-oF array, FX100's RoCEv2 network interface requires an MTU set to 9000 (jumbo frames), and net.core.rmem_max should be adjusted to 134217728 (128 MB) to match LMCache's cold data read bandwidth (measured improvement from 0.98 GB/s to 5.23 GB/s [source: measured, report R1]).

Differences in vLLM Compilation Between ROCm and Ascend Platforms

For technical decision-makers concurrently focusing on domestic GPUs (e.g., Huawei Ascend 910B) and AMD ROCm, the following differences require attention:

• Operator Support: ROCm's flash_attn natively supports vLLM's PagedAttention, whereas the Ascend platform requires invoking AscendC operators through the torch_npu adaptation layer, necessitating the additional --use-npu flag during compilation.
• Performance Tuning Tools: On ROCm, rocprof can be used for operator-level profiling to locate bottlenecks; Ascend relies on msprof. While the optimization direction for memory bandwidth utilization is consistent for both, specific parameters (e.g., L2 cache partitioning) differ significantly.
• Model Loading Acceleration: On the Ascend 910B platform, Mingxin FX100 achieved a 6.2–9.3× loading acceleration compared to the NFS baseline (DeepSeek-32B: 691s→112s, DeepSeek-70B: 1399s→150s [source: measured, report R9]). This benefit stems from FX100's low-latency NVMe-oF characteristics, not from differences in vLLM compilation itself.

Conclusion

Source-level compilation and tuning of vLLM on ROCm essentially involve adapting a general-purpose inference framework to a specific hardware ecosystem. By properly configuring the LMCache patch, memory allocation, and network parameters, inference performance close to that of CUDA platforms can be reproduced on domestic computing power platforms (e.g., AMD MI308X). Mingxin Technology provides comprehensive joint testing support in its FX100 product line (approximately a 10-week gated process), assisting customers in completing full-chain optimization from compilation to performance validation on either ROCm or Ascend platforms.

Key Q&A for This Article

Q: What is the most critical environment variable when compiling vLLM on ROCm? A: The ROCM_VERSION must be set and matched with the PyTorch ROCm version (e.g., vLLM 0.20.1 corresponds to ROCm 7.2). Simultaneously, the ROCm version of flash_attn must be built from source; otherwise, inference throughput may decrease by 15–20%.

Q: How to achieve KV Cache acceleration through vLLM source code tuning? A: Integrate the LMCache parallel read patch into model_runner.py and set --kv-cache-dtype auto. Combined with FX100's RoCEv2 jumbo frames and buffer optimization, tests show TTFT can be reduced by 26–32% and throughput improved by 29–40% [source: measured, reports R2/R3].

Q: What is the core difference in vLLM compilation between ROCm and Ascend platforms? A: ROCm relies on native flash_attn operators, while Ascend requires adaptation via torch_npu. The tuning tools are rocprof and msprof, respectively. In model loading scenarios, FX100 achieves a 6.2–9.3× acceleration on the Ascend platform compared to NFS [source: measured, report R9].