NVIDIA vs. Ascend KV Cache Efficiency: Measured Data and Architectural Differences

In AI inference scenarios, KV Cache efficiency directly determines long-context throughput and time-to-first-token (TTFT). NVIDIA and Ascend platforms exhibit significant differences in their architectural approaches. Based on measured data from the Mingxin FX100 on AMD MI308X and Huawei Ascend 910B platforms (measured, reports R2/R9), the Ascend platform excels in model loading acceleration (6.2–9.3×), while the NVIDIA ecosystem demonstrates a more mature software stack advantage in tiered KV Cache acceleration (throughput improvement of 29–40%). This article compares the two platforms across three dimensions: memory management mechanisms, disaggregated storage-compute architecture, and measured performance.

Memory Management Mechanisms: Divergent Paths of Paging and Pooling

KV Cache management in the NVIDIA ecosystem is exemplified by PagedAttention, which uses a paging mechanism to transform memory fragmentation issues into schedulable page table operations. According to Efficient Memory Management for Large Language Model Serving with PagedAttention, the core motivation of this mechanism is to address KV Cache memory fragmentation and waste, enabling inference frameworks like vLLM to achieve near-zero-waste memory utilization. This design is particularly critical in long-context scenarios—when sequence length exceeds single-GPU memory capacity, the paging mechanism allows KV Cache to span pages, and with GPU Direct Storage (per NVIDIA GPUDirect Storage documentation), it bypasses the CPU bounce buffer to establish a direct data path between GPU and storage devices.

The Ascend platform, by contrast, emphasizes unified scheduling of heterogeneous memory. While the HBM capacity and bandwidth of the Huawei 910B lag behind NVIDIA's high-end GPUs, it adopts a different strategy for storage-side acceleration. Mingxin measured data (report R9) shows that on the Ascend 910B platform, replacing NFS with an NVMe-oF array reduced DeepSeek-70B model loading time from 1399 seconds to 150 seconds (9.3× acceleration, measured, report R9). This result indicates that the Ascend platform has significant potential in storage I/O path optimization, though KV Cache memory management efficiency itself still depends on software stack maturity.

Disaggregated Storage-Compute Architecture: The Common Trend of KV Cache Pooling

According to Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving, a KVCache-centric disaggregated architecture can significantly improve resource utilization in inference systems through cross-node KV pooling and prefix cache reuse. This design philosophy has been implemented on both NVIDIA and Ascend platforms, but through different paths: the NVIDIA ecosystem relies more on mature libraries within the CUDA ecosystem, while the Ascend platform requires adaptation via self-developed frameworks.

Mingxin's measured data (report R2) on the AMD MI308X platform provides quantitative reference: with a 480B-parameter model and TP8 across three concurrency configurations, tiered KV Cache acceleration reduced TTFT p50 from 10.17–35.73 seconds to 7.53–26.35 seconds (a reduction of 26–32%). This result validates the effectiveness of the disaggregated architecture under long-context cold-restart workloads. Note that this test was conducted on the AMD platform; NVIDIA platform performance may differ due to software stack variations.

Measured Comparison: Efficiency Boundaries Across Platforms

The data above indicates that the NVIDIA ecosystem is more mature in KV Cache software stack optimization, while the Ascend platform demonstrates hardware synergy advantages in storage-side acceleration. These are not simply a matter of superiority or inferiority, but rather differences in applicable scenarios: NVIDIA suits scenarios with extreme demands on long-context inference throughput, while Ascend performs better in storage I/O-intensive tasks.

Conclusion

The differences in KV Cache efficiency between NVIDIA and Ascend fundamentally reflect a combination of software ecosystem maturity and hardware architecture design. For computing center builders, platform selection should align with specific workload characteristics: prioritize the NVIDIA ecosystem's KV Cache optimization capabilities for long-context inference, and evaluate the Ascend platform's I/O acceleration potential for storage-intensive tasks. The Mingxin FX series storage acceleration products have completed adaptation validation on both platform types, supporting a gate-based joint testing collaboration model of approximately 10 weeks. Customers with relevant needs are welcome to contact us for measured evaluation.

Key Q&A

Q: What is the core difference between NVIDIA and Ascend in KV Cache efficiency? A: The NVIDIA ecosystem is more mature in KV Cache software stack optimization, with measured throughput improvements of 29–40% (measured, reports R2/R3); the Ascend platform excels in storage I/O acceleration, with model loading acceleration of 6.2–9.3× (measured, report R9).

Q: What are the measured results of tiered KV Cache acceleration on the AMD platform? A: With a 480B model under TP8 configuration, TTFT p50 decreased from 10.17–35.73 seconds to 7.53–26.35 seconds, a reduction of 26–32% (measured, report R2); throughput improved by 29–40% (measured, reports R2/R3).

Q: How should platform selection weigh the two options? A: Prioritize the NVIDIA ecosystem's KV Cache optimization capabilities for long-context inference, and evaluate the Ascend platform's I/O acceleration potential for storage-intensive tasks. It is recommended to conduct gate-based joint testing based on specific workload characteristics.

References

1. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
2. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
3. MLPerf Inference: Datacenter Benchmark Suite Results — https://mlcommons.org/benchmarks/inference-datacenter/
4. SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
5. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — https://arxiv.org/abs/2205.14135
6. NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html
7. PyTorch Documentation — https://pytorch.org/docs/stable/index.html