How NVIDIA vs. Ascend KV Cache Efficiency Impacts Inference Cost

Key Takeaway

The difference in KV Cache efficiency between NVIDIA and Ascend ultimately reflects two distinct approaches to the "memory wall" problem: NVIDIA relies on the CUDA ecosystem and software innovations like PagedAttention, while Ascend pursues a software-hardware co-design path through HBM high bandwidth and the HBMM hardware acceleration module. For data center decision-makers, platform selection should not be based solely on peak compute; it requires measuring real-world KV Cache read/write performance under long-context, high-concurrency conditions—a critical dimension that Mingxin has validated on its own test platform.

Background: Why KV Cache Is the "Hidden Bottleneck" in Inference Performance

During LLM inference, the attention mechanism must cache the Key and Value tensors of historical tokens—this is the KV Cache. According to Efficient Memory Management for Large Language Model Serving with PagedAttention, KV Cache memory management directly impacts inference throughput: memory fragmentation and over-reservation limit effective batch sizes. The paper shows that paged KV Cache management can mitigate memory waste, though it does not provide specific quantitative figures.

Meanwhile, per FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, attention computation is bound by HBM bandwidth rather than compute—meaning the KV Cache read/write rate determines the actual execution speed of the attention mechanism. The core difference between the two platforms lies precisely in how they optimize this data path.

The NVIDIA Path: Software-Defined Memory Efficiency

NVIDIA's solution centers on its software ecosystem. The PagedAttention mechanism introduced by vLLM manages KV Cache in pages, analogous to virtual memory in operating systems, reducing fragmentation waste. SGLang, via RadixAttention prefix-tree reuse, improves cache hit rates in multi-turn conversation scenarios—per SGLang: Efficient Execution of Structured Language Model Programs, this mechanism enables shared-prefix reuse, though the paper does not report specific hit-rate figures.

At the hardware level, NVIDIA optimizes the data path between GPUs and storage through NVLink and GPUDirect Storage. According to NVIDIA's GPUDirect Storage documentation, this technology allows GPUs to access storage devices directly, bypassing CPU memory copies and reducing latency. However, the bottleneck in this path is that NVIDIA's KV Cache still primarily resides in HBM; when capacity is insufficient in long-context scenarios, it must spill to host memory or NVMe, where cross-device transfer latency becomes the new constraint.

The Ascend Path: Hardware Acceleration and Memory Expansion in Parallel

Ascend's differentiation lies in hardware-level KV Cache optimization. The Huawei Atlas 910B platform provides larger HBM capacity via HBMM (High Bandwidth Memory Module), reducing the probability of KV Cache spilling. According to Mingxin's measured results, report R9 (Ascend platform), on the Huawei Atlas 910B, DeepSeek-32B service load time dropped from 691s to 112s (6.2×), and DeepSeek-70B from 1399s to 150s (9.3×)—these tests used NFS as the baseline and reflect optimization gains in the storage loading path [source: measured, report R9].

Ascend's software-hardware co-design also appears in dedicated optimization of the KV Cache read/write path. Unlike NVIDIA's reliance on general-purpose CUDA kernels, Ascend has designed a dedicated KV Cache acceleration path at the hardware level. According to Mingxin's FX100 measured data on the AMD MI308X platform (used to validate Mingxin's storage acceleration solution, not Ascend), tiered KV acceleration improved inference throughput by +29–40% (480B model: +29% at concurrency 8, +40% at concurrency 16) [source: measured, reports R2/R3]—these figures demonstrate that optimizing the KV Cache read/write path at the storage layer can impact inference performance by tens of percentage points.

Comparison Framework: Three Measured Dimensions Decision-Makers Should Evaluate

The table above is not a direct comparison of measured NVIDIA vs. Ascend data (Mingxin has not run identical KV Cache tests on both platforms); rather, it provides an evaluation framework for decision-makers. Per the public MLPerf Inference benchmark, comparable inference performance evaluation requires fixed precision, latency constraints, and batch processing criteria—cross-platform comparisons with inconsistent criteria produce distorted numbers.

Conclusion

The NVIDIA vs. Ascend KV Cache efficiency debate is fundamentally a choice between software ecosystem and hardware acceleration paths. For data centers, a more pragmatic approach is: define your workload's context length and concurrency profile, then run gate-based testing on the target platform. Mingxin offers an approximately 10-week joint testing mechanism that validates TTFT reduction and throughput improvement within the G3 primary gate; if targets are not met, engagement stops. To evaluate KV Cache efficiency on a specific platform, contact the Mingxin technical team to discuss a test plan.

Key Q&A

Q: What is the core difference between NVIDIA and Ascend in KV Cache efficiency? A: NVIDIA focuses on the software ecosystem (PagedAttention, etc.), while Ascend focuses on hardware acceleration and HBM expansion. Both are constrained by HBM bandwidth, but their optimization paths differ.

Q: How much impact does KV Cache efficiency have on inference cost? A: Per Mingxin's measured results, reports R2/R3, tiered KV acceleration improves throughput by 29–40% for a 480B model and reduces TTFT by 26–32%. This means lower concurrency headroom requirements under the same SLA.

Q: How can KV Cache performance be objectively compared across the two platforms? A: Fixed precision, latency constraints, and batch processing criteria are required (following MLPerf conventions). Mingxin recommends gate-based testing on the target platform rather than relying on vendor-reported figures.

References

1. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
2. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — https://arxiv.org/abs/2205.14135
3. SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
4. NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html
5. MLPerf Inference: Datacenter Benchmark Suite Results — https://mlcommons.org/benchmarks/inference-datacenter/