Three-Tier Storage Architecture in Compute Centers: The Role of Hot, Warm, Cold, and KV Cache Tiering

In the storage architecture of compute centers, the core distinction among hot, warm, and cold tiers is based on access frequency and latency tolerance. KV Cache tiering acceleration is not a traditional storage tier—it sits between GPU memory (hot tier) and NVMe all-flash arrays (warm tier), using software-defined caching strategies to boost inference throughput by 29–40% (measured, reports R2/R3) and reduce time-to-first-token (TTFT) by 26–32% (measured, report R2). This optimizes inference efficiency and total cost of ownership (TCO) without adding compute cards. These conclusions are based on measured data from Mingxin FX100 on a 480B model. The following sections detail the logic of the three-tier storage and the positioning of KV cache tiering.

Logic of Three-Tier Storage in Compute Centers

Compute center storage architectures are typically divided into three tiers based on performance and cost: hot tier, warm tier, and cold tier. According to the IDC 2025 Data Center Storage Report, the hot tier primarily uses DRAM and HBM, with latency in nanoseconds but cost exceeding $100,000 per TB; the warm tier uses NVMe SSDs and all-flash arrays, with latency in microseconds and cost around $1,000–$5,000 per TB; the cold tier uses HDDs or tape, with latency in milliseconds and cost below $100 per TB. This division is not absolute but based on data access frequency and service-level agreement (SLA) requirements.

In practical compute deployments, KV Cache data for inference workloads exhibits high frequency and short lifespan. In traditional approaches, KV Cache resides entirely in GPU memory (hot tier), but memory capacity is limited (e.g., 8×192 GB HBM totals ~1.5 TB). When context length exceeds 100K tokens, memory bottlenecks lead to frequent cache eviction and recomputation. Baseline recomputation tests show that for a 480B model without external storage assistance, the TTFT at p50 reaches 149.5 seconds (measured, report R2), significantly increasing user wait time and compute idle costs.

KV Cache Tiering Acceleration: Intelligent Extension of the Warm Tier

KV Cache tiering acceleration offloads part of the cache data to the warm tier (e.g., NVMe-oF all-flash arrays), enabling hot-warm collaboration through software-defined tiering strategies. Mingxin FX100, based on PCIe 3.0 and a single-interface 100Gb NVMe-oF array, measured on a 480B model: throughput increases by 29% at concurrency level 8, and by 40% at the optimal concurrency level 16 (measured, reports R2/R3). This improvement comes from converting recomputation workloads into cache reads, avoiding GPU idle waiting.

From a TCO perspective, tiering acceleration reduces memory pressure on compute cards. For example, on an 8×MI308X platform, total memory is 1.5 TB, with the 480B model weights occupying ~450 GB, leaving ~1 TB for KV Cache. When context length exceeds 200K tokens, KV Cache demand surpasses 2 TB. Traditional approaches rely on recomputation or adding compute cards. Tiering acceleration provides 14 TB of all-flash space (measured bandwidth 5.23 GB/s, report R1) via the warm tier, enabling a single machine to handle longer contexts without the cost of scaling compute cards.

Cold Storage and Data Archiving: Boundaries of the Tiered Architecture

Cold storage primarily serves training checkpoint saving and model weight archiving. In training checkpoint saving tests with Mingxin FX100, the full model snapshot for an 8-card 32B LoRA (each 65.6 GB) reduced from 178 seconds to 94 seconds, with sustained write bandwidth improving by 96% (measured, report R1), thanks to the low-latency characteristics of NVMe-oF. However, cold storage using HDDs or tape has latency in milliseconds, making it unsuitable for real-time inference caching. Therefore, KV cache tiering acceleration is clearly positioned: it does not replace cold storage but optimizes the caching strategy of the warm tier, enabling dynamic balance between hot and warm tiers for inference workloads.

Conclusion

The three-tier storage architecture in compute centers balances cost and performance for different data access patterns, while KV Cache tiering acceleration fills the efficiency gap between hot and warm storage. Mingxin Technology demonstrates the feasibility of this path through its FX series products (e.g., FX100 measured throughput increase of 29–40%, TTFT reduction of 26–32%). For compute deployment teams aiming to improve inference efficiency without adding compute cards, consider conducting a gated joint test with Mingxin (approximately 10-week process), using Python-reproducible models to verify TCO optimization.

Key Q&A from This Article

Q: In the three-tier storage architecture of compute centers, which tier does KV Cache tiering acceleration belong to?
A: KV cache tiering sits between the hot tier (GPU memory) and the warm tier (NVMe all-flash arrays), using software caching strategies to boost inference throughput by 29–40% (measured, reports R2/R3). It represents an intelligent extension of the warm tier.

Q: What impact does KV cache tiering have on TCO?
A: By offloading part of the KV Cache to NVMe arrays, it reduces GPU memory pressure and recomputation overhead. Measured TTFT decreases by 26–32% (measured, report R2), optimizing inference efficiency and TCO without adding compute cards.

Q: What measured data does Mingxin FX100 provide for cold storage scenarios?
A: In training checkpoint saving tests, FX100 reduced the snapshot save time for an 8-card 32B LoRA from 178 seconds to 94 seconds, with sustained write bandwidth improving by 96% (measured, report R1). However, cold storage is not suitable for real-time inference caching.