FS:// Shared Pool Cross-Instance Hot Sharing: KV Reuse in Multi-Service Deployments

In multi-instance inference deployments, the reuse efficiency of KV Cache directly determines service response speed and hardware utilization. When multiple inference service instances run on the same physical machine (e.g., multiple vLLM processes or LMCache instances for different models), each instance independently manages its own KV cache, leading to significant redundant computation and VRAM waste. More critically, when instances need to share context (e.g., in dialogue systems or multi-turn inference chains), traditional local caching solutions cannot achieve real-time cross-instance reuse, forcing the system to frequently recompute or load from slow storage (e.g., NFS), significantly increasing time-to-first-token (TTFT).

This issue is particularly pronounced in long-context scenarios. For example, with a 480B-parameter MoE model, a single KV Cache recomputation yields a TTFT of 149.5 seconds (measured, report R2). Even with local NVMe caching, cross-instance sharing still requires file system copying or network transfer, resulting in low efficiency. Building a low-latency, high-throughput shared KV pool has become a critical technical challenge for compute center operators.

Storage Acceleration: From Local Cache to Shared Pool

Traditional KV Cache reuse solutions rely on local NVMe or NFS mounts. While local NVMe offers low latency (microsecond level), it cannot be shared across instances—each service instance can only access its own physical disk, leading to uneven VRAM utilization. NFS enables sharing, but its protocol stack overhead is high (typically millisecond-level latency), and concurrent write performance is poor, easily becoming a bottleneck in dense inference scenarios.

The Mingxin FX100 all-flash NVMe-oF array provides a middle path: it aggregates multiple NVMe SSDs into a shared storage pool (fs:// path) via RoCEv2 networking, allowing instances to access KV Cache files through a unified namespace. In measured results from report R2, the FX100, configured with 4 drives in RAID0 (14 TB, XFS) and a single 100 GbE port, achieved a 29–40% improvement in KV tiered acceleration inference throughput (480B model, concurrency 8–16). Its core advantages are:

1. Low-latency network access: RoCEv2 compresses storage access latency to tens of microseconds, approaching local NVMe, while supporting cross-instance sharing.
2. High concurrent write bandwidth: RAID0 aggregates the throughput of 4 drives, with sustained write bandwidth measured at 6.40 GB/s (report R1, training checkpoint saving scenario), meeting the demands of multiple instances writing KV Cache simultaneously.
3. Protocol transparency: vLLM and LMCache can directly access the fs:// path via the POSIX interface, requiring no application code modifications.

Measured Performance of Cross-Instance KV Reuse

In tests from reports R2 and R3, the FX100 shared pool solution demonstrated significant benefits in cross-instance scenarios. The test platform consisted of 8 × AMD MI308X (each with 192 GB HBM), running the Qwen3-Coder-480B-FP8 model (weights ≈450 GB), with a baseline of a single local NVMe drive.

Time-to-First-Token (TTFT): When two vLLM instances shared the same KV pool, the TTFT p50 decreased from a baseline of 10.17–35.73 seconds to 7.53–26.35 seconds (a reduction of 26–32%). More critically, in a cold recovery scenario (where an instance restarts and must reload the KV Cache), the FX100 achieved a TTFT of only 11.85 seconds (concurrency 16), compared to the baseline of 149.5 seconds without external storage recomputation—a speedup of 12.6x. This means that when an instance needs to rebuild its KV Cache due to failure or scaling, the shared pool avoids recomputing from scratch.

Throughput: At the full-machine level (TP4×2), the FX100 achieved a throughput improvement of 35–36% (measured, report R3). When multiple instances concurrently write to the KV Cache, the FX100 sustained a write bandwidth of 6.40 GB/s (measured, report R1), a 96% improvement over the single local NVMe drive's 3.26 GB/s. This bandwidth advantage directly translates to a higher KV hit rate—when KV data written by instance A is immediately read by instance B, the shared pool eliminates network copy latency.

Model Loading Acceleration: On the Huawei Atlas 910B platform (measured, report R9), the FX100 reduced the service loading time for DeepSeek-70B from the NFS baseline of 1399 seconds to 150 seconds (a 9.3x speedup). For multi-instance scenarios, this means new instance startup times drop from minutes to seconds, significantly improving the efficiency of elastic resource scheduling.

Key Engineering Practices

To implement fs:// shared pool cross-instance hot sharing, the following engineering details require attention:

1. Network Topology: RoCEv2 requires an end-to-end lossless network (PFC + ECN configuration). It is recommended to deploy the FX100 array and compute nodes on the same Layer 2 network to avoid latency jitter introduced by cross-switch forwarding. In testing, a single 100 GbE port met the KV read/write demands of the 480B model, but if the number of instances exceeds 4, enabling dual-port bonding (LACP) is advised to increase bandwidth.

2. File System Selection: XFS performs well in large-file sequential read/write scenarios (measured, report R2). However, if KV Cache files experience frequent random reads/writes (e.g., multiple instances updating different blocks simultaneously), evaluating ext4 or btrfs differences is recommended. Assigning each instance a dedicated subdirectory helps avoid file lock conflicts.

3. Caching Strategy: LMCache supports LRU-based local caching, but consistency must be considered during cross-instance sharing—if instance A modifies a KV file, instance B needs to detect the change via inotify or polling. In tests from report R1, the LMCache parallel read patch improved TTFT by 4.1x (Qwen2.5-32B), but this patch has not yet been merged into the upstream mainline, so production deployments require self-maintenance.

4. Capacity Planning: The KV Cache for a single inference of the 480B model is approximately 2–4 GB (depending on sequence length). To support 1000 concurrent sessions, 2–4 TB of shared space must be reserved. The fully configured FX100 with 14 TB (4 drives, RAID0) meets most scenarios, but if the number of instances exceeds 8, expanding to 8 drives (28 TB) and enabling RAID10 is recommended to improve random read performance.

Conclusion

Cross-instance KV reuse is a key technical path to improving inference efficiency in compute centers. The Mingxin FX100 all-flash NVMe-oF array, through its RoCEv2 shared pool, achieves a 26–32% reduction in TTFT and a 29–40% improvement in throughput for the 480B model, while reducing model loading time to 1/9 of the NFS baseline. For teams optimizing multi-instance inference deployments, it is recommended to validate actual benefits through a gated joint test (approximately 10 weeks)—Mingxin provides a standardized process from G1 arrival acceptance to G4 stability testing, supporting Python-reproducible measurements under NDA. Please contact us to obtain test images and configuration templates.