Choosing KV Cache Placement Across GPU, CPU and SSD for Long‑Lived Sessions
agents gpu
| Source: ArXiv | Original article
Researchers propose placement policies for KV caches across GPU, CPU, and SSD to alleviate scarce GPU memory in long‑lived AI sessions.
A new arXiv pre‑print (2609.16215v1) examines how to allocate the key‑value (KV) cache that underpins transformer inference across GPU high‑bandwidth memory (HBM), CPU RAM and SSD storage for long‑lived sessions such as chat bots, autonomous agents and document‑question answering pipelines. The authors argue that KV caches quickly dominate the limited and costly GPU memory, forcing systems like Mooncake, LMCache, FlexGen, InfiniGen and AttentionStore to stretch beyond the device. Their study maps a grid of placement policies and cache sizes, revealing that even an oracle with perfect foresight cannot outperform a simple “no‑prefetch” strategy when migration traffic is considered.
The work matters because the KV cache is a primary bottleneck in scaling large language models (LLMs) on commodity hardware. As developers push models into continuous interaction loops, the memory footprint of stored attention states can exceed the capacity of modern GPUs, driving up hardware costs and limiting deployment flexibility. By quantifying the trade‑offs of moving KV data to slower tiers—CPU memory or SSD—the paper offers a framework for inference engines to make cost‑aware decisions without sacrificing latency, especially when Flash Attention is enabled and compression can be applied.
The next step will be watching how inference stacks incorporate these placement policies. Early signals come from tooling that already lets users tune KV cache types (e.g., Ollama’s compression flag) and from hardware reviews that compare iGPU bandwidth and HBM availability for local LLMs. If framework developers adopt the no‑prefetch baseline and extend it with smarter migration heuristics, we could see broader support for multi‑tier KV caching in production‑grade services. Follow‑up research may explore adaptive prefetching, tighter integration with SSD‑based paging, and real‑world benchmarks on the systems mentioned in the paper.
Sources
Back to AIPULSEN