LowRAM proposes low‑rank keys and amortised values for composable KV cache compression at test time
training
| Source: HF Papers | Original article
Researchers present LowRAM, a test‑time method compressing KV caches in autoregressive models with low‑rank keys and amortised values, easing memory growth that limits batch size and throughput.
A new paper titled **LowRAM: Low‑Rank Keys and Amortised Values for Composable KV Cache Compression at Test‑Time** proposes a two‑pronged approach to shrink the memory used by the key‑value (KV) cache that underpins autoregressive decoding. By applying a low‑rank approximation to the keys and replacing the stored values with a lightweight predictor that can be amortised across tokens, the authors achieve a **26.9× reduction in cache size while retaining 94.5 % of the model’s baseline performance on long contexts**. The technique is deliberately asymmetric – keys and values are treated as structurally distinct – and can be stacked with existing post‑training tricks such as quantisation and cache eviction, further amplifying the compression gains.
The KV cache grows linearly with context length, quickly becoming the dominant memory consumer in large language model (LLM) serving. This limits batch size, throttles throughput, and forces costly hardware upgrades for applications that need very long prompts or real‑time interaction. LowRAM’s aggressive yet composable compression directly tackles that bottleneck, promising cheaper inference and the ability to run larger contexts on the same hardware footprint. The results complement earlier work on test‑time model compression that focused on redundancy across tokens, layers, or feature dimensions, showing that a targeted redesign of the cache data structures can deliver orders‑of‑magnitude savings.
The community will now watch for integration of LowRAM into popular inference stacks such as vLLM, TensorRT‑LLM, or open‑source serving libraries. Follow‑up studies are likely to explore how the predictor for values behaves across diverse model families, whether the method scales to multimodal or instruction‑tuned models, and how it interacts with emerging compression schemes like LoopVQ. If the approach proves robust in production, it could reshape the economics of long‑context LLM deployments across the Nordic AI ecosystem and beyond.
Sources
Back to AIPULSEN