KV-LTM: Persistent Per-Session KV Caches Serve as Long-Term Memory for Personal Assistants
| Source: ArXiv | Original article
A new approach called KV‑LTM introduces persistent per‑session key‑value caches to give personal assistants long‑term memory, bypassing the derivation step that current systems insert between queries and evidence.
A new arXiv paper titled **“KV‑LTM: Persistent Per‑Session KV Caches as Long‑Term Memory for Personal Assistants”** proposes a fundamentally different way to give AI assistants lasting recall. The authors demonstrate that today’s long‑term‑memory (LTM) stacks—whether RAG‑based retrieval, extracted‑fact stores or rolling summaries—insert a derivation or selection step between a user query and the evidence, leading to measurable content loss. In a system‑level benchmark on LongMemEval, two commercial assistants fell far short of a single model that read the conversation history offline, underscoring the gap.
KV‑LTM tackles the problem by persisting the key‑value (KV) cache generated by the model for each session. Instead of recomputing attention tensors for every new turn, the cache stores both the raw conversation text and the corresponding KV tensors, effectively creating a per‑session memory tier. This design preserves the original dialogue verbatim while eliminating the repeated “pre‑fill” computation that now bottlenecks most assistants.
The proposal matters because it promises two immediate gains: higher fidelity of remembered content and lower inference cost. For users, it could mean assistants that truly recall past interactions across days or weeks, reducing the need to repeat context. For providers, the reduction in redundant computation translates into cheaper, faster serving—an advantage as personal assistants scale to billions of daily sessions.
The next steps will be watching whether major assistant platforms adopt KV‑LTM or similar cache‑based hierarchies, and how future benchmarks compare its performance against existing LTM pipelines. Follow‑up research may also explore integration with emerging memory‑management frameworks that treat KV caches as first‑order objects in LLM serving.
Sources
Back to AIPULSEN