We surpass mem0 on LongMemEval-S retrieval, +11.6 P@1, using a fully local memory layer—no LLM at write time
| Source: Mastodon | Original article
A new AI system with a fully local memory layer outperformed mem0 on the LongMemEval‑S retrieval benchmark, boosting P@1 by 11.6 points across the full 500 set without using an LLM at write time.
A new open‑source memory system called **Compass** has outperformed Mem0 on the LongMemEval‑S retrieval benchmark, posting an 11.6‑point lift in precision‑at‑1 over the full 500‑item test set. On the EverMemBench‑Dynamic suite (n = 500) Compass achieved scores ranging from 44.4 % to 47.3 %, compared with Mem0’s 37.09 %, Zep’s 39.97 % and MemOS’s 42.55 %.
Compass stores session text verbatim and embeds it locally with the BGE‑m3 model. Crucially, it writes memory without invoking any large language model; all “intelligence” is deferred to read time, where an utterance‑type router classifies queries (single‑session, multi‑session, temporal, knowledge‑update, etc.) and directs each to a specialised retrieval unit. The approach eliminates LLM‑based fact extraction, graph construction and cloud calls, making writes effectively free and lossless.
The result matters because token efficiency remains a decisive factor for production‑grade AI agents. Mem0’s own research highlighted that its algorithm can keep retrieval calls under 7 000 tokens while still scoring 92.5 on LoCoMo and 94.4 on LongMemEval. Yet the benchmark gains come with a reliance on LLMs at write time, adding latency, cost and privacy concerns. Compass demonstrates that a fully local pipeline can not only cut those overheads but also deliver higher retrieval accuracy, suggesting a viable path for cost‑sensitive deployments and on‑premise solutions.
The next steps to watch include broader adoption of Compass‑style architectures in open‑source agent frameworks, further testing on the full LongMemEval suite and related benchmarks such as BEAM, and potential refinements to the embedding layer or routing logic. Industry observers will also be keen to see whether other projects, building on the memory‑evaluation tools we covered in earlier reports on Mem0, adopt a “no‑LLM‑at‑write” stance to balance performance with operational efficiency.
Sources
Back to AIPULSEN