Short answer. In a study posted to arXiv on 25 October 2026, researchers from UIUC, Tsinghua and three other groups put 11 open-source agent memory systems up against plain retrieval over the raw conversation history. On most benchmarks and models, most memory systems scored lower than that plain baseline, and usually cost far more to build. The authors call the loss a “memory tax”, and their fix is simple: keep a path back to the raw history.
Checked on 28 October 2026.
What they compared
The paper is “Memory Tax: Dissecting the Net Value of Agent Memory” (arXiv 2610.34073). Its question is blunt: if you build a memory layer for an agent, how much better is it than just searching what was actually said?
On one side are 11 memory systems in four families:
- flat stores: A-Mem, SimpleMem, MemOS;
- layered or lifecycle memory: MemoryOS, LightMem, OpenViking;
- graph memory: HippoRAG2, PlugMem, MAGMA;
- trained memory policies: Mem-T, Memory-R2.
On the other side is “naive” RAG. The history is cut along conversation turns, with short turns merged into chunks of about 1,000 characters, and retrieval mixes keyword search (BM25) with embeddings and a reranker. The two-round version checks whether the first results are enough and, if not, searches again.
The test sets were LongMemEval-s, PersonaMem-v2 and MINT-Eval, run on three models: Qwen3.5-4B, Qwen3.6-27B and GPT-5.6 Luna. Every system used the same embedding model, so the comparison is about memory design, not about who has the better embeddings.
Worth knowing before you share this: popular commercial memory products such as Mem0, Zep and Letta were not part of the test. The paper says nothing about how they would score.
What they found
On LongMemEval-s with GPT-5.6 Luna, two-round RAG scored 90.0%. The best memory systems, A-Mem and SimpleMem, reached 84.0%. OpenViking reached 63.0%. Even dumping the whole history into the context window gave 84.0%.
The pattern held on the other test sets too. There were a few exceptions: PlugMem and Mem-T beat RAG on MINT-Eval with GPT-5.6 Luna, by 3.3 and 3.9 points. So memory isn’t useless. It wins some questions that RAG misses, it loses others that RAG gets right, and in most setups the losses were bigger.
The authors also checked whether RAG only won because it was well tuned. They swapped chunking, retrieval method, embedding model and reranker, and most variants stayed within a few points of the default. Their conclusion is that RAG has been underrated as a baseline in earlier memory papers.
Where the tax comes from
The paper traces the losses to three causes.
- Facts get lost when memory is built. Summarising or restructuring the history can drop a detail, and nothing later can bring it back.
- Facts survive but get harder to find. One fact gets split across several memory entries, mixed in with unrelated material, or paraphrased until it no longer matches the question. In the authors’ replay, memory systems recovered fewer answers on the first search pass than RAG did.
- Memory breaks more easily when the model changes. Memory built with one model and read by another lost more accuracy than RAG did under the same swap. That matters for any agent that will outlive its current model.
There is one caveat that works in memory’s favour. The benchmarks are friendly to retrieval: a simple keyword method already scores above 50% on some of them. On tasks that really need abstraction, memory has more room to help. The authors saw this with task-experience memory, where turning long web-task traces into reusable workflows did beat RAG, while on tool-use tasks several memory methods fell behind.
What to do with this
The fix the authors suggest doesn’t need a new architecture.
- Keep the raw history, and search it alongside memory. Retrieve from the source history and from memory at the same time, then rerank the results together before answering. In their tests this cut the tax across several memory systems.
- Let the agent remember how it found things. Store past questions together with the evidence that answered them, so a fact that took three searches to find is one search away next time.
- Measure against plain RAG. The authors argue for reporting a RAG baseline next to memory results, because it shows how much room is really left. Our advice goes a step further: if you build or buy a memory layer, test it against plain hybrid retrieval over the same history. Without that number you can’t tell whether the memory is helping.
In practice: treat summaries, facts and memory graphs as an index over the history, not a replacement for it. If the original record is gone, so is your way to check what the memory says.
Related reading
- Claude Code Memory Optimization: Auto Memory, Auto Dream, and the Bugs Documentation Doesn’t Mention
- How I Picked an Embedding Model for Arcanada
- Datarim: Do Agents Dream of Electric Sheep?
FAQ
Is agent memory worse than RAG?
Not always. In this study most memory systems scored below plain hybrid RAG over the raw history on conversational benchmarks, but some won on specific tasks. The net result depends on how much the memory loses compared with what it adds.
What is the memory tax?
It’s the set of questions plain retrieval answers correctly but a memory system gets wrong, because building memory lost the fact, made it harder to find, or made results depend more on the model.
Does this study say Mem0 or Letta lose to RAG?
No. Mem0, Zep and Letta were not tested. The 11 systems tested are open-source research and industry systems such as A-Mem, MemOS, HippoRAG2 and OpenViking.
Should I keep raw history next to agent memory?
The authors suggest yes: searching the raw history alongside memory and reranking both together recovered facts the memory had lost or buried.
Source: Xuan He et al., “Memory Tax: Dissecting the Net Value of Agent Memory”, arXiv:2610.34073, submitted 25 October 2026. https://arxiv.org/abs/2610.34073