The Hidden Layer https://essays.bloo-mind.ai/ Bloo-Mind AI Essays quarto-1.9.37 Tue, 19 May 2026 23:00:00 GMT The Benchmark Theatre: Why Almost Nothing You’ve Read About Agent Memory Scores Is True Dell Zhang https://essays.bloo-mind.ai/posts/2026-05-20-mem-eval/

TL;DR

  • Almost every headline number on the agent long-term memory (LTM) leaderboard collapses on contact with a careful reader, and the failure modes are not exotic. They are the same handful of evaluation sins, repeated by nearly every system, including the polite ones. The Mem0 vs Zep vs Letta dispute, the LoCoMo audit, the MemPalace fiasco, and the new MemoryBench / AMemGym dynamic-evaluation papers are all pointing at the same six problems, from rather different angles.
  • The deepest issue is structural, not moral. The current benchmarks (LoCoMo, LongMemEval) are short enough to fit comfortably inside a frontier context window, contain double-digit percentages of broken ground truth, are scored by remarkably lenient LLM judges that accept the majority of vague wrong answers, and are run by each vendor on its own pipeline using its own prompts. Under those conditions, “state of the art” is whatever the marketing team has decided it is on any given Tuesday.
  • If you are picking an LTM system today, ignore the published score. Run your own evaluation with full-context and filesystem-search baselines on your own data, audit the ground truth before you trust it, and demand multi-seed numbers with the judge prompt published in full. If a vendor will not share the judge prompt and the eval harness, the score does not, in any useful sense, exist.

Key Findings

  1. The flagship benchmarks are corrupted. Penfield Labs’ systematic audit of LoCoMo (Penfield Labs, 2026), the most-cited conversational-memory benchmark, found 99 score-corrupting errors in 1,540 questions — a 6.4 % rate, nearly double the “at least 3.3% errors across the 10 datasets” that Northcutt, Athalye, and Mueller reported in Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks (2021). The theoretical ceiling on the published LoCoMo is therefore about 93.6 %. Scores reported above that ceiling — including EverMemOS’s 95.96 % single-hop and 91.37 % multi-hop — are mathematically impossible without taking credit for getting wrong answers wrong in the right direction.

  2. The judges accept what they should reject. The standard LoCoMo judge (GPT-4o-mini with the original prompt) accepted 62.81 % of intentionally wrong-but-topical answers in Penfield’s stress test. Vague answers that named the right topic but missed every specific detail sailed through nearly two times in three. That is, conveniently, precisely the failure mode of weak retrieval, and the benchmark is now rewarding it.

  3. The competitive numbers are a public fight, not a measurement. Mem0’s April 2025 paper claimed SOTA over Zep on LoCoMo. Zep’s blog post — titled, with admirable directness, “Lies, Damn Lies, & Statistics” — claimed Zep actually beat Mem0 by 24 %, then quietly corrected to 10 % (75.14 % ± 0.17 against Mem0’s best of ~68 %) after Mem0’s CTO documented an arithmetic error: Zep had included Category 5 answers in the numerator while excluding them from the denominator, inflating the headline by roughly 25 percentage points. Letta then strolled in and reported 74.0 % using a plain filesystem and grep, no memory system at all, undercutting both. It is, on balance, not the field’s finest hour.

  4. The hottest project of 2026 was a marketing artefact. MemPalace, credited to actress Milla Jovovich and engineer Ben Sigman, reached 1.5 million tweet impressions and over 5,400 GitHub stars in twenty-four hours on a claim of “100 % on LoCoMo” and “the first perfect score ever recorded on LongMemEval.” Within forty-eight hours, the project’s own BENCHMARKS.md was found to confess, on line 461, that the 100 % had been produced by writing three patches for three specific dev-set questions — “teaching to the test”, in the file’s own words — and that the LoCoMo 100 % had been achieved with top-k = 50 against a candidate pool that maxes out at 32 sessions, which reduces the retrieval step to the rather less glamorous task of pouring the entire conversation into Claude Sonnet.

  5. The benchmarks measure context-window management, not memory. LongMemEval-S, the second-most-cited memory benchmark, uses “approximately 115k tokens per problem” (Wu et al., 2025). Current frontier models support 200K to 1M tokens. The entire test corpus, in other words, fits inside a single prompt. On LongMemEval-S, GPT-4o with full-context scored about 60.2 %; Mastra’s “observational memory” scored 84.23 %. The benchmark is therefore measuring how cleverly a system organises text inside an already-sufficient context window. This is not, by any reasonable definition, long-term memory.

  6. The new wave of “dynamic” benchmarks shows the old leaders losing. Three 2025-2026 papers — MemoryBench (Tsinghua), AMemGym (HKUST/Meituan, ICLR 2026), and CASCADE — explicitly construct interactive, on-policy, multi-domain test environments. MemoryBench’s authors report being “surprised to find that the time of existing memory-based methods used to memorize input context and conducting online inference are extremely long” and that “none of the advanced memory-based LLMsys (i.e., A-Mem, Mem0, or MemoryOS) can consistently outperform RAG baselines that simply use all task context and feedback logs as retrieval corpus.” This contradicts, with considerable directness, the published numbers of all three systems.

Details

Hook: The Milla Jovovich Story, and Why It Is Not a Joke

On 5 April 2026, an account named MemPalace/mempalace pushed a Python repository to GitHub (MemPalace contributors, 2026). The next day, a developer named Ben Sigman — by day the CEO of a Bitcoin-lending marketplace called Libre — tweeted the launch. The tweet credited the actress Milla Jovovich, of Resident Evil and The Fifth Element, as co-author. The repository’s headline claim was unambiguous and somewhat unhinged: “100 % on LoCoMo” and “the first perfect score ever recorded on LongMemEval. 500/500 questions, every category at 100 %.”

By the next morning the tweet had crossed 1.5 million impressions, the repo had over 5,400 stars, and the news cycle had begun. Forbes ran an uncritical puff piece. Cybernews wrote it up under the headline “Milla Jovovich creates MemPalace AI memory tool.” A “MemPalace” memecoin appeared briefly on pump.fun, with a 50/50 creator-reward split, and was pumped and dumped within twenty-four hours — pour one out, perhaps, for the late retail bagholders. The repo crossed 26,900 stars and 3,300 forks within a few days.

The choice of name was, on its face, lovely. The “memory palace” technique — formally, the method of loci — is the oldest mnemonic device in the Western tradition. It is attributed to the Greek poet Simonides of Ceos, who, having stepped outside a banquet hall moments before the roof came in, was able to identify each crushed corpse by remembering where its owner had been sitting. Cicero adapted the trick for Roman oratory; Matteo Ricci taught it to sixteenth-century Chinese scholars; Benedict Cumberbatch’s Sherlock Holmes turned it into a Tumblr meme in the 2010 BBC series, where the line “I need to go to my mind palace” became one of those phrases the show is now contractually obliged to deliver at least once an episode. Applying that scaffold to an LLM’s context store — wings for projects, halls for memory types, rooms for topics, drawers for verbatim text — is a perfectly sensible piece of design.

The problem was not the metaphor. The problem was that the marketing version of MemPalace and the codebase version of MemPalace turned out to be two quite different artefacts. Within hours, Penfield Labs published a six-thousand-word teardown. It catalogued, in order:

  • The LoCoMo bypass. The runner set top_k = 50 against conversations that contain at most 32 sessions, which means the entire candidate pool is always returned and the retrieval step reduces to reading comprehension by Claude Sonnet over every session. The project’s own BENCHMARKS.md says this verbatim: “the embedding retrieval step is bypassed entirely.” It would be hard to put it more plainly without simply admitting it on a billboard.
  • The LongMemEval metric category error. LongMemEval is an end-to-end QA benchmark scored by a GPT-4 judge. The MemPalace runner does retrieval only — it never generates an answer and never invokes a judge. It returns the top-five sessions and gets credit if any one of the gold session IDs is in that set. That is recall@5 on a labelled dataset, a meaningfully easier task than the one the LongMemEval leaderboard scores. Calling 96.6 % R@5 “the highest published LongMemEval result” is not so much a mistake as a category change announced in passing.
  • The hand-tuned 100 %. The hybrid v4 mode that produced the perfect score was built by inspecting the three remaining wrong answers in the dev set and writing targeted code for each: a quoted-phrase boost for one question, a person-name boost (“Rachel”) for another, and “I still remember” / “when I was in high school” patterns for a third. Three patches for three specific questions. Then the same five hundred questions were rerun and the result published as perfect. The project’s own benchmarks file, on line 461, calls this “teaching to the test.” Discovering this confession in the repository is rather like finding, on page 461 of the prospectus, a note explaining that the company has no products.
  • Features absent from the code. The launch tweet declared that “contradiction detection catches wrong names, wrong pronouns, wrong ages before you ever see them.” The relevant file, mempalace/knowledge_graph.py, contains precisely zero occurrences of the substring “contradict”; its only deduplication is an exact-match check on (subject, predicate, object) triples. Conflicting facts about the same entity may therefore accumulate indefinitely, which is — one assumes — not the advertised behaviour.
  • AAAK is not lossless. The tweet claimed “30x lossless compression.” The dialect module truncates sentences at fifty-five characters and provides no round-trip decoder. The same BENCHMARKS.md measures AAAK at 84.2 % R@5 against raw mode’s 96.6 %, a 12.4-point regression. Lossless compression cannot, definitionally, cause measured quality loss.

On 7 April, Sigman and the maintainers posted a public “what we got wrong” note in the README, walking back every inflated claim and thanking the issue filers. The retraction is, to its considerable credit, candid: “Brutal honest criticism is exactly what makes open source work, and it’s what we asked for.” By the second week of April, the headline 100 % had quietly become 96.6 %, with the appropriate caveat that this is R@5 on session retrieval and not the leaderboard metric. The honest version of the project — verbatim storage plus default ChromaDB embeddings beats LLM-extraction approaches at session retrieval — is genuinely interesting and would have been worth a blog post. It did not require a perfect score.

The MemPalace story matters not because MemPalace was uniquely bad. It matters because every individual failure mode in MemPalace is a failure mode the rest of the AI-memory field has been quietly committing for two years, only without an actress’s name to make the press care. Penfield’s authors, who maintain their own memory project and have an obvious axe to grind, said this in the article’s most important sentence: “The methodology errors are common across the field. The honesty gap between the repository and the marketing is arguably the bigger story. The celebrity name is the reason anyone heard about it.” Which is to say: the only unusual thing here was that anyone bothered to look.

The Mem0 vs Zep vs Letta Affair

Six months before MemPalace, the same dynamic played out among the serious projects, in slower motion and without the benefit of a memecoin.

In April 2025, Mem0 published Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory on arXiv (2025). The paper reported, on the LoCoMo benchmark, a 26 % relative improvement over OpenAI’s built-in memory, p95 latency reduced by 91 %, and tokens reduced by 90 %. Mem0’s best configuration scored about 68 %. The paper benchmarked against six categories of baseline, including Zep, which Mem0 helpfully reported at 65.99 %.

On 6 May 2025, Zep published a blog post titled Lies, Damn Lies, & Statistics: Is Mem0 Really SOTA in Agent Memory? (2025) The post argued that when Zep was implemented correctly, its score on LoCoMo was 84 %, not the 65.99 % Mem0 had reported, putting Zep 24 % ahead of Mem0 rather than 3 % behind. Zep’s specific complaints: Mem0’s harness used a user-graph designed for single-user conversations but assigned both speakers to the user role; Mem0 appended timestamps to message bodies instead of using Zep’s created_at field; and Mem0 ran searches sequentially instead of in parallel.

Three weeks later, on Zep’s own GitHub, Mem0’s CTO Deshraj Yadav opened an issue with the rather dry title Revisiting Zep’s 84 % LoCoMo Claim: Corrected Evaluation & 58.44 % Accuracy (2025). The accusation was specific and damning: Zep’s calculation had included Category 5 (the adversarial category everyone agrees to exclude) in the numerator but excluded it from the denominator, mechanically inflating the score by approximately 25 percentage points. Mem0 reran Zep’s pipeline with the correction and ten independent seeds; the result was 58.44 % ± 0.20, not 84 %. Mem0 also noted, with a certain composure, that Zep’s original article had reported only a single run and had modified the system prompt in a way that gave Zep an advantage no other baseline received.

Zep then conceded. Daniel Chalef, Zep’s CEO, edited the original blog post: “we erred in how we calculated Zep’s LoCoMo score. We’ve updated the article to reflect Zep’s corrected result is 75.14 % +/- 0.17, with Zep outperforming Mem0 by 10 %.” The 10 % lead was still, technically, a lead. But the absolute number had moved from 84 % to 75.14 %, which is to say by more than a full standard deviation of the published variance across the field. The post about lies, damn lies and statistics had, on consideration, been a touch lighter on statistics than the title implied.

While that was happening, Letta — the company that came out of the original MemGPT paper — published Benchmarking AI Agent Memory: Is a Filesystem All You Need? (2025) The headline finding was that a Letta agent running GPT-4o-mini, with no specialised memory tools at all, scored 74.0 % on LoCoMo simply by storing the conversation history in a file and giving the agent grep, search_files, open, and close. This beat Mem0’s reported 68.5 % for its best graph configuration. The conclusion the post drew, with admirable understatement, was that “current memory benchmarks may not be very meaningful” and “memory is more about how agents manage context than the exact retrieval mechanism used.” This is the polite way of saying that everyone in the room had built a graph database to do the work of a Unix utility shipped in 1973.

So, to summarise: on the same benchmark, against the same dataset, in the same season, the three best-funded memory companies in the world reported, respectively, that their system was state of the art (Mem0, ~68 %), that the competitor’s number was wrong by twenty-five points (Zep, 84 % → 75 %), and that you could match any of them with a Bash builtin (Letta, 74 % via filesystem and grep). The reasonable observer’s response is not to pick a winner. The reasonable response is to stop trusting LoCoMo, and possibly to take a long walk.

What Is Actually Wrong With LoCoMo

LoCoMo (Maharana et al., 2024) was the first conversational-memory benchmark to push past the toy regime: ten synthesised dialogues, each with 19 to 32 sessions and averaging 16,000-26,000 tokens, with 1,986 question-answer pairs (1,540 non-adversarial). It was, when it appeared, a genuine contribution. It is now structurally unfit for what it is being used for, in six identifiable ways.

1. The ground truth is broken. Penfield’s audit found 99 score-corrupting errors in 1,540 questions, a 6.4 % rate. The flavour of the errors is instructive. One question asks about a car a speaker mentioned; the answer key says “Ferrari 488 GTB”; the source conversation contains only the phrase “this beauty” and an image caption reading “a red sports car”; the car model exists only in the annotator’s internal search-string field, which no memory system ingests. As Penfield puts it: “Systems are evaluated against facts they have no access to.” Twenty-four other questions attribute statements to the wrong speaker, so any system with accurate speaker tracking will systematically disagree with the answer key. One temporal question requires that “Last Saturday” on a Thursday resolve to the preceding Saturday; the answer key says Sunday; “a system that performs the date arithmetic correctly is penalised.” The theoretical ceiling, given this noise floor, is about 93.6 %. Systems reporting above that ceiling — and EverMemOS reports 95.96 % single-hop and 91.37 % multi-hop, with an overall score 1.25 points from the aggregate ceiling — are, mathematically, claiming credit for getting the wrong answers wrong in the right direction.

2. The judge is lenient. LoCoMo is scored by an LLM judge — usually GPT-4o-mini — with the original prompt explicitly instructing it to be “generous.” Penfield’s adversarial test fed the judge intentionally wrong but topically adjacent answers for all 1,540 questions. The judge accepted 62.81 % of them. Wrong-name and wrong-date errors were caught about 89 % of the time, but vague answers that named the right topic and missed every specific detail passed nearly two-thirds of the time. That is the failure mode of weak retrieval — finding the right conversation but extracting nothing specific — and the benchmark is rewarding it. When the noise floor of the judge itself is sixty percentage points wide, score differences smaller than that are not so much uninterpretable as actively misleading.

3. The conversations are short enough that “no memory” is competitive. Zep’s own blog post conceded that Mem0’s own results showed a full-context baseline (just feed the entire conversation to the LLM) scoring about 73 %, which beat Mem0’s best result of 68 %. If feeding all the text yields a better answer than the specialised memory system, the benchmark is not stressing memory. It is stressing context-window management. This is the AI equivalent of running the London Marathon and discovering the winner took the bus.

4. There is no standardised pipeline. Each system uses its own ingestion method (defensible, given architectural differences), its own answer-generation prompt (less defensible), and sometimes a different judge model (frankly indefensible). Scores from different vendors are then plotted in the same table as if they shared methodology. They do not.

5. Reproducibility is documented to fail. Third-party reproduction attempts on EverMemOS yielded 38.38 % against the official 92.32 % (issue #73). Mem0 has multiple open reproducibility issues (#3004 and #3944). The widely circulated Chinese-language cross-evaluations on Zhihu document a similar pattern: when authors run a competitor’s pipeline themselves, they routinely cannot match the published number, and the competitors respond that “the web API version has more optimisations” not in the open-source release. This is, charitably, a feature for paying customers. Uncharitably, it is unfalsifiable benchmarking. The Zhihu cross-evaluations also note an under-discussed version-drift problem: “Zep upgraded from v2 to v3 after its official LoCoMo results were published. EverMind’s evaluation used Zep v3, which may explain why Zep scored higher in EverMind’s horizontal eval than in other vendors’ evals.” When the version of the system being benchmarked is itself a moving target, the leaderboard is no longer measuring algorithms; it is measuring release cadence.

6. The category sizes do not support statistical inference. Per-category Wilson-Score 95 % confidence intervals show that 56 % of adjacent-pair comparisons across published systems are statistically indistinguishable; the Open-Domain category, with n = 96, requires a 15-point gap to distinguish any two systems. Only Mem0 documents a multi-run methodology (10 independent seeds); most systems report single-run point estimates. In a field where single-point differences of one or two percentage points are routinely advertised as victories, this is, to put it gently, a category error in basic statistics.

LoCoMo-Refined, an open project at mem-eval-suite/LoCoMo_refined, is the only serious attempt at a fix. It rewrites the judge prompt around the principle “inclusive without contradiction, complete without overreach”: answers must cover all required information, must not add unsupported content, and must align time information strictly. On 300 manually annotated samples, the new judge (Qwen3-14B with the refined prompt) achieved 86.33 % agreement with humans, against 43.67 % for the original LoCoMo setup. The team also cleaned 337 problematic samples — ambiguous wording, reversed subject-object relationships, time information inconsistent with the source conversation — releasing 1,382 questions in the refined dataset. Under the stricter rules, “the same system predictions score noticeably lower,” which is precisely what one would expect if the old benchmark had been overly forgiving. The project is, regrettably, almost unknown outside the Chinese-language research community for the prosaic reason that it has not yet had its arXiv moment in English. This is the sort of gap the field could probably fix in an afternoon.

The Six Reasons The Scores Are Not Real

The MemPalace story, the Mem0–Zep–Letta affair, and the LoCoMo audit are not separate scandals. They are three different views of the same underlying problem, which can be stated plainly in six points.

Reason 1: The LLM is doing most of the work — and you cannot tell how much

When MemPalace’s hybrid v4 pipeline scored 100 % on LongMemEval, the actual retrieval step was bypassed; Claude Sonnet was reading every session and answering by reading comprehension. When Letta’s filesystem agent scored 74.0 % on LoCoMo, GPT-4o-mini was running grep iteratively. When MemoryBench tested A-Mem, Mem0, and MemoryOS, Tsinghua’s authors found that “none of the advanced memory-based LLMsys can consistently outperform RAG baselines that simply use all task context and feedback logs as retrieval corpus.” The “memory system” in many production stacks is, on inspection, a thin wrapper around an LLM call. The score is therefore dominated by the wrapper’s choice of base model and prompt, not by the architecture being marketed. Until ablations hold the model constant — and almost none do — comparing two memory systems on benchmark numbers is comparing two LLM configurations with a layer of XML in between.

Reason 2: LLM-as-judge has known, measured, severe biases — and is rarely audited

The standard practice across LoCoMo, LongMemEval, MemoryBench, and most newer benchmarks is to score with an LLM judge. Three well-documented biases corrupt this in practice:

  • Self-preference. Wataoka, Takahashi, and Ri (2024) found that “LLMs assign significantly higher evaluations to outputs with lower perplexity than human evaluators, regardless of whether the outputs were self-generated. This suggests that the essence of the bias lies in perplexity and that the self-preference bias exists because LLMs prefer texts more familiar to them.” The practical consequence is that any memory system whose answers happen to look familiar to the judge — same model family, similar fine-tuning distribution, similar prose style — gets a measurable score bump that has nothing to do with whether the answer is correct. The judges, in effect, prefer their own dialect.
  • Length and vagueness bias. Verbosity bias (preferring longer answers) and the “vague-but-topical” bias documented by Penfield (62.81 % acceptance rate on intentionally wrong vague answers) systematically reward the wrong behaviour: paraphrasing the question back at the user rather than producing a specific recalled fact.
  • Position bias. When the same two answers are presented in reversed order, judges’ verdicts flip more often than is statistically tolerable; Shi et al.’s IJCNLP 2025 systematic study (2025) found that judge model choice had the single largest impact on positional bias of any variable tested.

The fix is mechanical and rarely applied: validate the judge adversarially with intentionally wrong answers, publish the judge prompt verbatim, and report the human-agreement rate. The LoCoMo-Refined project did this and showed a forty-point gap between the original setup (43.67 % human agreement) and the refined one (86.33 %). The other published scores are, by implication, running on the lower number.

Reason 3: The corpora fit in a context window

LongMemEval-S, with “approximately 115k tokens per problem” (Wu et al., 2025), fits inside any 200K-token frontier model. LoCoMo, at 16-26K tokens per conversation, fits in essentially any modern context window with room to spare for a chat about the weather. Mem0’s own paper showed full-context baselines beating Mem0. Mastra showed GPT-4o full-context scoring 60.2 % and observational memory scoring 84.23 % on LongMemEval-S, which means the benchmark is measuring how cleverly text is organised inside an already-sufficient context, not whether information can be recalled from outside it. A long-term memory benchmark whose entire corpus fits in a single prompt is, on the kindest reading, not measuring long-term memory. It is measuring chain-of-thought efficiency in a particularly inefficient way. Penfield’s audit puts the point unsparingly: “LongMemEval-S tests whether a model can locate information within 115K tokens. That is a useful capability to measure, but it is a context window test, not a memory test.”

Reason 4: Conflict of interest — the system authors are running the evaluations

The Mem0 paper benchmarked Zep with a misconfigured Zep harness, mapping both speakers to the user role and bypassing the SDK’s timestamp API. Zep’s response benchmarked Zep with a corrected harness, a Zep-favourable prompt, a single seed, and (as later acknowledged) an arithmetic error. Mem0’s response re-corrected the arithmetic. Letta benchmarked Letta, and so did Cognee, LangMem, A-Mem, MemoryOS, EverMemOS, Supermemory, ByteRover, OMEGA, Hindsight, SuperLocalMemory, ENGRAM, and now MemPalace. Each system author is the sole party with deep enough knowledge of their own system to configure it correctly, and the sole party with a strong incentive to misconfigure the competition. This is, to be quite clear, the configuration that science has been politely refusing to use since approximately 1665. The result is exactly as one would expect: every system reports being state of the art on the benchmark of its choice. The Zhihu cross-evaluations note that “only MemOS and EverMemOS have publicly disclosed complete horizontal evaluation methodologies,” which is to say that of the dozen-plus systems on the leaderboard, only two have published reproducible cross-evaluations against the others. The other ten or so are essentially advertisements.

Reason 5: Single-trial reporting with no statistical significance

Mem0’s paper used 10 independent runs and reported standard deviations. This is, somewhat embarrassingly for the field, the high end of current practice. Most published numbers — Zep’s original 84 % LoCoMo, Letta’s 74 %, MemPalace’s 96.6 %, EverMemOS’s 92.32 %, the various OMEGA numbers — are single-run point estimates. Given that LoCoMo’s per-category sample sizes range from 96 (Open-Domain) to 841 (Single-Hop), an 8.8x ratio, the Wilson-Score 95 % confidence intervals around these single points are remarkably wide. Even with ten clean reruns, Penfield calculates that Open-Domain remains 3.0x less precise than Single-Hop. None of the public score tables show error bars. The field is, in effect, reporting third-decimal-place precision on data that does not honestly support second-decimal-place precision. One could be forgiven, on reading these tables, for thinking they came from a sales deck.

Reason 6: Static benchmarks miss the actual job

This is the deepest critique, and the one the field is — finally, belatedly — starting to address. Static QA benchmarks ask: “Given this fixed log of past interactions, can the memory system retrieve the right answer to this question?” Real agent memory has to support a continuous loop: the agent acts, the user reacts, the system writes that reaction back into memory, and the next decision is conditioned on the updated state. Three recent papers attack this gap explicitly.

MemoryBench (Ai et al., 2025). The authors construct 20,000 cases across eleven datasets in three domains, four task formats, and two languages, with a user-feedback simulation framework. They benchmark A-Mem, Mem0, and MemoryOS against vanilla and SFT baselines on Qwen3-8B. The headline finding, on which the authors are entirely direct about contradicting prior published results: “none of the advanced memory-based LLMsys can consistently outperform RAG baselines that simply use all task context and feedback logs as retrieval corpus” — and previously published wins on LoCoMo “do not generalise” outside reading-comprehension settings. They are also “surprised to find that the time of existing memory-based methods used to memorize input context and conducting online inference are extremely long.” MemoryOS, per the paper, “exhibited better consistency in terms of inference time, but it took much longer time in memory construction (mostly longer than 17 seconds per case).” Mem0 could not finish two of their task types (“Open-Domain” and “LiSo”) because its memory pipeline could not process the input in reasonable time. Some methods, the paper drily notes, would have required “days or even weeks” to run on-policy across all datasets. A memory system that beats the leaderboard but cannot run the test in a working week is not, in practice, a memory system anyone is going to deploy.

AMemGym (Jiayang et al., 2026). This is the cleanest critique of off-policy evaluation in print. The authors note that all existing benchmarks “rely on static, off-policy data as context, limiting evaluation reliability and scalability.” Worse, “off-policy evaluation introduces reuse bias, undermining memory optimization and configuration selection.” In their direct comparison, the off-policy ranking of memory implementations disagrees with the on-policy ranking by up to three positions. Concretely, on the on-policy benchmark, the best agentic-write configuration scored 0.291 against 0.253 off-policy — and switching from off-policy to on-policy made the same configuration jump three ranks. The agents themselves degrade catastrophically: when ground-truth state is injected into context (the upper bound), all evaluated LLMs scored above 0.8; “however, as the interaction history grows with state updates, their performance drops sharply, with most models falling below 50 % of their upper bounds. Some models even perform no better than random guessing in later periods.” Claude Sonnet 4, the best model they tested, peaks at 0.336 on-policy. This is the field’s polite way of saying the kings, on closer inspection, are not wearing anything in particular.

CASCADE (Guo et al., 2026). The third paper formalises “deployment-time learning” as a contextual bandit problem and shows, in the words of the abstract, that “across 16 diverse tasks spanning medical diagnosis, legal analysis, code generation, web search, tool use, and embodied interaction, CASCADE improves macro-averaged success rate by 20.9 % over zero-shot prompting while consistently outperforming gradient-based and memory-based baselines” — implying, by simple arithmetic, that the memory baselines being beaten were leaving substantial room on the table that the static benchmarks had simply failed to surface.

Several other efforts (MEMTRACK from Patronus AI, MemoryArena, Mem2ActBench, MemGUI-Bench, LifeBench, ConvoMem, StoryBench) point in the same direction: when memory is evaluated in interactive, multi-session, action-conditional settings, the current SOTA falls off a cliff. Patronus’s MEMTRACK (Deshpande et al., 2025) reports that “the best performing GPT-5 model only achieves a 60 % Correctness score on MEMTRACK” — and the more elegant moral across the entire literature is that the “memory systems” beating each other by single points on LoCoMo are roughly indistinguishable from a no-memory baseline once you switch to a task where memory actually matters.

Notable Numbers, For The Record

Because the field’s reported numbers are unstable and the noise floor handsomely exceeds many of the gaps being claimed, I will not produce a ranked leaderboard. The following are widely cited published scores that any sensible reader should treat as version-dependent, vendor-contested, and prone to revision without notice:

  • Mem0 on LoCoMo: ~66.9 % base; ~68.5 % graph variant (Mem0 paper, 2025).
  • Zep on LoCoMo: 65.99 % (as reported in the Mem0 paper) → 75.14 % ± 0.17 (corrected, 10 runs, by Zep in 2025) → 58.44 % ± 0.20 (re-corrected by Mem0 on Zep’s own pipeline). Pick your preferred reality.
  • Letta on LoCoMo: 74.0 % using a filesystem and grep only, GPT-4o-mini (Letta, August 2025).
  • MemPalace on LoCoMo: 60.3 % R@10 in raw mode, no LLM rerank (per the project’s own honest documentation); the headline 100 % is invalid for the reasons set out above.
  • Zep on LongMemEval-S: 71.2 % with GPT-4o (Zep 2025); also reported 63.8 % on the temporal slice elsewhere.
  • Mastra OM on LongMemEval-S: 94.87 % (vendor-reported, GPT-5-mini actor); against ~60.2 % for GPT-4o full-context — the gap reflects context-window management, not memory retrieval.
  • EverMemOS on LoCoMo: 92.32 % overall; third-party reproduction on issue #73 reported 38.38 %.
  • MemPalace on LongMemEval-S: 96.6 % R@5 raw mode — but this is recall on labelled session IDs, not the leaderboard’s QA-judge metric; the hand-tuned 100 % is invalid.
  • A-Mem, Mem0, MemoryOS on MemoryBench: none consistently beats the RAG-with-all-context baseline (Ai et al., 2025).
  • GPT-5 on MEMTRACK: 60 % Correctness, the ceiling among all tested models (Deshpande et al., 2025).

The instructive observation is the spread. On LoCoMo alone, the same paper (Mem0’s) puts Zep at 65.99 %, the corrected version puts it at 58.44 %, Zep’s own (corrected) blog post puts it at 75.14 %, Letta puts a filesystem at 74.0 %, the full-context baseline runs around 73 %, MemPalace’s honest score is 60.3 %, and EverMemOS claims 92.32 % which a third party reproduces at 38.38 %. That is a range of 54 percentage points, on the same benchmark, with the same dataset, across the same approximate year, attributable not to algorithms but to evaluation choices. A field in which the same system can score 38 % or 92 % depending on who is running the test is not a field with reliable benchmarks. It is a field with very enterprising marketing departments.

What The Dynamic Benchmarks Actually Test

A short note on what a properly-shaped LTM benchmark looks like, because the next round of leaderboards will inevitably use these and the public reading them deserves a chance to understand the difference.

On-policy means the evaluation conversation is generated during the test, by interaction with the system being evaluated, rather than replayed from a fixed transcript. This matters because the agent’s own behaviour shapes what the user says next, which shapes what the memory system has to store, which shapes the next decision. Off-policy evaluation on a fixed transcript treats memory as a retrieval task; on-policy evaluation treats it, more honestly, as a control loop.

State-grounded means each user has a structured underlying state (preferences, facts about their life, ongoing tasks), and the user simulator’s job is to reveal portions of that state through natural conversation, not merely to ask questions about it. AMemGym is the cleanest example. Its simulator pulls personas from NVIDIA’s Nemotron-Personas (100K personas), samples question schemas, generates state-evolution trajectories with narrative life events, and writes ground-truth answers for each (question, state) pair. The metric is whether the assistant produces the right personalised answer given everything the user has gradually exposed across the conversation. The role-play user simulator (GPT-4.1) is meta-evaluated against humans at Gwet’s AC1 >= 0.96 — a level of judge-validation almost no static benchmark approaches.

Long-horizon means more than a single session, and ideally enough turns that the test exceeds any plausible context window. AMemGym’s “extra” configuration requires 512K+ tokens of context, putting it comfortably beyond the comfortable range of any current frontier model.

Action-conditional means the agent is actually doing something — taking actions in an environment with consequences — and the memory has to support those actions rather than merely answer questions about the past. SWE-Bench, WebArena, and Patronus’s MEMTRACK fall in this family. MEMTRACK’s GPT-5 ceiling of 60 % correctness is, in my reading, the single most important data point in the entire LTM literature: the best model in the world, on a memory-and-state-tracking task across realistic enterprise tools (Slack, Linear, Git), is at 60 %. The marketing material, by curious coincidence, almost never mentions this.

Once you start using benchmarks of this shape, the entire current LTM leaderboard collapses into noise. The systems are not being tested on the thing they are being sold to do.

Recommendations

For practitioners actually choosing an LTM system today, here is what I would do. These are staged so you can stop at the first one that gives you a clear answer.

Stage 1 — now. Ignore the leaderboard. Build a full-context baseline and a filesystem-with-grep baseline on a sample of your data. If your task fits in a 200K-token context, the full-context baseline will be embarrassingly competitive, and you will have just saved yourself a vendor relationship. If grep matches your vendor’s number within five points, your vendor is selling you grep with a logo on it. Threshold to escalate: a memory system needs to beat full-context plus grep by at least ten points on your own task before it justifies its existence on your stack.

Stage 2 — next quarter. If you do actually need a memory layer, run two independent evaluations using two different judge models with two different (published, adversarially-validated) prompts. Demand multi-seed numbers — at least ten runs — and ask the vendor for the standard deviation. If they have not measured it, they do not know what they are selling. Threshold to abandon: if the variance across seeds is larger than the gap to your second-choice vendor, you are choosing on noise, which is to say you are not really choosing.

Stage 3 — when the benchmarks improve. Track MemoryBench, AMemGym, MEMTRACK, ConvoMem, and LoCoMo-Refined. When two or more of these become the standard reporting metric and the vendor leaderboards converge on the on-policy numbers, the field will have had its honest reckoning. Until then, the direction of progress on dynamic benchmarks is informative, but the absolute numbers on static benchmarks are decorative.

Stage 4 — strategic. If you are building an agent product whose value genuinely depends on memory, the most defensible position is to own your evaluation. Build a small, private, adversarially-judged, on-policy benchmark over your real use cases and update it monthly. Use it as the gate for any vendor swap. This is the same lesson the search-quality teams learned in the late 2000s; the LTM field is, on a generous reading, about ten years behind that curve.

Decision rules. Use a managed memory service (Mem0, Zep cloud) when the operational simplicity is worth more than the benchmark uncertainty — early prototypes, customer-support bots, simple personalisation. Use the agent-managed-memory pattern (Letta, MemGPT-style) when the agent will run long enough that the model’s own tool use will outperform any external memory abstraction. Use local-first stores (Cognee, ChromaDB-on-disk, MemPalace in its honest mode) when privacy or cost dominates. Treat all benchmark deltas under ten points as noise, and any vendor unwilling to publish their judge prompt as a polite no.

Caveats

  • I have skin in the game. Every author writing on this subject does. I am writing from the perspective of someone who has shipped memory-augmented agents and who has, on more than one occasion, been the person who chose between Mem0 and Zep and Letta and ended up rolling their own. Treat my preference for adversarially-validated, on-policy evaluation with the appropriate suspicion of someone who has been comprehensively burned by the alternative.
  • The honest core of several projects is still genuinely interesting. MemPalace’s raw verbatim retrieval beating LLM-extraction approaches on session retrieval is, I think, a real and useful negative result. Zep’s temporal knowledge graph does seem to help on knowledge-update questions. Letta’s filesystem result is the most important single methodological data point of 2025. Mem0 is, by some distance, the most operationally polished managed service. The critique here is not of the engineering. It is of the marketing’s relationship to the engineering.
  • The benchmarks are improving in real time. LoCoMo-Refined exists. MemoryBench, AMemGym, MEMTRACK, and CASCADE are all from the last twelve months. The field is in the middle of correcting itself, which is to its credit. The numbers in this article are accurate to mid-May 2026 and will be stale within a quarter; the structural critique should age rather better than the specific scores.
  • The Penfield Labs critique is partisan. Penfield maintains a competing memory project, and has the natural axe to grind. I have cross-checked their numerical claims (99 errors in 1,540 questions; 62.81 % judge acceptance of vague wrong answers; the SHA256-verified audit repository at dial481/locomo-audit) and they hold up; the audit dataset is open and independently inspectable. But the editorial framing in their Substack post is, like Zep’s “Lies, Damn Lies” post and Mem0’s GitHub-issue rebuttal, a vendor critique of another vendor. The field’s structural problem is that this remains the only kind of public critique it routinely receives.
  • Reproducibility is now an explicitly political variable. Multiple systems have responded to public reproduction failures with a polite shrug and the explanation that “the cloud / paid version has additional optimisations.” This is a perfectly reasonable business model and a perfectly disastrous epistemic practice. If you are evaluating any LTM system, demand that the cloud version be benchmarked against the open-source version on the same task, and demand to see the gap. Where this gap is hidden, assume it is large.
  • The MemPalace authors deserve the partial credit they have taken. Of all the parties in this saga, the MemPalace team is the one whose public response to scrutiny was the most rapid and the most candid. Sigman and Jovovich pushed a corrected README within forty-eight hours, listed every error, and thanked the issue filers by name. Their headline numbers were inflated; their post-mortem behaviour was, by the field’s standards, exemplary. That this is worth pointing out says rather more about the field than it does about MemPalace.

Further Reading

For readers who would like to follow this research direction more systematically — the products, tutorials, surveys, benchmarks, and papers etc., that actually define the agent-memory field — I maintain a curated, regularly updated list with collaborators at Awesome-Agent-Memory. It is where I would point anyone trying to get oriented quickly. Pull requests, are, as ever, very welcome.

References

Ai, Q., Tang, Y., Wang, C., Long, J., Su, W., & Liu, Y. (2025). MemoryBench: A Benchmark for Memory and Continual Learning in LLM Systems. https://arxiv.org/abs/2510.17281
Chalef, D. (2025). Lies, Damn Lies, & Statistics: Is Mem0 Really SOTA in Agent Memory? Zep blog post. https://blog.getzep.com/lies-damn-lies-statistics-is-mem0-really-sota-in-agent-memory/
Chhikara, P., Khant, D., Aryan, S., Singh, T., & Yadav, D. (2025). Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory. https://arxiv.org/abs/2504.19413
Deshpande, D., Gangal, V., Mehta, H., Kannappan, A., Qian, R., & Wang, P. (2025, October). MEMTRACK: Evaluating Long-Term Memory and State Tracking in Multi-Platform Dynamic Agent Environments. NeurIPS 2025 SEA Workshop. https://arxiv.org/abs/2510.01353
Guo, S., Du, Y., Chen, H., Chang, Y., & Wang, J. (2026). CASCADE: Case-Based Continual Adaptation for Large Language Models During Deployment. https://arxiv.org/abs/2605.06702
Jiayang, C., Ru, D., Qiu, L., Li, Y., Cao, X., Song, Y., & Cai, X. (2026, March). AMemGym: Interactive Memory Benchmarking for Assistants in Long-Horizon Conversations. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2603.01966
Letta Team. (2025). Benchmarking AI Agent Memory: Is a Filesystem All You Need? Letta blog post. https://www.letta.com/blog/benchmarking-ai-agent-memory
Maharana, A., Lee, D.-H., Tulyakov, S., Bansal, M., Barbieri, F., & Fang, Y. (2024). Evaluating very long-term conversational memory of LLM agents. Annual Meeting of the Association for Computational Linguistics (ACL). https://arxiv.org/abs/2402.17753
MemPalace contributors. (2026). MemPalace: Local-First AI Memory. GitHub repository. https://github.com/MemPalace/mempalace
Northcutt, C. G., Athalye, A., & Mueller, J. (2021). Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks. Advances in Neural Information Processing Systems (NeurIPS). https://arxiv.org/abs/2103.14749
Penfield Labs. (2026). We Audited LoCoMo: 6.4% of the Answer Key is Wrong. Reddit, r/MachineLearning. https://www.reddit.com/r/MachineLearning/comments/1s54cvg/d_we_audited_locomo_64_of_the_answer_key_is_wrong/
Shi, L., Ma, C., Liang, W., Diao, X., Ma, W., & Vosoughi, S. (2025, December). Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge. Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL). https://doi.org/10.18653/v1/2025.ijcnlp-long.18
Wataoka, K., Takahashi, T., & Ri, R. (2024, October). Self-preference bias in LLM-as-a-Judge. NeurIPS 2024 Safe Generative AI Workshop. https://arxiv.org/abs/2410.21819
Wu, D., Wang, H., Yu, W., Zhang, Y., Chang, K.-W., & Yu, D. (2025). LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2410.10813
Yadav, D. (2025). Revisiting Zep’s 84% LoCoMo Claim: Corrected Evaluation & 58.44% Accuracy. GitHub issue, getzep/zep-papers #5. https://github.com/getzep/zep-papers/issues/5
]]> agent memory https://essays.bloo-mind.ai/posts/2026-05-20-mem-eval/ Tue, 19 May 2026 23:00:00 GMT The Master Algorithm of AI, Finally? On LLM-Empowered Heuristic Learning Dell Zhang https://essays.bloo-mind.ai/posts/2026-05-12-master-alg/

TL;DR

  • Pedro Domingos’ The Master Algorithm (2015) posed the central question of ML in suitably grand terms: find one learner that unifies the five tribes, learns from data without per-domain engineering, and acquires knowledge of arbitrary complexity. Ten years on, nobody has produced it, and people seem to have stopped looking.
  • Four independent works published in the last few months — Jiayi Weng’s “Learning Beyond Gradients” (2026), Jun Wang’s Memento(2025), Memento 2 (2025) and Memento-Skills (2026) series, and Andrej Karpathy’s LLM-wiki (2026b) and AutoResearch (2026a) — have, with no apparent coordination, converged on the same paradigm. I shall call it LLM-Empowered Heuristic Learning (HL): a closed loop in which a frozen LLM edits an external, human-readable artefact (rules, skills, wiki, code) rather than its own weights.
  • Scored against Domingos’ three criteria, HL is the strongest candidate for the master algorithm we have ever had. It plausibly satisfies all three. The honest reasons to hesitate are not the obvious ones — Turing-completeness handles arbitrary complexity perfectly well, thank you — but harder questions about whether HL can discover genuinely new knowledge, whether the compute economics survive contact with reality, and whether a “system” architecture is the kind of master algorithm Domingos was looking for in the first place.

Domingos’ Question, Restated

Before assessing whether anything qualifies as the master algorithm, it helps to be precise about what Domingos was asking. In The Master Algorithm, the criteria are not delivered as a single numbered list — that would have been too tidy — but they are clear enough across the book and his subsequent talks:

  1. Unification. A master algorithm must be capable of doing what each of the five tribes’ algorithms can do — symbolic rule learning, neural-network function approximation, evolutionary search, probabilistic inference, and analogical reasoning. The book’s narrative arc is precisely the search for an architecture that combines them.
  2. Learning from data without per-domain engineering. The master algorithm should be a general learner. Domingos is explicit that hand-engineered, domain-specific systems do not count, no matter how impressive — the lesson he draws from the 1980s expert-systems era, when the rules worked beautifully provided you employed enough PhDs to keep writing them.
  3. Knowledge of arbitrary complexity. The learner must be able to represent and acquire knowledge with no in-principle ceiling. Domingos invokes universal Turing machines and uses the language of “all the knowledge in the world,” which one suspects he means more or less literally.

These three criteria are the spine of this article. I shall resist the temptation to add a fourth (efficiency, interpretability, alignment, what have you); Domingos did not, and HL deserves the same playing field he gave his preferred candidate, Markov Logic Networks (MLNs).

A note before going further. Domingos has been publicly sceptical of LLMs — calling them inefficient, hallucination-prone, and “not truly intelligent” — and his “Tensor Logic” work is his own attempt at a unifier. I have no evidence that he has commented on Heuristic Learning specifically. The provocation in this article’s title is therefore mine, not his; I suspect he would, if pressed, ask me to step outside. What follows is an argument that HL has a stronger claim to satisfying his three criteria than any candidate he himself has proposed — and that he may be wrong to withhold the title.

What HL Actually Is

The Weng blog: an anomaly that became a paradigm

Jiayi Weng’s post opens with what he calls “the anomaly.” While maintaining EnvPool in his spare time, he wanted a cheap way to test that game environments behaved correctly without burning compute on neural-network rollouts. He asked Codex (GPT-5.4) to write a rule-based Breakout policy. The score progression — 387 → 507 → 839 → 864 — has by now become a small Internet artefact, and the appendix walks through each jump with verbatim code, RAM-byte probes, video replays, and trial logs. What grabbed me was not the score. It was the structure of what Codex produced: not a policy.py of any recognisable kind, but a self-maintaining experimental system with action probes, state readers, ball-landing predictors, stuck-loop detectors, regression tests, replay videos, and trials.jsonl. Codex did not just write a policy; it built a small laboratory and then began running experiments in it.

From this, Weng abstracts to a clean definition:

“HL is built out of program code… the object being updated is software structure rather than neural-network parameters… Its updates do not use backpropagation. The coding agent directly edits policies, state detectors, tests, configuration, or memory.”

He names the maintained artefact a Heuristic System (HS). The crucial framing — and where Weng is genuinely original — is the maintenance curve. Expert systems and rule bases failed in the 1980s not because rules are useless, but because the people maintaining them turned out to be expensive, mortal, and prone to leaving for industry. Spinning machines changed the production curve for textiles; coding agents change the maintenance curve for heuristics. That single analogy reframes about forty years of AI history.

The table at the heart of the post compares Deep RL to HL across six axes (policy, state, action, feedback, update, memory) and lists HL’s properties: explainability, sample efficiency, regression-testability, constrained overfitting via multi-seed evaluation, and partial immunity to catastrophic forgetting. Weng is honest about the limits. HL “cannot do everything neural networks can do.” On Atari games like Atlantis, VideoPinball, UpNDown, and StarGunner, PPO still smokes the coding agent. Montezuma’s Revenge required 86 hand-stitched macro-actions for a single 400-point run — which is to say, not so much a learned policy as a rather long film script.

Memento, Memento 2, Memento-Skills: the same idea, formalised

While Weng was poking Codex in his spare time, Jun Wang’s group at UCL was doing the load-bearing theoretical work. The lineage:

  • Memento (AgentFly) (Zhou et al., 2025). The empirical paper. Memory-augmented Markov Decision Process (M-MDP), neural case-selection policy, frozen LLM. Top-1 on GAIA validation at 87.88% Pass@3, 79.40% on the test set, 66.6% F1 / 80.4% PM on DeepResearcher. Case-based memory adds 4.7–9.6 absolute points on out-of-distribution tasks.

  • Memento 2: Learning by Stateful Reflective Memory (J. Wang, 2025). The theory paper, single-authored. Wang defines a Stateful Reflective Decision Process (SRDP) as a tuple of the usual MDP ingredients — states, actions, transition dynamics, rewards, and a discount factor — augmented with two extra terms: a memory, and a frozen LLM acting as a base policy. The composite reflective policy mixes that base policy over the cases currently in memory:

    Read = policy improvement (KL-regularised soft Bellman backup); Write = policy evaluation (memory rewriting). The paper proves convergence to an optimal retrieval policy under bounded rewards and a discount factor below one, monotonic policy improvement on the fast time scale, and an asymptotic value gap that grows with the maximum reward and the inverse square of one-minus-discount, and shrinks as the memory’s coverage of the state space improves — the residual error being governed by the LLM’s generalisation error and the retrieval error. Wang’s central thesis:

    “We argue that the key mechanism for continual adaptation, without updating model parameters, is reflection: the agent’s ability to use past experience to guide future actions.”

  • Memento-Skills: Let Agents Design Agents (Zhou et al., 2026). The instantiation. Skills are stored as structured markdown folders containing a SKILL.md declarative spec plus prompts and executable code. A behaviour-trainable router (a tuned Qwen3-Embedding-0.6B called “Memento-Qwen”) selects skills; the frozen LLM executes; failures trigger failure-attribution, skill rewrite, or skill discovery, with a unit-test gate to prevent regression. The agent starts from 5 atomic skills and grows to 41 after GAIA learning and 235 after HLE learning. GAIA training 65.1% → 91.6% over three reflective rounds; HLE 30.8% → 54.5%. On the GAIA test set, Memento-Skills scores 66.0% against the Read-Write ablation’s 52.3% — a +13.7-point gap that quantifies the value of skill-level reflection over plain case-based reasoning.

What makes the Memento–Weng pairing entertaining is that neither side has noticed the other. Wang’s papers and Weng’s blog post are months apart; the venues could not be more different (formal mathematical theory on one side, an engineer’s blog with mp4 replays on the other); and yet they converge on the same five-step loop: Observe → Read → Act → Feedback → Write. They differ in representation — Memento-Skills uses markdown skill folders manipulated through retrieval; Weng uses a single growing Python codebase manipulated directly by Codex — but they are unmistakably siblings, possibly twins separated at birth.

Karpathy’s LLM-wiki and AutoResearch: the consumer version

Andrej Karpathy released two artefacts in the same season pulling in the same direction.

LLM-wiki (Karpathy, 2026b), published 4 April 2026, describes a pattern in which an LLM agent reads raw documents and compiles them into a persistent, human-readable Obsidian-compatible markdown wiki — index file, entity pages, [[wiki-links]], provenance, contradictions flagged. Karpathy’s framing is explicit: “Stop re-deriving, start compiling.” The wiki, not RAG retrieval at query time, is the persistent, compounding artefact. The gist itself reports that his research wiki on a single ML topic grew to “~100 articles and ~400,000 words,” which is to say, somewhat longer than this article.

AutoResearch (Karpathy, 2026a), released 7 March 2026, is the same idea pointed at ML research. A ~630-line single-GPU stripped-down nanochat training script (train.py), an immutable evaluator (prepare.py), and a human-authored program.md. A coding agent edits train.py, trains for five minutes, keeps changes that lower val_bpb, discards the rest. Karpathy’s own two-day run produced ~700 experiments and stacked 20 additive improvements that dropped “Time to GPT-2” from 2.02 hours to 1.80 hours.

If you squint, this is just Weng’s Heuristic Learning with a different objective metric. Karpathy himself has framed the progression as vibe coding → agentic engineering → autonomous research. As Fortune reported on 17 March 2026, he announced on X that “All LLM frontier labs will do this. It’s the final boss battle,” adding that doing it is “just engineering” and “it’s going to work.” Frontier labs have, on the available evidence, called several things “the final boss battle” in the past eighteen months, and the boss appears to keep respawning. Read this one as a billboard, not a forecast. The ratchet only accepts immediate improvements, so it cannot do “worse before better” — a real limitation, flagged in the repo’s own GitHub issues with a touching lack of embarrassment.

The upstream tradition

It would be lazy to credit Weng and Wang with inventing all this. The lineage is clear: Voyager (G. Wang et al., 2023) introduced the ever-growing executable skill library in Minecraft; Reflexion (Shinn et al., 2023) introduced verbal reinforcement learning with text-based episodic memory; Self-Refine (Madaan et al., 2023) demonstrated iterative refinement with single-model feedback; Eureka (Ma et al., 2023) had GPT-4 evolve reward functions for IsaacGym; FunSearch (Romera-Paredes et al., 2023) combined a frozen LLM mutation operator with an evaluator to evolve Python heuristics, finding new bin-packing rules and a cap-set construction at n = 8. What Weng and Wang add is not the loop. It is the explicit claim that the loop, not the model, is the unit of learning, and that this is a paradigm rather than a clever hack on top of the current one.

Scoring HL Against Domingos’ Three Criteria

Criterion 1: Unification of the five tribes

This is the criterion HL satisfies most spectacularly, and the reason it deserves the master-algorithm conversation at all. The runtime architecture pulls in:

  • Connectionist core. The frozen LLM is a deep neural network. Function approximation, distributed representation, gradient-trained perception — all there, in the substrate.
  • Symbolist outer layer. The learned object is symbolic: Python code, markdown skill files, decision rules, [[wiki-links]]. This is the artefact that grows, not the LLM. The thing being updated is exactly what a Symbolist would want updated.
  • Evolutionary loop. Karpathy’s AutoResearch is literally a ratchet (mutate → evaluate → keep if better). FunSearch is an island-model evolutionary algorithm. Memento-Skills’ failure-attribution-then-rewrite is a directed mutation operator. Weng’s coding-agent edits are guided mutations.
  • Analogizer retrieval. Memento’s case-based reasoning is literally CBR; the M-MDP formalism and the trained retrieval policy μ are pure Analogizer machinery. Memento 2’s Read step is k-nearest-neighbour over experience.
  • Bayesian flavour, weakly. The retrieval policy μ is in effect a posterior over which experiences are relevant; the LLM’s next-token distribution is a learned conditional. The Bayesians get the least obvious representation in HL, but they are not absent. (One might charitably say they are operating under deep cover.)

Domingos’ own preferred candidate, Markov Logic Networks, attempted to unify Symbolists and Bayesians, with hooks to the other tribes. His more recent Tensor Logic tries to unify Symbolists and Connectionists in a single differentiable formalism. HL takes a strikingly different path: rather than seeking one mathematical object that contains all five tribes, it gives each tribe a layer in a runtime system. By the unification criterion alone, HL is the most complete unifier yet proposed.

Verdict on Criterion 1: passes, possibly definitively.

Criterion 2: Learning from data without per-domain engineering

This is where the picture becomes more nuanced.

In Weng’s blog, the coding agent receives only the environment interface and a reward signal, and writes the entire policy. There is no domain-specific feature engineering, no per-game heuristic library hand-coded by humans. In Memento-Skills, the agent starts from 5 atomic skills and discovers 230 more on its own. In AutoResearch, Karpathy hands the agent train.py and a metric; the agent does the rest. In LLM-wiki, the agent reads raw documents and produces structured knowledge with no human-written extraction rules.

So at the level of the HL system, the criterion looks well satisfied: across radically different domains (Atari, MuJoCo, GAIA, HLE, ML research, document corpora), the same architecture works with the same affordances.

There is, however, a subtler form of “per-domain engineering” hiding in plain sight. The LLM itself was pretrained on enormous quantities of human-engineered text. The agent does not start from a blank slate; it starts from GPT-5.4 or Claude or Qwen. Is that a violation of Criterion 2?

I do not think it is, and here is why. Domingos’ criterion was about the learner, not the substrate. Humans bootstrap on language, culture, and several million years of evolved priors; this does not disqualify human learning from being general. AlphaZero bootstraps on the rules of chess. Every learner sits on some substrate. What Criterion 2 forbids is per-task hand-engineering — and HL avoids that. The pretrained LLM is the substrate, like the visual cortex; the HL loop is the learner.

The harder question — whether HL can ever exceed what is latent in its substrate — I shall defer to Criterion 3. For Criterion 2 as Domingos stated it, HL passes.

Verdict on Criterion 2: passes, with the caveat that “no per-domain engineering” is read as “no per-task engineering on top of a general substrate.”

Criterion 3: Knowledge of arbitrary complexity

The naïve worry is that HL is “parasitic on the LLM” and therefore cannot represent or acquire anything the LLM does not already contain. This conflates bootstrapping with ceiling.

The learned artefact in HL is Turing-complete code (Weng, AutoResearch, Memento-Skills’ SKILL.md files include executable Python) or Turing-completeness-equivalent symbolic structures (LLM-wiki with backlinks and rules; Memento’s growing case base). There is no upper bound on the size or complexity of these artefacts. Memento-Skills has already demonstrated growth to 235 skills; nothing in the architecture stops it from growing to 235,000, except patience and electricity. The artefact’s representational ceiling is the Turing-computable functions — exactly the ceiling Domingos imagines for his master algorithm.

So in principle, Criterion 3 is satisfied. HL passes the formal test.

But three substantive worries sit just below the formal test, and these are the real reasons to hesitate before declaring victory:

  1. Discovery versus recomposition. Can HL discover new knowledge — knowledge not already latent in the LLM — or is it limited to recomposing existing LLM priors? The honest empirical answer is partially. FunSearch found mathematical objects (cap-set at n = 8, new bin-packing heuristics) that were genuinely novel and not present verbatim in the training data, via the LLM-mutation-plus-evaluator pattern. That is genuine discovery, and it counts. But Memento-Skills’ 235 skills are mostly recompositions of LLM priors into task-specific scaffolds, and Karpathy’s AutoResearch found incremental engineering wins rather than scientific surprises. The weak form of Criterion 3 — that the learner can in principle acquire any Turing-computable function from sufficient data — is satisfied. The strong form — that the same is true in practice, the version in which you would trust HL to discover the next AlphaGo move 37 — remains unproven. Reasonable people can disagree about whether this constitutes a Criterion 3 failure or a separate efficiency concern.

  2. Compute economics. Turing-completeness in principle is not tractable scaling in practice. Every edit costs a frontier-model inference. The compute curves in Weng’s paper are environment-step curves, not total-FLOP curves. If HL can in principle represent arbitrary complexity but in practice cannot pay for the search, the master-algorithm claim is weaker than it sounds. Domingos did not list efficiency as a criterion, but he plainly meant the master algorithm to be useful — not merely existent in some Platonic sense, like the answer to chess.

  3. One algorithm or one system? The deepest reading question. In The Master Algorithm, Domingos frequently writes as if he is looking for a single algorithm in the strict sense — one learning rule, one mathematical object, one training loop. HL is not that. It is a system with layered components, each from a different tribe. If the master-algorithm question is “what is the single learning rule?”, HL is disqualified because it has at least four (gradient descent in the LLM substrate, KL-regularised Bellman backup in the retrieval policy, evolutionary ratcheting at the artefact level, code-editing at the policy level). If the question is “what is the architecture that learns generally?”, HL is the strongest answer. I lean toward the latter reading, but a Domingos purist could reasonably push back.

Verdict on Criterion 3: passes the formal test (Turing-complete artefacts grow without bound). The deeper questions — discovery versus recomposition, compute economics, one-algorithm versus one-system — are open, and they matter for whether HL is the master algorithm or merely a candidate.

The Honest Verdict

By Domingos’ own criteria, Heuristic Learning has a stronger claim than:

  • Markov Logic Networks (passes Criterion 1 partially — Symbolists + Bayesians — but never delivered Criterion 3 in practice).
  • Deep Neural Networks (pass Criterion 3 spectacularly, fail Criterion 1: pure Connectionism is not unification, however much its enthusiasts wish it were).
  • Genetic Programming (passes Criterion 1 partially and Criterion 3 in principle, fails Criterion 2 on most domains without enormous compute).
  • Bayesian Networks (pass Criterion 2 elegantly on structured data, fail Criterion 3 in practice owing to inference intractability).

HL is the first candidate to pass all three plausibly. That is a serious claim and I am not making it lightly. The reason it can pass all three is precisely that it gives up on finding one algorithm and accepts that a master architecture, with components from each tribe, is what unification actually looks like.

Whether you call that the master algorithm depends on your reading of Domingos. The strict reading — one mathematical object, one learning rule — says no. The functional reading — one general-purpose system that learns from data with no per-task engineering and represents knowledge of arbitrary complexity — says yes.

My honest position: HL is the master algorithm if you accept the system-architecture reading, and the closest thing we have ever had to it under any reading. The objections I find genuinely troubling are not the formal-criteria ones but the two empirical ones. First, we have not yet seen HL discover knowledge that meaningfully exceeds its LLM substrate outside narrow domains like FunSearch. Second, the compute economics are unresolved. Those are the questions the next eighteen months will answer. For the first time in a decade, somebody might actually have to put up or shut up — Domingos very much included.

Where HL Sits Among the Classical Rule Learners

For the article to be even-handed, it is worth being concrete about what HL gains and loses relative to classical Symbolist methods.

Decision Trees (CART, C4.5, ID3), Random Forests, XGBoost, LightGBM. Extraordinarily sample-efficient, interpretable, well-understood theoretically. They dominate Kaggle on tabular data and will continue to do so long after the current enthusiasm has subsided. They cannot integrate world knowledge or operate on raw perceptual or textual inputs. HL inherits knowledge and language but loses almost all the formal guarantees.

Genetic Programming and Inductive Logic Programming. GP’s whole game is evolving symbolic programs — exactly what FunSearch does at scale. ILP (see Cropper & Dumančić, 2022) learns first-order logic rules with strong inductive bias and tiny data requirements, but fails catastrophically on noisy real-world inputs. Differentiable ILP (NLIL, GLIDR, LNNs) tries to fix this. HL is a complementary path: instead of differentiable rules, use natural-language-mediated rule editing. Less rigorous, considerably more flexible, and — for now — considerably more expensive.

Rule extraction from neural networks. A long literature (TREPAN, DeepRED, and others) attempts to distil rules from trained networks. HL inverts the problem: it generates rules natively in code form, with the LLM as a stand-in for the symbolic search engine.

Side by side:

Axis Classical rule learning LLM-empowered HL
Interpretability High High (code is readable)
Sample efficiency Excellent on tabular Excellent given priors
Generalisation Often poor OOD Inherits LLM priors
Compositionality Strong for ILP/GP Strong via skill libraries
Formal guarantees Sometimes (PAC-style) Mostly absent
Compute cost Tiny Frontier LLM at every step

Recommendations

What follows is for practitioners. I write it from the perspective of someone building a company in this space, which is to say with all the predictable biases. Caveat lector.

Stage 1 — now. If your problem has a clear scoring function and an LLM can plausibly write code that addresses it, run a Weng-style or AutoResearch-style ratchet loop overnight. Constraint: keep the artefact small enough that the agent can re-read it in one context window. Threshold to escalate: if a 100-experiment overnight run lifts your metric by more than 5% over your current baseline.

Stage 2 — next quarter. Build a Memento-style case bank for any agent you deploy in production. The 4.7–9.6 absolute-point lift on OOD tasks reported in the original Memento paper is the kind of result you can usually reproduce in-house if your task has any task-to-task structure at all. Threshold to abandon: if cases never get retrieved more than once, your task is too narrow and the overhead is not paying for itself.

Stage 3 — when frontier coding models cross the next threshold. Move from per-task program-evolution to a Memento-Skills-style cross-task skill library. The benchmark to watch is HLE. When an open-source agent with a frozen base model crosses 60% via skill accumulation alone, it is time to rebuild your stack around skill memory as a first-class primitive, rather than as an afterthought one bolts on at the end of the sprint.

Stage 4 — research bet. The unsolved problem is the HL ↔︎ NN bridge: how do you periodically distil what the skill library has learned back into the model weights without destabilising the LLM? This is the post-training problem Weng explicitly defers, and it is also the most direct route to resolving the “discovery versus recomposition” worry under Criterion 3. Anyone who solves this gets the actual prize, and they will not have to share it with the Bayesians.

Decision rules. Use HL when interpretability and per-edit sample efficiency matter, and when your action space is naturally programmatic. Use Deep RL when perception dominates, when you have cheap simulation, or when the policy is fundamentally continuous and non-symbolic. Use both when you can — this is the System-1 / System-2 hybrid most of us will eventually ship, whether we plan to or not.

Caveats

  • Mixed source types. Several of the central sources are blog posts or GitHub artefacts, not peer-reviewed papers. Treat these as primary evidence of where leading practitioners’ thinking is heading, not as settled science. The peer reviewers will, in their own good time, catch up.
  • Theory versus empirics. Memento 2’s convergence theorems hold under bounded-reward and γ < 1 assumptions and require the memory coverage radius rₘ to shrink. The constants εₗₗₘ and δₘ are not bounded analytically. The theory is a useful organising framework, not a guarantee, and it should not be confused for one.
  • Benchmark inflation. GAIA and HLE are saturating quickly. Memento’s 87.88% on GAIA validation is impressive, but baseline comparisons in some ablations use Qwen2.5-7B while Memento itself uses GPT-4.1 plus o4-mini, which is rather like racing a Ferrari 812 Superfast against a Ford Fiesta and being surprised at the outcome.
  • The Domingos framing is mine, not Domingos’. Domingos has not commented on HL specifically. His public position is sceptical of LLMs: interpretability matters more than scale, LLMs lack true understanding, regulate applications and not algorithms. He would, I strongly suspect, not call HL the master algorithm. This article is an argument that he ought to.
  • I have, as the disclosure forms put it, an interest. Bloo-Mind builds agentic systems and we use techniques in this family. Treat my enthusiasm with appropriate suspicion. The honest test of any of these ideas is whether they survive industrial deployment over the next eighteen months. We do not yet have that data, and anyone who tells you otherwise is, in the precise technical sense, selling something.

References

Cropper, A., & Dumančić, S. (2022). Inductive logic programming at 30: A new introduction. Journal of Artificial Intelligence Research. https://arxiv.org/abs/2008.07912
Domingos, P. (2015). The Master Algorithm: How the Quest for the Ultimate Learning Machine Will Remake Our World. Basic Books. https://www.basicbooks.com/titles/pedro-domingos/the-master-algorithm/9780465065707/
Karpathy, A. (2026a). AutoResearch. GitHub repository. https://github.com/karpathy/autoresearch
Karpathy, A. (2026b). LLM-wiki. GitHub gist. https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f
Ma, Y. J., Liang, W., Wang, G., Huang, D.-A., Bastani, O., Jayaraman, D., Zhu, Y., Fan, L., & Anandkumar, A. (2023). Eureka: Human-Level Reward Design via Coding Large Language Models. https://arxiv.org/abs/2310.12931
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., Gupta, S., Majumder, B. P., Hermann, K., Welleck, S., Yazdanbakhsh, A., & Clark, P. (2023). Self-Refine: Iterative Refinement with Self-Feedback. Advances in Neural Information Processing Systems (NeurIPS). https://arxiv.org/abs/2303.17651
Romera-Paredes, B., Barekatain, M., Novikov, A., Balog, M., Kumar, M. P., Dupont, E., Ruiz, F. J. R., Ellenberg, J. S., Wang, P., Fawzi, O., Kohli, P., & Fawzi, A. (2023). Mathematical discoveries from program search with large language models. Nature. https://doi.org/10.1038/s41586-023-06924-6
Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., & Yao, S. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. Advances in Neural Information Processing Systems (NeurIPS). https://arxiv.org/abs/2303.11366
Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., & Anandkumar, A. (2023). Voyager: An Open-Ended Embodied Agent with Large Language Models. https://arxiv.org/abs/2305.16291
Wang, J. (2025). Memento 2: Learning by Stateful Reflective Memory. https://arxiv.org/abs/2512.22716
Weng, J. (2026). Learning Beyond Gradients. https://trinkle23897.github.io/learning-beyond-gradients/
Zhou, H. et al. (2025). Memento: Fine-tuning LLM Agents without Fine-tuning LLMs (AgentFly). https://arxiv.org/abs/2508.16153
Zhou, H. et al. (2026). Memento-Skills: Let Agents Design Agents. https://arxiv.org/abs/2603.18743
]]> continual learning https://essays.bloo-mind.ai/posts/2026-05-12-master-alg/ Mon, 11 May 2026 23:00:00 GMT Hello, World! Dell Zhang https://essays.bloo-mind.ai/posts/2026-05-11-hello/ This is a test post.

Equations work: .

And display math too:

Code chunks execute:

import numpy as np
np.random.seed(42)
samples = np.random.randn(1000)
print(f"mean={samples.mean():.4f}, std={samples.std():.4f}")
mean=0.0193, std=0.9787
]]> https://essays.bloo-mind.ai/posts/2026-05-11-hello/ Sun, 10 May 2026 23:00:00 GMT