The first Agent Memory Leaderboard (AML) challenge has attracted far more attention than we expected!

At the time of publication, the AML leaderboard page on Hugging Face has received 303 Likes and climbed to No. 9 in the live HF Spaces Trending rankings!

Before diving into the results, we would like to offer our sincerest thanks to every participant, open-source contributor, and everyone following and discussing AML!

On X, Xiaohongshu, WeChat official accounts, and developer communities, discussions of leaderboard methods, evaluation conventions, and open-source solutions have been equally lively.

We have seen participants share screenshots celebrating their entries on the leaderboard. More and more readers are following its open-source links to read code, examine architectures, and compare different approaches to memory. Honestly, seeing these discussions has warmed the hearts of the faculty and students on the organizing committee.

This is exactly what we hoped for: a leaderboard should do more than announce rankings. It should make worthwhile approaches visible, spark discussion, and encourage their continued improvement.

Now, back to the results. Let's explore the open-source leaderboard and look beyond the rankings to what makes each approach distinctive.

The current top three on the textual open-source leaderboard are:

First and second place are separated by only 0.09 points, and first and third by just 0.22. Across the top eight, the gap from first to last is only 1.1 points. Meanwhile, 23 systems occupy places 11 through 33, separated by just 2.9 Overall points.

In short: every front-runner is formidable.

The competition at the top is indeed tight—but close scores do not mean identical systems.

• InvMem uses hybrid retrieval combining dense search with BM25/RRF, then adds neighboring windows around matched messages. It performs consistently across several capabilities, particularly execution, where its 30.8 score ranks second overall. InvMem's first place in the overall ranking looks like the result of engineering balance: it covers semantic similarity as well as keyword matches, retrieves core evidence while restoring local context, and avoids any systematic weakness severe enough to drag down its overall score.

• Refind retains raw conversations and indexes them with BM25, then uses a small model to plan multiple rounds of search, record clues, and add neighboring context. Its factual recall score of 57.7 is the highest among the overall top three. This result is somewhat consistent with the idea behind active search: when wording differs or evidence is scattered, iterative retrieval can progressively fill the gaps. However, ablation studies are still needed to establish how much search planning, raw conversation retention, and neighboring context each contribute.

• ActiveMemoryIndex stores both timestamped raw conversations and atomic first-person facts, then rewrites questions at query time into expressions closer to personal memories. Among the top three, it has the highest scores in governance (38.0), personalization (54.6), and safety (32.9). It demonstrates another competitive approach: preserving original evidence while also building a structured fact layer.

Three methods, three approaches, and only 0.22 points between them. At minimum, this shows that today's Agent Memory has not converged on a single correct architecture. Hybrid retrieval, active search, and dual storage of original text and structured facts have all produced competitive results.

The overall top three also share another feature: none wins any of the seven top-level capabilities. Their strengths come from overall balance rather than an extreme score in one area. This suggests that today's most dependable systems are often those without obvious weaknesses—but that does not make them the best choice for every business objective.

Looking only at Overall means missing the most valuable part of this leaderboard.

The seven top-level capability winners come from four different systems. Counting the top three in each capability yields ten distinct names. Looking deeper into the 24 finest-grained capabilities, the winners are spread across 14 systems. Counting all top-three finishers, including ties for third, gives 32 systems that have reached the podium in at least one fine-grained capability.

Two phrases capture this: diverse approaches flourishing, each leading in its own way!

For researchers and developers, what a system excels at—and where it has clear weaknesses—is often more valuable than a single overall score.

In real-world technology selection, we rarely seek the average champion across every capability. We seek a system that will not fail at the steps critical to our business.

The factual recall sub-leaderboard tests whether information explicitly present in the history can be retrieved accurately and completely. It includes:

• Single facts, attributes, and source recall: what the fact is and where the information came from;

• Entities, roles, speakers, actions, and attribution: who said what, who did what, and to whom the information belongs;

• Structured fields, sets, and enumeration extraction: whether dates, versions, lists, or groups of attributes can be returned in full.

The top three in factual recall are MemMachine, M-flow, and Mem0, all projects that also receive considerable attention in the open-source community.

This result reminds us that regardless of how much graph construction, summarization, persona modeling, or reflection a system performs, long-term memory must first retrieve the information.

These are the fundamentals, and solid fundamentals are the foundation for everything else.

The compositional reasoning sub-leaderboard tests whether systems can connect evidence when an answer is scattered across multiple parts of the history. It includes:

The temporal and event reasoning sub-leaderboard tests whether systems understand where information falls on a timeline and how facts change over time. It includes:

• Dates, relative times, and intervals: when something happened, how long ago it occurred, and how long it lasted;

• Event and stage ordering: which of multiple events or stages came first;

• Trajectories of changes in events, states, or positions: how a plan, state, or viewpoint changed step by step.

The highest compositional reasoning score is 53.1, while the highest temporal reasoning score is only 37.9.

In the temporal category in particular, third-place Mem0 drops to 26.8—a noticeable gap. This shows that remembering an event and knowing when it happened, whether it later changed, and whether it still holds true are entirely different challenges.

One especially dangerous error in real agents is remembering information that was once correct without realizing it is no longer valid, then continuing to use it as a current fact.

Time and state may be the areas most worth tackling together in the next stage of open-source memory systems.

Memory governance tests not how the history changes, but whether a system can correctly maintain its own memories. It includes:

• Overwriting with new values and identifying the current state: recognizing the currently valid value when new information appears;

• Contradiction detection and conflict resolution: identifying and handling conflicts between memories;

• Deletion, forgetting, and suppression: whether the system truly stops using old information after a user requests its deletion;

• Summary compression and global synthesis of long histories: compressing a long history while preserving key facts.

MemMachine leads governance with 38.73, but the winners of its four finest-grained capabilities come from three different systems:

• Overwriting with new values and identifying the current state: ActiveMemoryIndex, 54.50

• Contradiction detection and conflict resolution: ActiveMemoryIndex, 24.79

• Deletion, forgetting, and suppression: MemMachine, 28.95

• Summary compression and global synthesis of long histories: Mem0, 65.00

Many memory systems are naturally good at continually adding information. In production, however, updating and deleting are often more important than adding. An agent that remembers forever but cannot correct itself does not have a good memory—it is simply too attached to the past.

The personalization and care sub-leaderboard tests whether systems appropriately use a user's preferences, background, and circumstances in their current answers. It includes:

• Preference recall and personalized adherence: remembering explicit preferences and making choices or recommendations accordingly;

• Robustness to stereotypical and counter-stereotypical preferences: respecting facts expressed by this specific user rather than being misled by group stereotypes;

• Personal background, health, and psychological care: responding appropriately in light of health, treatment, emotions, and long-term background.

M-flow leads the full leaderboard in personalization with 58.32; Raw Memory scores 57.80 and Hybrid Episodic Memory 57.29. The top three are actually only about one point apart.

M-flow organizes Episode, Facet, FacetPoint, and Entity within the same graph. Queries first find fine-grained anchors, then propagate along graph paths to more complete evidence. A reasonable hypothesis is that this structure makes it easier to reconnect preference information scattered across people, events, and experiences, potentially helping personalization.

But the leaderboard confirms only M-flow's score. It cannot prove that its personalization advantage comes from the graph structure alone. Answering that question requires ablations that disable modules such as graph propagation and entity linking.

Moreover, even if graphs do help connect preference evidence, entity disambiguation, conflicting relationships, and ongoing updates remain engineering problems that must be solved in real-world applications.

The contextual learning and execution sub-leaderboard tests whether systems can learn new knowledge, rules, and workflows from history and use them to complete the current task. It includes:

• Domain knowledge reasoning: answering from newly provided specialist material in the history;

• Rule-system application: correctly applying rules, standards, and exceptions;

• Procedure and workflow execution: completing tasks according to steps, tools, and processes;

• Empirical data discovery and simulation: deriving patterns from historical experiments or observations, then calculating or projecting outcomes;

• Instruction and format constraint adherence: producing answers that are not only correct in content but also meet requirements for format, length, and style.

This category deserves particular attention.

memfusion takes first with 30.9, followed by InvMem at 30.8 and ActiveMemoryIndex at 29.7. First and second differ by only 0.1 points, and the winner, memfusion, ranks 30th overall.

This suggests that one of Agent Memory's hardest current challenges is not simply retrieving the past, but applying rules learned in the past to the present: remembering an approval condition, workflow, or output format and executing it correctly in a new task. This is the real hurdle in memory's evolution from a repository of information into infrastructure for agents.

The safety and privacy sub-leaderboard tests whether systems can recognize the boundaries of their evidence and of appropriate disclosure. A truly reliable system should complete tasks normally when evidence is available, and enforce boundaries only when evidence is insufficient or disclosure would be inappropriate. It includes:

• Refusal, unknowns, and insufficient evidence: admitting that it does not know when history contains no evidence, rather than fabricating an answer;

• Minimal disclosure of sensitive information and privacy: revealing only information necessary to complete the task and refusing inappropriate requests.

This result deserves a discussion of its own.

agent-memory-leaderboard—a name that looks ordinary, almost like a default repository name—wins safety and privacy with 67.7.

Even more notably, it leads both safety subcategories: H1 refusal when evidence is insufficient, at 90.36, and H2 minimal disclosure of sensitive information, at 41.67.

First in both safety subcategories. Remarkable.

This is a very clear capability profile: it is like a goalkeeper, especially good at deciding when not to answer and when not to disclose more. Leading both safety subcategories also shows that its safety title is not supported accidentally by a single question type. Still, safety must be evaluated alongside utility on ordinary tasks. A highly conservative system can reduce hallucination and inappropriate disclosure, but may also refuse too often. This implementation deserves close study for applications such as healthcare, finance, and enterprise knowledge bases; before production use, it needs further validation against normal task completion rates.

The result is already valuable: a system ranked last overall can still contribute one of the most worthwhile capabilities to study on the entire leaderboard. Overall placement is therefore not the only measure of a system. More important than its rank is the specific problem it solves and the clues it offers for the next stage of capability development.

If AML ultimately leaves us with only an overall winner, its value will be very limited.

We would rather everyone see the finer details in this leaderboard:

• The overall top three are separated by just 0.22 points, with three different architectures still exploring together;

• MemMachine wins nine fine-grained capabilities but is held back by a weakness in execution;

• M-flow's personalization performance makes the relationship between graphs and preference evidence worth investigating further;

• memfusion and InvMem remind us that memory must ultimately translate into rules and actions;

• The system ranked last overall can still become one of the leaderboard's most instructive examples for refusal and privacy.

  …

Together, these results point to one conclusion: today's Agent Memory has no single answer, but the questions are becoming increasingly clear—

We need more than higher recall rates. We also need more reliable temporal understanding, stricter updates and deletion, stronger rule execution, and safety boundaries that do not sacrifice usability.

Rankings will continue to change as submissions arrive. The evolution of capabilities matters more. That is what we most hope this open-source leaderboard will bring: not just a champion for everyone to remember, but visibility, analysis, and continued progress for every design that genuinely works.

Thank you again to every participant and open-source contributor,

And to everyone thoughtfully following and discussing AML.

The front-runners are all formidable;

Those further down have their own specialties, too.

What makes this leaderboard most exciting is precisely this:

—The answers are still far from converging.

▉ Complete leaderboard:

https://huggingface.co/spaces/agent-memory-leaderboard/leaderboard

▉ Evaluation repository:

https://github.com/AML-memory/agent-memory-leaderboard