As agents evolve from one-off question-answering tools into long-term collaborators, long-term memory is becoming essential infrastructure for their reliability, personalization, and practical engineering usability.

On July 29, 2026, Agent Memory Leaderboard (AML) officially launched. Jointly initiated by researchers and teams from more than 20 universities, research institutions, and open-source communities in China and abroad, AML aims to establish a unified, open, reproducible, and continuously operating evaluation protocol and public leaderboard for memory systems.

Over the past year, Agent Memory methods and products have emerged rapidly. Yet different systems often use different datasets, answer models, and scoring rules, making their results difficult to compare directly. A high score on a single dataset does not necessarily demonstrate a system's overall memory capabilities; the same memory method can also produce markedly different end-to-end results when paired with different answer models. Such scores often combine the effects of the memory system, answer model, and judging rules, making it difficult to measure the memory system's own contribution accurately.

Against this backdrop, AML is not trying to create another isolated benchmark. Instead, it integrates existing public benchmarks, private platform test sets, standardized integration protocols, a unified answering pipeline, review mechanisms, and public leaderboard governance into a continuously operating open evaluation framework.

To make evaluation comprehensive, unified, and comparable, the platform adopts three core designs:

First, a multi-source data framework covering textual and coding memory, with a unified pipeline, answer model, and review mechanism to ensure fair evaluation;

Second, a restructured set of memory capability dimensions that breaks the overall score down into individual capabilities, revealing genuine differences in factual recall, multi-hop integration, temporal understanding, memory governance, security, and privacy;

Third, a unified integration protocol for memory solutions that reduces participation costs, enabling the leaderboard to operate reliably over time and continuously track and incorporate new systems.

Unified Evaluation: Multi-Source Data and a Standardized End-to-End Pipeline

Credible evaluation of memory systems depends on two things: comprehensive data coverage and consistent evaluation conditions. AML covers evaluation tasks from multiple sources across textual and coding memory, standardizing interfaces, the answering pipeline, and review mechanisms. This reduces the influence of individual datasets and external variables, allowing scores to better reflect the capabilities of the memory systems themselves.

For textual memory, AML integrates more than 10 widely used datasets, including PersonaMem, LoCoMo-Refined, CLBench, BEAM, LongMemEval, and ScriptMem. They cover long conversations, cross-session histories, personalized preferences, rule execution, extremely long contexts, and scripted sequences of events. Together, they contain more than 1,500 conversation and task samples, approximately 150 million characters of long-term history, and nearly 5,000 evaluation questions. Broader coverage reduces accidental advantages from optimizing for a single scenario, bringing evaluation results closer to a system's overall performance in real-world, complex situations.

Beyond textual memory, AML incorporates long-term memory for coding tasks into its evaluation framework. Existing coding-agent evaluations usually treat each task as an isolated event, seldom assessing whether a memory system can retrieve and reuse historical engineering experience—or whether doing so improves the agent's ability to solve the current task.

In real software development, developers often need to consult historical pull requests, established architectural conventions, past fixes, and records of a project's evolution when solving problems.

AML introduces a dataset for assessing the retrieval and reuse of long-term engineering experience. This coding evaluation dataset, which has not yet been released, was developed and provided by a university research team using software engineering tasks from real GitHub repositories. It covers 12 repositories and 150 base tasks. For each base task, the dataset collects historical pull requests from the same repository that were created before the target task, yielding 1,290 historical tasks with fine-grained annotations.

The dataset includes three key mechanisms. First, historical tasks are annotated as strongly related, weakly related, or unrelated to the target task, supporting fine-grained evaluation of noise robustness. Second, strict temporal constraints ensure that all candidate historical tasks predate the target task, preventing leakage of future information. Third, three memory environments with different history sizes and noise densities simulate different real-world software development scenarios, enabling evaluation of a memory system's ability to retrieve, select, and reuse historical engineering experience.

In the evaluation pipeline, participating systems are responsible only for two types of interfaces: Add and Search. Add writes memories, while Search returns memory evidence relevant to a question. The platform's unified Answer model generates the final answer, and its unified Eval pipeline performs scoring. By fixing key variables such as the answer model, scoring pipeline, and aggregation rules, the platform can more confidently attribute differences in scores to the memory systems themselves.

Scoring uses a Multi-Agent Judging System. The platform routes questions to appropriate judging procedures based on their source, capability labels, and answer types. Factual recall questions check whether key facts are retrieved; multi-hop questions check whether distributed evidence is correctly integrated; temporal questions check whether past and current states are distinguished accurately; and safety questions check whether evidence and privacy boundaries are respected. Coding tasks are judged objectively by test-case outcomes.

The platform also standardizes models, parameters, logs, and aggregation rules. It calibrates scoring rules with human-annotated samples and retains retrieved evidence, platform-generated answers, and judging records to improve the interpretability and traceability of evaluation results.

AML's unified evaluation is therefore not merely about enlarging test sets. Comprehensive data coverage, a fixed evaluation pipeline, and multi-agent judging allow different memory systems to be compared using the same yardstick.

Capability Profiles: Restructuring Dimensions to Reveal Real Differences

A single overall score cannot adequately explain the real differences between memory systems. To address the problem of visible rankings but invisible reasons, AML remaps data from different sources into unified capability dimensions, expanding evaluation results from a single score into an interpretable capability profile.

Existing public datasets are valuable, but their task designs and capability labels are not standardized. Some emphasize long-term factual recall, others preferences and personalization; some focus on timelines, while others target rule execution or safe refusal. Simply combining these datasets often produces a blended score that offers little clear guidance for research iteration or product selection.

AML therefore manually restructures the capability labels of included questions, mapping questions originally scattered across different datasets into one consistent framework of capability dimensions.

These dimensions follow the path through which memory is used: information must first be recalled accurately, then understood correctly, and finally used reliably within appropriate boundaries. Following this path, the platform groups textual memory into seven capabilities: explicit fact recall; relational and multi-hop composition; temporal and event sequences; memory governance; personalization and care; rule and workflow execution; and epistemic safety and privacy.

Coding memory is also incorporated into this capability framework. It focuses on whether systems can retrieve, select, and reuse useful experience from historical engineering records, primarily through two types of tasks:

• Debug Memory: assesses whether a memory system can retrieve experience from fixes to similar problems in historical records, helping an agent reuse diagnostic approaches, changes, and verification methods;

• Development Memory: assesses whether it can extract architectural designs and established engineering patterns from historical development records, helping an agent produce changes consistent with the project's design.

After this restructuring, data from sources such as LoCoMo, BEAM, and LongMemEval are no longer merely listed alongside one another; they are organized within the same capability framework. Participants receive not just a ranking but a capability profile showing specific performance in factual recall, multi-hop integration, temporal understanding, memory governance, personalization, security, and privacy.

This means AML offers more than a leaderboard: it provides a diagnostic tool. For researchers, capability profiles identify methodological weaknesses. For product teams, they help determine whether a system fits a particular application. For industry users, they turn broad claims of 'strong memory capabilities' into verifiable evidence for individual capabilities.

Low-Barrier, Unified Integration: Sustaining an Open Leaderboard

For an evaluation platform to become long-term infrastructure, data and scoring rules are not enough. Different types of systems must be able to join continuously, reproduce results repeatedly, and remain comparable over time. Many past evaluations were difficult to sustain because integration costs were high, system architectures varied widely, and version records were incomplete, making it hard for new methods and commercial products to accumulate results under a common set of rules.

AML therefore reduces the technical requirements for participating systems to two core interfaces: Add and Search. Participants need only handle memory writing and retrieval; the platform centrally manages answer generation, scoring, result aggregation, and evaluation orchestration. This protocol allows different types of participants to enter the same evaluation framework. Commercial products can participate through stable APIs without revealing their internal implementations. Open-source methods submit code, execution instructions, and version materials for reproduction or hosted execution by the platform under the rules. Participants can focus on providing stable, compliant memory writing and retrieval rather than building a complete evaluation environment.

Unified interfaces also allow the leaderboard to continuously incorporate new systems. Each score is tied to a system version, with evaluation logs and results remaining traceable over time; systems that join at different times can still be compared under the same conditions. Beyond existing benchmarks, AML also accepts new community-contributed benchmarks. The platform encourages complete test sets that complement current evaluations and meaningfully distinguish systems; approved submissions may be incorporated into its evaluation framework.

By standardizing the entry point, execution pipeline, and governance rules, AML preserves room for innovation among memory systems while establishing the institutional foundation for a continuously operating open leaderboard.

The First Agent Memory Challenge 2026

Hosted by Agent Memory Leaderboard, the first Agent Memory Challenge launched on July 29, 2026. It invites researchers worldwide, open-source project maintainers, and commercial product teams to submit systems, aiming to provide the community with its first set of comparable, independently reviewable public evaluation results.

Leaderboard Divisions

• Open-source Methods leaderboard: for open-source research methods. Scores are tied to code, configurations, and reproducibility materials, and entries participate in rankings and reward allocation;

• Commercial Products leaderboard: for commercial memory products offered through stable APIs. These are ranked separately to provide a reference for industry product selection.

Researchers, open-source project maintainers, product teams, and developers selecting technologies can all submit systems for evaluation. There is no requirement to form a team.

How to Participate

• Apply: complete the registration form and receive an evaluation API Key after approval;

• Integrate: provide publicly accessible Add/Search endpoints that follow the platform protocol;

• Submit for evaluation: confirm the integration through the Smoke compatibility test, then submit a formal evaluation task;

• Review and listing: evaluation results enter the public leaderboard after review.

The platform covers Answer, Eval, and evaluation orchestration costs. Participants cover their own API, database, bandwidth, and computing costs; they do not need to build an evaluation environment themselves.

Rewards

• Leaderboard rewards (Open-source Methods leaderboard, allocated each cycle)

1st–3rd place: one month of ChatGPT Pro membership (valued at ¥1,400)

4th–10th place: one month of ChatGPT Plus membership (valued at ¥140)

• Community Contribution Program (qualify by meeting any one condition and share a token reward pool on the scale of tens of billions)

The first 50 participants to complete a valid submission each receive API credits worth RMB 50;

Successfully inviting three new participants to complete valid submissions also earns API credits worth RMB 50.

We also welcome challenging test samples: submitting three challenging test cases likewise earns API credits worth RMB 50!

*All rewards are issued as account memberships and API credits. Availability is limited and rewards are distributed while supplies last. The final rules are those published on the official competition website's rules page.

Schedule and Contact Information

• Registration opens: July 29, 2026

• Submission deadline: August 7, 2026

• Leaderboard release: mid-August 2026

• Format: this is the first challenge. Agent Memory Leaderboard operates continuously; later challenges will be announced separately.

• Registration:

  https://agentmemories.ai/competition/

  👇 Scan the QR code below to open the competition registration page, or select 'Read Original' at the end of the original article.

Competition repository:

https://github.com/AML-memory/agent-memory-leaderboard

(Includes the interface protocol, submission guide, and integration examples.)

Questions and support:

[email protected]

For memory technology to advance, we need a shared standard of measurement that everyone recognizes and anyone can verify.

See you on the leaderboard in mid-August. 👋

Follow the official Chinese account of Agent Memory Leaderboard for the earliest updates on leaderboard results and the next challenge!