As large-model agents move from single-turn interactions to long-term collaboration, long-term memory has become an important foundation for continuity, stability, and personalization. Different memory systems currently tend to use different datasets, answer models, and evaluation workflows, making published scores difficult to compare directly. Through a unified Add / Search interface contract, fixed evaluation protocols, and auditable execution workflows, the AML Challenge aims to build a more open, fair, and reproducible Agent Memory Benchmark. It provides researchers and developers with a comparable evaluation environment and promotes research exchange, engineering validation, and industrial applications of long-term memory technology.

The first AML Challenge leaderboard was officially released on August 12, 2026. Since its publication, AML has continued to attract attention from researchers, open-source communities, and industry teams, along with considerable positive feedback and constructive suggestions.

As of this article's publication, the AML website has received more than 300,000 visits. Since launch, AML's official page has consistently ranked among the top ten on the Hugging Face Spaces Trending chart, and recently held first place for several consecutive days. For an evaluation project focused on frontier research and engineering practice, these figures reflect not only the reach of Cycle 1 but also growing community interest in unified Agent Memory evaluation and widespread demand for public, comparable results.

Host: China Society of Image and Graphics (CSIG)

Organizers: Nanjing University, Zhejiang University, Datawhale, and the CSIG Working Committee on Enterprise Liaison and Standardization

Unified evaluation platform: Agent Memory Leaderboard (AML)

AML Steering Committee: Representatives of research teams from nearly 30 universities and research institutions, providing guidance and review on AML's long-term evaluation direction, evaluation methods, and leaderboard governance.

Starting at 00:00 on September 20, 2026, participants can visit the website's Evaluation page to submit registration information and apply for an Evaluation Key.

During the challenge, integration and evaluation will be conducted on a rolling basis in submission order.

There is no registration fee.

The challenge is open worldwide to universities, research institutions, open-source communities, independent research teams, and individuals. Individual entries, team entries, and teams formed across organizations are welcome. There are three tracks, evaluated and ranked separately: Textual, Coding, and Multimodal.

• Textual track: Evaluates precise factual recall, multi-hop logical connections, temporal sequence reasoning, personalized alignment with user profiles, and memory security boundaries in long-term, multi-turn dialogue.

• Coding track: Focuses on software engineering, evaluating a system's ability to retrieve debugging processes, development experience, and project context from historical tasks to support subsequent development and troubleshooting.

• Multimodal track: Evaluates memory for images, audio, video, and text, along with cross-modal associations, clue retrieval, and contextual continuity.

Each track has an Open-source Methods leaderboard and a Commercial Products leaderboard:

• Open-source Methods leaderboard: Provide a publicly verifiable GitHub repository, pin a commit or version, and describe the method's provenance, authors, and implementation details.

• Commercial Products leaderboard: Internal implementation need not be disclosed, but a fixed product version and stable, available Add / Search APIs are required.

A participant may enter multiple tracks but may select only one participant division.

Add: Receives evaluation memories and performs storage, organization, indexing, and updates.

Search: Returns relevant memories or evidence for a question; it does not generate the final answer.

The platform centrally handles task orchestration, Answer generation, Eval (automated evaluation), result review, and leaderboard publication. Public Smoke tests check interface structure, authentication, and the end-to-end workflow; Full evaluations determine official scores. Private questions, answers, and other protected materials are not disclosed to participants.

[Submission workflow]

1. Read the participation rules and API integration guide, then choose your track and participant division.

2. Freeze your submitted version and prepare system documentation, runtime environment details, and interface materials.

3. Submit registration information and apply for an Evaluation Key on the Evaluation page.

4. Configure your Add / Search interfaces and complete a Smoke test.

5. After confirming that your interfaces are stable and response structures comply with the contract, start an official Full evaluation.

6. View your private results on the Evaluation page.

7. After the platform reviews materials, versions, results, and compliance, eligible scores will be added to the public leaderboard.

Once a Full task is accepted, the submitted version is frozen. Code, images, APIs, authentication information, and capacity settings must remain consistent with the declared submission.

Under this cycle's arrangements, each participant may initiate at most 2 Full evaluations during the complete challenge cycle. After a task succeeds or is partially completed, the participant enters a 30-day cooldown in that track. Failed or canceled tasks are handled according to the page's instructions and the current cycle's quota policy. The final leaderboard score is based on the latest valid Full evaluation result.

[Interface Operation, Result Review, and Compliance Requirements]

After a participant initiates a Full evaluation on the Evaluation page, the platform will schedule resources and start evaluation within 48 hours. The 48-hour limit applies only to resource scheduling and evaluation startup; it does not mean the task will finish within 48 hours. From the start of resource scheduling until task processing is complete and any necessary follow-up review has ended, participants should keep their interfaces publicly accessible and maintain stable API contracts, authentication methods, and capacity settings. Overall runtime is typically about 2–10 days, depending on implementation complexity and evaluation scale.

In principle, participants may stop their memory management service once the Full task is complete and the platform has not requested further review. If result, version, or compliance review identifies an issue requiring additional verification, the platform will notify participants at their registered email address. Please monitor your email and Evaluation page status, restore interface services when requested, and provide runtime logs, version materials, or other review materials specified in the notice.

During evaluation, every user_id, task, and sample must be processed independently. Evaluation memories must not be shared across users, tasks, teams, or runs. Hardcoding, benchmark leakage, prompt injection, manual real-time answering, result manipulation, malicious leaderboard gaming, and undisclosed code reuse are strictly prohibited.

Evaluation data and derived copies may be used only for this evaluation task, not for training, fine-tuning, product analytics, dataset reconstruction, or external distribution.

The combined prize pool across the three tracks is RMB 150,000. Prizes are available only for the Open-source Methods leaderboards.

AML will continue to solicit Benchmark Contributions. Universities, research institutions, open-source teams, product teams, and community members are welcome to submit benchmarks, test scenarios, data, or suggestions for evaluation dimensions with clear provenance, appropriate difficulty, complementary value, and verifiability, as well as proposals to improve public evaluation contracts. Submissions must not contain private evaluation traces, participant data, access credentials, or internal information from commercial systems. Please preserve upstream author attribution and license information. Contact us through the AML website's Benchmark Contribution section or the email address below.

• Registration and Evaluation Key application:

  https://agentmemories.ai/evaluation

• Participation rules:

  https://agentmemories.ai/rules

• API integration guide:

  https://agentmemories.ai/api-guide

• Challenge repository:

  https://github.com/AML-memory/agent-memory-leaderboard

• Contact email:

  [email protected]

Please follow the AML website and its official WeChat account, “Memory Leaderboard,” for subsequent announcements about registration, interface contracts, evaluation arrangements, and score publication.

Eligible universities, research institutions, companies, open-source teams, and individual researchers are welcome to participate and help advance the development and evolution of the Agent Memory Benchmark.