Agent Memory Leaderboard
FROM THE AML TEAM

The AML Blog

Ideas, research, and updates from the team building Agent Memory Leaderboard.

A closer look at agent memory.
← All posts

AML Is in Full Swing: Participant FAQ, Part 2

Cycle 2 Announcements8 min read

Agent Memory Leaderboard


Cycle 2 of Agent Memory Leaderboard is in full swing! This cycle covers Coding, Multimodal, and Textual memory, and we welcome participation across all three tracks. We especially encourage teams to explore Coding and Multimodal, where the field is currently less crowded. Sign up and showcase your work! Drawing on recent participant emails about application confirmations, version registration, cross-track integration, and troubleshooting, we have compiled this supplementary FAQ to help teams address practical integration issues.




Application Confirmation and Email Communication







Q1: Do I need to submit a new application after receiving a request for additional information?


If you are supplementing an existing application, please reply in the original email thread. There is no need to resubmit the same request if the organizing committee can identify your application. State the system name and version, and address each requested item. A new application is needed only when the application itself changes or the committee explicitly asks you to reapply.





Q2: If I have already provided my application ID, do I need to include it in every reply?


If you continue in the original email thread and the committee has already identified your application, you do not need to repeat the same ID. When starting a new thread, discussing multiple systems, or dealing with multiple versions, please identify the relevant application to avoid mixing up different requests.





Q3: My application was approved, but I have not received the Key email. Should I reapply?


Please first check your spam folder, email filtering rules, and the email address entered in your application. Then contact the committee to check the delivery record for the original application.




New Versions and Cross-Track Integration







Q4: Is a version name alone sufficient to register a new version?


Along with the version name, please provide the corresponding fixed implementation, models, key dependencies, and a description of changes from the previous version. This information helps distinguish versions accurately and keeps the registration consistent with the implementation actually submitted.





Q5: Why is version registration still pending after I received written approval for an eligibility arrangement?


Eligibility confirmation and version registration are separate steps. After receiving written approval, check the requirements in the original email and confirm that the new version name, fixed implementation, and required materials are complete. Supply any missing information so registration can proceed. Continuing in the original thread also helps connect the registration with the eligibility arrangement already confirmed.





Q6: Can I use my existing Key to register a new version for another track?


First check whether the target track uses the same Add/Search URLs already bound to your Key. If the URLs are the same, you may register a new version under the existing Key and start evaluation after meeting the target track's Smoke and admission requirements. If the URLs differ, follow the endpoint-change process to apply for a Key, identifying the original system and application and the intended track so the requests can be linked. Registering a new version does not allow you to resume an old task with a new implementation.





Q7: Does applying for a new Key for new endpoints give me another Full quota?


An endpoint-change application is used to review and bind a new integration configuration. The evaluation quota for the same participant identity remains unchanged; it does not provide extra Full runs. Please plan integration and evaluation within your existing quota.





Q8: Will an existing task automatically switch to a newly registered version?


No. An accepted task remains bound to its original version and evaluation configuration. Check the arrangements for the old task separately from the new version's eligibility, and do not use the new implementation for the old task by mistake.

When resuming an old task, keep its original evaluation implementation and configuration unchanged. A new implementation that changes Add/Search behavior must follow the new-version process and must not be used directly to resume an old task.




Endpoint Configuration and Capacity Declarations







Q9: Can I add a Health URL that was omitted from my original application?


Please contact the committee with the original application, system version, and proposed Health URL. The committee will determine how to add it based on integration requirements and the current task status. To preserve configuration consistency, do not change configurations already bound to tasks yourself.





Q10: If a service upgrade enables larger requests, should I report it?


Please report the actual request-body, storage, and processing capacity, and whether the upgrade changes the implementation. Even if only gateway or resource settings change, explain whether endpoint behavior is affected. Changes to models, algorithms, or retrieval logic must be handled as a new version.




Preflight Checks and Participant Endpoint Issues







Q11: If the port is reachable but there is no HTTP response, is the endpoint working normally?


Port connectivity alone does not establish that the endpoint is working. Check whether the request is blocked by the gateway, application, or a downstream service. Provide connection or timeout records to help locate where the request stops. If no HTTP response was received, state that explicitly.





Q12: Can I change the Add/Search URLs when registering a new version?


This is not currently supported. Registering a new version under the same evaluation Key cannot change its bound Add/Search URLs. To change them, submit a new evaluation Key application through the official website, including the original system, original application ID, new version, and an explanation of the endpoint changes for review and recordkeeping. Once approved, complete the appropriate track's Smoke test with the newly registered version before proceeding to Full. This process supports consistent recordkeeping; we apologize for the inconvenience.




Troubleshooting Communication and Task Status







Q13: The page shows a failure, but my service logs are still updating. Should I keep changing the service?


First check the task ID, time, and evaluation stage to establish whether the logs belong to the same task and whether they come from requests still finishing or from other active tasks. Once those details are aligned, identify what needs attention to avoid accidentally changing a service that is still running.





Q14: Should I start a new email thread to follow up after the committee has confirmed my request?


We recommend continuing in the original thread, specifying the unresolved step and the issue you currently face. This preserves the communication history, connects the follow-up with previous confirmations, and reduces repeated explanations.




Model Use and Leaderboard Eligibility







Q15: Does the Academic leaderboard require the specified models? Can an open-source model such as Qwen3-Embedding-4B replace text-embedding-v4?


For the Academic (Open-source Methods) leaderboard, the embedding model must be text-embedding-v4, and LLM-related components must use gpt-4o-mini. Rerankers are unrestricted. Please follow these requirements when preparing your Academic configuration. If you wish to use other models, see the Industry leaderboard participation guidance in Q17.

In your application, specify the actual model names, versions, purposes, and deployment or service arrangements. Distinguish embedding models, LLMs, and rerankers so the configuration can be checked. These requirements concern models used within participant systems. Answer/Eval models are configured centrally by the platform and cannot be selected or replaced by participants.





Q16: Which leaderboard can an open-source project using other open-weight or self-trained models enter?


You may apply for the Industry leaderboard. Commercialization is not required: non-commercial open-source projects are also eligible, providing a participation route for teams using other open-weight or self-trained models. When choosing a leaderboard, note that Industry entries are not eligible for Cycle 2 cash prizes. Choose according to your actual model configuration and participation goals.




Explore Coding and Multimodal Tracks

Validate More Capabilities of Your Memory System




Cycle 2 evaluation is still in full swing. The Coding and Multimodal tracks give research teams, open-source projects, and product teams opportunities to validate memory systems in different task settings and demonstrate their technical strengths. Choose tracks that match your system's strengths and put your research and product capabilities to the test on real tasks.

Coding Track

If your team focuses on accumulating software engineering experience, analyzing historical errors, and reusing effective experience, the Coding track lets you test the role of memory systems on real coding tasks and gain evaluation evidence for further improvement.

Multimodal Track

If your team focuses on image evidence retrieval, image-text associations, cross-session visual memory, or long-term multimodal memory, the Multimodal track lets you assess those capabilities and demonstrate your system's performance in these settings.

Cross-Track Participation

Teams already participating in the Textual track may also explore Coding and Multimodal according to their capabilities, gaining insight into the system's applicability and opportunities for improvement across more task dimensions. Whether you focus on one setting or explore cross-track applications, we welcome evaluations that showcase your technical strengths.

Contact Information

For other questions, please contact us through the official email address. We will continue updating the documentation and FAQs based on practical integration experience.

For specific eligibility, evaluation arrangements, and exceptional cases, the latest rules on the official website and written confirmations from the organizing committee take precedence.