Q1: Who can participate? Can we enter multiple tracks?

Universities, research institutes, companies, open-source communities, independent teams, and individuals are all welcome. Teams may also include members from different organizations.

A participant may enter one or more of the Textual, Coding, and Multimodal tracks, but must choose a single division for this cycle: the Open-source Methods leaderboard or the Commercial Products leaderboard. Admission, Smoke and Full quotas, and results are managed separately for each track.

Q2: Is a GitHub repository or Docker image alone sufficient for the Open-source Methods leaderboard?

No. In Cycle 2, participants in both the Open-source Methods and Commercial Products leaderboards must deploy stable, accessible Add/Search APIs themselves. The platform does not automatically pull repositories, start containers, or host services on participants' behalf.

Open-source entries must also provide a public repository, a pinned commit, dependencies, execution instructions, and a description of the original work and the changes made. These materials support reproduction and review; they do not replace a live service.

Q3: What are the main differences between the open-source and commercial divisions?

Both divisions use the same registration entry point and Add/Search integration boundaries. The main differences concern disclosure and eligibility:

• The Open-source Methods leaderboard requires a publicly available version that can be verified and reproduced, and compliance with this cycle's Add/Search model constraints. Teams that meet the rules and pass review may be considered for prizes.

• Commercial Products entries may keep their internal implementation private, but must fix the product, model and API versions, and key operating conditions. They are not eligible for prizes.

Q4: How many tracks can one AML Key access? Are quotas shared across tracks?

One AML Key can be bound to multiple tracks. Textual, Coding, and Multimodal evaluations are separate and do not share Smoke or Full quotas.

For this cycle, each AML Key has a maximum of 30 Smoke runs and 2 Full runs per track. Using a Full run in the Textual track does not consume the Coding or Multimodal track's Full quota.

Q5: What materials should we prepare for registration?

At a minimum, prepare: a contact person and email address; team or organization; system name and fixed version; division and track; Add/Search URLs; authentication method and access credentials; operating instructions; concurrency and rate limits; typical request duration; request-body and storage capacity; models and external dependencies; a multimodal-processing declaration; and service-maintenance arrangements.

If you provide a Health URL or an approved asynchronous Add Status URL, submit these as well. Open-source entries also need a public repository, a pinned commit, and a description of the source work and modifications.

Q6: Is registration free? Who pays for evaluation costs?

Registration is free. The platform handles the unified Answer and Evaluation stages and evaluation orchestration. Participants pay for their own APIs, databases, model calls, networking, storage, compute, and operations.

Q7: Must open-source entries call gpt-4o-mini? What if the system uses no generative model?

When an Open-source Methods entry calls a generative model during Add/Search, it must use this cycle's designated model, gpt-4o-mini. If the entire Add/Search pipeline makes no generative-model calls, there is no need to add a meaningless call. Mark this item as N/A in your submission and explain the embedding, reranker, or other components actually used.

If the checklist wording does not fully match your system's design, obtain written confirmation through the official email address before starting Full. Do not use an ambiguous checkbox selection in place of accurate disclosure.

Q8: Can open-source entries use models they have fine-tuned themselves?

This cycle's published requirements explicitly restrict generative models used in Add/Search by open-source entries. Self-fine-tuned models, distilled models, and online parameter updates within an evaluation task must not be assumed to be approved.

If a self-trained model is integral to your method, separately disclose its base model, training process, training data, how the weights are made available, and its actual purpose in your admission application. Obtain an explicit integration decision before Full. Systems that depend on unapproved self-trained models may also consider the Commercial Products leaderboard, depending on their disclosure and product characteristics.

Q9: Can published evaluation data such as LoCoMo-Refined and BEAM be used to train the submitted model?

Public subsets and examples included in this cycle's evaluation are intended for understanding tasks, API prechecks, and evaluation reproduction. They should not be used directly to train or fine-tune submitted systems, or to overfit to evaluation questions. Data received during formal evaluation, and any derived copies, may only be used for that evaluation and necessary review—not for training, product analytics, dataset reconstruction, or external distribution.

When using external training data, disclose its source, license, relationship to this cycle's evaluation datasets, and measures taken to prevent data contamination.

Q10: Will the platform release a complete, standardized Add dataset for local simulation?

Public examples are intended to verify fields, types, ordering, isolation, and idempotency semantics. Restricted evaluation data will not be released in full for local integration testing.

Participants can follow the API Guide to create their own synthetic data covering long messages, multiple sessions, timestamps, duplicate request_id values, empty retrieval results, and error responses. After admission, Smoke is the platform's official small-scale, end-to-end compatibility check. Smoke uses a separate evaluation identity and namespace, which must not be mixed with Full's memory space.

Q11: What exactly does an HTTP 200 response from Add mean?

It means the batch has been successfully persisted and can be retrieved immediately by subsequent Search calls. Do not return synchronous success merely because a request has been received or queued, or while indexing is still incomplete.

If your system requires asynchronous writes, first pass the asynchronous-integration review for the relevant track, bind an Add Status URL, and respond according to the official status protocol. Returning a task_id alone does not satisfy the asynchronous-integration requirements.

Q12: Does the platform change request_id when retrying Add?

For normal network errors, timeouts, or retryable status codes, the platform reuses the same request_id and payload for the same logical request. Your system should use request_id as the idempotency key to handle cases where a write succeeds but its success response is lost in transit.

Do not cache idempotency state only within a single process. Necessary idempotency records must survive stateless restarts during Full or recovery.

Q13: What do user_id and session_id represent?

user_id defines the memory-isolation boundary for Search. Memories belonging to different user_id values must never be mixed.

session_id identifies the conversation a piece of content came from and helps organize history, but does not replace user_id. One user_id may contain multiple sessions. Search requests generally do not supply a session_id filter, so relevant history across sessions should remain retrievable.

Q14: Does the options field in Search contain the correct answer? How should it be used?

For multiple-choice questions, all answer choices may be supplied in options, naturally including the correct one. However, the platform does not identify which option is correct or send additional gold-standard answers. Open-ended questions do not supply options; the field is empty.

Your system may use or ignore options. Since the choices include distractors, we do not recommend treating all options as facts or indiscriminately appending them to retrieval text.

Q15: Is messages[].timestamp always present?

This is an optional field in the base API. When supplied by the platform, it represents event time. Preserve it in the original order rather than unconditionally replacing it with the time the API request arrived.

Your system must also accept requests without timestamp. Do not turn an optional field into a mandatory prerequisite for Add.

Q16: How many items should Search return? Must it return exactly 100?

No. top_k is the maximum number of items. Formal Textual evaluation uses top_k=100. Your system may return fewer than 100 items, but must not exceed the limit specified in the request. When no relevant memory is found, return the normal empty response shown below:

The returned order determines evidence priority. Items near the beginning are more likely to enter the platform's Answer context.

Q17: Must score be comparable across queries and strictly decreasing?

score is optional and represents your system's own relevance score, not an official evaluation result. The base API does not require scores to be comparable across queries or prescribe a particular normalization method.

Evidence priority is determined by the order of the data array. Scores may tie, but the returned order must be stable, explainable, and consistent with the system's declared relevance ranking.

Q18: Can we combine multiple memories into a narrative before returning them?

Within the scope of your declared method, you may summarize, structure, or organize evidence from legitimately stored memories, subject to three boundaries:

1. Content must come only from the permitted history for that user_id.

2. Do not introduce future information, external answers, or gold-standard answers.

3. Do not generate a final answer directly from the current question and disguise it as historical memory.

If your system uses a generative model to combine evidence, disclose the model, the purpose of its prompts, and its version, and comply with your division's model requirements.

Note that returned memory content may be truncated if it exceeds the limit, which can affect performance.

Q19: In Textual evaluation, are all Add calls always completed before Search begins?

The standard batch Textual pipeline enters Search after the corresponding write phase completes. Cycle 2 also includes Streaming continuous-memory evaluation, which interleaves Add and Search in event order. Therefore, return from Add only when the operation has actually completed.

Your system must support both retrieval after batch writes and the sequence “write some content—search—write more—search again.” Strict isolation between evaluation identities and user_id values remains essential.

Q20: How long can an Add/Search request take? How are rate limits and temporary errors handled?

In this cycle, the maximum waiting time for an individual Add or Search request is 30 minutes.

When rate limiting is necessary, return HTTP 429 with a valid Retry-After header. The platform performs bounded retries for agreed network errors, timeouts, and certain temporary status codes. Deterministic authentication, path, JSON-type, or required-field errors will not resolve themselves through repeated retries.

Q21: How should the Health endpoint be implemented?

If you configure a Health URL, provide an unauthenticated GET endpoint. Any 2xx response indicates a healthy service. Health is only used to check service accessibility; it must not expose keys, model configuration, database addresses, or other internal information.

Q22: Does the Coding track also use Add/Search? How is it scored?

Yes. Participants first receive relevant or noisy historical engineering trajectories through Add. While working on a code repository and task, the platform's problem-solving agent calls Search and uses the retrieved experience as supporting context. The agent must then actually modify code in an isolated environment. The platform's task verifier determines whether the patch solves the problem.

This cycle contains 150 base tasks. Each runs under both relevant and noisy memory conditions, giving 300 evaluation units in total. These conditions describe the memory environment and should not simply be treated as two difficulty levels.

Q23: Can a caption-only multimodal system participate?

You may apply for caption-only participation, but your API must correctly accept arrays containing both text and images rather than failing when image fields are present. Submission materials and public leaderboard information must accurately disclose whether the system processes original images, uses OCR, relies only on captions, and returns image evidence.

Caption-only processing is not equivalent to understanding original images. Specific scoring components and prize eligibility follow this cycle's frozen multimodal configuration.

Q24: Can multiple tracks share the same service?

Yes, but separate quotas do not automatically isolate compute resources. When multiple Full runs execute simultaneously, load on servers, databases, indexes, model quotas, and bandwidth may accumulate.

If tracks share a service, strictly isolate state between runs and user_id values, and reserve capacity for aggregate concurrency, peak storage, and the potential scope of failures.

Q25: Does passing Smoke guarantee that Full will finish?

No. Smoke checks small-scale API compatibility and the end-to-end pipeline. It does not establish sustained concurrency, storage capacity, indexing speed, external model quotas, or long-term stability at full evaluation scale.

Before starting Full, recheck request success rates, P95/P99 latency, 429 and 5xx responses, storage and memory, upstream model balances and call quotas, alerts, and incident contacts.

Q26: If Full does not reach 100% completion, will partial results be published?

Only a fully successful Full run that passes review can produce an official result and enter the public leaderboard. If individual records still fail, evaluation units are missing, or artifacts are incomplete, the denominator must not be reduced to present a partial result as complete.

The platform performs bounded retries for temporary errors and resumes the same task when eligible. Completed steps that pass consistency checks may be reused, but participants must retain the original version, memory state, idempotency records, and run namespace.

Q27: After a failed or cancelled Full run, can we simply create a new one without consuming quota?

Do not assume this. Automatic retries, record requeueing, and checkpoint recovery within the same root evaluation still belong to one Full run. Recovery does not count as another Full run.

If the failure originates from the participant's API, dependencies, or submitted implementation, that Full run is generally considered to have been initiated; do not expect an automatic quota refund. For verified platform-infrastructure failures, the organizing committee handles recovery or eligibility based on task records. When an error occurs, prioritize resuming the original task rather than repeatedly creating new ones.

Q28: How is the cooldown between two Full runs calculated?

In this cycle, each AML Key may run Full at most twice per track. The second run unlocks 30 days after the first Full run successfully completes. Automatic retries and checkpoint recovery within the same task do not trigger an additional 30-day cooldown.

Having a quota of two runs does not guarantee enough time to use both within the schedule. Teams planning a second evaluation should work backward from the November 4 evaluation cutoff and complete their first Full run as early as possible.

Q29: Can we change endpoints or models, or fix the algorithm, while Full is running?

No. Once Full is accepted, the system version, endpoints, authentication, evaluation contract, and scoring parameters are bound to that task. Operational stateless restarts are allowed, but data, models, and protocol behavior must remain unchanged.

If you must change a model, algorithm, prompt, or implementation in a way that affects results, register a new version and complete integration again. Do not connect a new implementation directly to an old task's checkpoint.

Q30: Why is a completed Full run not yet on the public leaderboard?

After Full finishes, its results are initially private to the bound identity. They must still pass completeness, version, scoring, and compliance review. Only approved, valid results may enter the relevant public leaderboard.

The official result is the latest valid Full run in the same cycle and track. A later run that is incomplete or has not passed review does not automatically replace an earlier valid result. Once the later run becomes valid, the higher of the two scores is not selected.

Q31: How soon must evaluation data be deleted after a task completes?

Within 30 days of task completion, delete the evaluation data and derived copies associated with that task. Obtain the platform's written permission in advance if an extension is necessary. We recommend keeping internally verifiable deletion records, such as the task ID, deletion time, execution result, and an audit-log summary.

If the task is still being recovered or the platform has requested a review, first confirm the permitted retention scope and new deadline. Keeping a service online does not permit indefinite retention of evaluation data.

Q32: What information helps the organizing committee diagnose a problem faster?

Please email:

[email protected]

Where possible, include:

• System name, fixed version, track, and division.

• Application record or Run/Job ID.

• Incident time and time zone.

• Whether the run is Smoke or Full, and the affected stage: Add, Search, Answer, Evaluation, or review.

• Error code, expected behavior, and actual behavior.

• Scope of impact and troubleshooting already performed.

• Sanitized logs and a timeline that can be shared safely.

Do not email access keys, complete restricted evaluation data, gold-standard answers, or information about other participants.

The organizing committee generally provides an initial reply within 5 working days of receiving your email. If you have not received a reply after that period, follow up in the original email thread and check your spam folder.

For other questions, please contact us through the official email address. We will continue updating the documentation and FAQ based on actual integration experience.