Agent Memory Leaderboard 使用统一的端到端流程评估不同记忆系统。数据库、索引和内部架构不作限制。开源方法榜（学术榜）的 Embedding 模型须使用 text-embedding-v4，LLM 相关组件须使用 gpt-4o-mini；Reranker 不作限制。商业产品榜（工业榜）不受上述模型限制。Add / Search 接口须符合现行规范，评测过程和检索结果须能够复核。

先提交 Evaluation Access Request，完成参测系统、版本、在线 Add / Search 接口和鉴权方式审核。第二期所有榜单均要求参赛方自行部署接口，AML 不接受仅提交仓库后代为部署。

1. 01

   提交准入申请

   选择申请新 Key，或输入已有 Key 为其增加版本；填写联系人、系统版本、Add / Search 接口、鉴权、容量和运行限制。开源方法另附仓库与来源披露。

2. 02

   完成 Key 绑定

   审核通过后，新申请会签发 Leaderboard API Key；新增版本申请会直接绑定到原 Key，无需管理员再次手工录入接口信息。

3. 03

   运行公开 Smoke

   平台直接校验参赛方提交的在线接口，并按照现行同步规范执行 Add → Search → Answer → Evaluate Smoke 流程，确认接口可用后再进入正式评测。

4. 04

   完成 Full 前置检查

   选择 full 前，必须逐项勾选提交清单：smoke 已通过、API 契约正确、运行说明完整，以及原创性披露和诚信承诺均已完成。

5. 05

   提交评测任务

   在 Evaluation 页面验证 API Key，选择已绑定的版本，填写唯一的 Run Label 后提交 full 评测。正式任务使用统一的数据集套件和 Top K。

6. 06

   查看进度与审计状态

   任务依次执行写入、检索、回答、评分和结果汇总。页面会显示当前阶段、完成进度和错误摘要；我们同时保留复核所需的过程记录。

7. 07

   查看结果与发布状态

   评测结果首先仅对绑定的 API Key 可见。成功完成的 full 任务会进入公榜资格校验和审核队列；审核通过后生成公榜提交记录。

参赛选手配置 Add 和 Search 的接口地址，请求和响应格式按照现行规范固定，不随 URL 路径变化。生产环境建议使用 HTTPS。URL 中不得包含用户名、密码等凭据，也不得指向私有、回环或链路本地地址。

{
  "request_id": "eval:run_abc123:locomo_refined:conv-0:chunk-0",
  "messages": [{
    "role": "user",
    "timestamp": 1704067200000,
    "content": "memory text"
  }],
  "user_id": "eval:run_abc123:locomo:conv-0",
  "session_id": "eval:run_abc123:sample:0"
}

HTTP 200
{
  "success": true,
  "request_id": "eval:run_abc123:locomo_refined:conv-0:chunk-0",
  "user_id": "eval:run_abc123:locomo:conv-0",
  "session_id": "eval:run_abc123:sample:0"
}

{
  "query": "Which answer best matches the memory?",
  "options": ["A. First answer", "B. Second answer"],
  "user_id": "eval:run_abc123:locomo:conv-0",
  "top_k": 100
}

{
  "data": [{
      "id": "mem_1",
      "content": "remembered fact text",
      "score": 0.87,
      "created_at": "2026-07-01T12:00:00Z"
  }]
}

参测接口应使用标准 HTTP 状态码，并返回便于排查问题且不含密钥的错误信息。平台业务错误通常采用 {"detail":{"reason":"..."}} 格式；字段校验失败时，HTTP 422 会返回结构化错误明细。

Agent Memory Leaderboard compares memory systems under a consistent end-to-end evaluation. Databases, indexes, and internal architectures are unrestricted. Open-source Methods (Academic) entries must use text-embedding-v4 for embeddings and gpt-4o-mini for LLM-related components; rerankers are unrestricted. Commercial Products (Industry) entries are not subject to these model restrictions. The external Add/Search API and retrieval evidence must be conformant and auditable.

Open-source Methods and Commercial Products submissions both use participant-hosted Add/Search APIs and follow the same review, Key issuance, and public Smoke flow. AML does not deploy repository-only submissions.

1. 01

   Submit the access request

   All entries provide deployed Add/Search endpoints, authentication, capacity, and operational details. Open-source Methods entries additionally disclose the public repository, original authors, technical report, and method changes.

2. 02

   Complete key binding

   Approval either issues a new Leaderboard API Key or binds the submitted version to the verified existing key without administrator re-entry.

3. 03

   Run public Smoke

   The platform checks the participant-operated endpoints and runs the synchronous Add → Search → Answer → Evaluate Smoke flow. AML does not build or deploy participant systems.

4. 04

   Complete the Full gate

   Before selecting full, confirm the interactive checklist: smoke passed, API contract followed, run instructions complete, original work disclosed, and no integrity violations.

5. 05

   Start the job

   Verify the API Key on the Evaluation page, select a bound version, assign a unique Run Label, and submit the full evaluation. Formal jobs use the fixed suite and platform Top K.

6. 06

   Monitor progress and audit status

   The job proceeds through ingestion, retrieval, answering, scoring, and aggregation. The interface reports stage, progress, and error summaries while the platform retains evidence required for review.

7. 07

   Review results and publication status

   Results are initially visible only to the bound API Key. A successful full job enters the public-eligibility gate and review queue; approval creates a public submission.

Participants configure the Add and Search URLs; request and response schemas are fixed and do not vary by path. Production deployments should use HTTPS. URLs must not embed credentials or resolve to private, loopback, or link-local addresses.

{
  "request_id": "eval:run_abc123:locomo_refined:conv-0:chunk-0",
  "messages": [{
    "role": "user",
    "timestamp": 1704067200000,
    "content": "memory text"
  }],
  "user_id": "eval:run_abc123:locomo:conv-0",
  "session_id": "eval:run_abc123:sample:0"
}

HTTP 200
{
  "success": true,
  "request_id": "eval:run_abc123:locomo_refined:conv-0:chunk-0",
  "user_id": "eval:run_abc123:locomo:conv-0",
  "session_id": "eval:run_abc123:sample:0"
}

{
  "query": "Which answer best matches the memory?",
  "options": ["A. First answer", "B. Second answer"],
  "user_id": "eval:run_abc123:locomo:conv-0",
  "top_k": 100
}

{
  "data": [{
      "id": "mem_1",
      "content": "remembered fact text",
      "score": 0.87,
      "created_at": "2026-07-01T12:00:00Z"
  }]
}

Participant endpoints should use standard HTTP status codes and return actionable errors without secrets. Platform business errors normally use {"detail":{"reason":"..."}}; HTTP 422 provides structured field-validation details.