A lightweight long-term memory approach: less processing, greater fidelity

AML's invitation noted that we ranked first across the entire leaderboard on two subcategories: Overwriting Old Values and Determining Current State (54.50), and Contradiction Detection and Conflict Resolution (24.79).

One point needs to be made clear first:

ActiveMemoryIndex's storage layer uses only two SQL operations: CREATE and INSERT OR REPLACE. There is not a single UPDATE or DELETE. The system never updates or merges memories, invalidates old memories, detects contradictions, or resolves conflicts. Every written entry remains valid indefinitely, and all entries have equal standing.

Those two first-place results were therefore achieved without any governance mechanism.

This is not modesty; it is the starting point of this article. We have a publicly available supporting observation: some systems on this cycle's leaderboard do implement governance. FlowGrid labels conflicts and downweights old values without deleting them; its README reports a stale-value leakage rate of 0.53, yet its Memory Governance score is not high. We perform no governance at all and score 37.95, the highest.

If this is not a coincidence, it suggests that governance matters when it genuinely changes what is returned, not merely the score ordering. Labeling and downweighting leave the judgment to an intermediate layer that cannot take responsibility: old values still enter context, just a little lower in the ranking, and the reader model still reads them. Giving the reader complete, timestamped, unrewritten raw records and letting it determine which remain valid can be more reliable.

This is a hypothesis, not a conclusion. But it explains why the system has its current form.

The name ActiveMemoryIndex comes from the underlying research, An Index, Not a Store: The Model Does Remember — It Just Needs Its Notebook. The two words correspond to the study's two halves.

Active means memory written into model weights. The paper's main focus is online weight-level learning: LoRA adapters write new knowledge directly into a local 9B model's weights, round by round, making the model itself the store and retrieval an act of recalling from weights. This is the original meaning of active memory: memory is not sitting in an external index waiting to be queried; it becomes part of the model.

Index means an external index. Dual-write storage and register-matching retrieval—the Add/Search harness—were originally built as a control instrument for measuring the weight-level system on the InMind benchmark.

The challenge rules require gpt-4o-mini to be the only model used during Add and Search. As a result, the paper's strongest configuration (9B backbone + LoRA r=32), EWC regularization, the Benna-Fusi cascade, and per-user weight partitions were all excluded from the submission. We did not abandon them; the contract structurally cannot accommodate weight-level memory.

What we submitted was the measurement instrument, not the thing being measured.

AML should also be aware of this: the current contract excludes the entire family of model-as-storage architectures. The leaderboard currently measures the external-index family. Fixing the generative model is the right way to compare memory systems fairly, but it also defines the scope of coverage.

Writing: dual write. Messages in each Add request are stored twice. One copy is verbatim text, prefixed with a [YYYY-MM-DD HH:MM] timestamp and speaker labels based on role. The other consists of first-person atomic facts extracted by gpt-4o-mini from the entire block. Both copies are encoded by the same local embedding model and written to the same SQLite table, with the transaction committed before returning 200. The contract requires memories to be immediately searchable after writing.

Retrieval: rewrite the question in the user's own voice. The corpus contains first-person chat logs, while queries are third-person questions such as “What does John plan to do?” They are in different registers. We ask gpt-4o-mini to rewrite the question as a sentence in which the user recalls their own history—for example, “Did I mention John's plans?”—then combine its embedding with the original query embedding for ranking.

To be explicit, this is HyDE with N=1 (Gao, Ma, Lin, Callan, 2022, arXiv:2212.10496). Because (1-w)·(q·d) + w·(r·d) = ((1-w)q + w·r)·d, weighted similarity and embedding averaging are the same operation; w=0.5 is precisely the paper's default form. Our only distinctive element is the prompt: generating a recall question rather than a hypothetical answer.

Returning results: raw text first. The selected 100 entries are ordered with verbatim records before extracted facts, while preserving relevance order within each group. This changes only the ordering, not the set.

What is absent: graphs, hierarchical summaries, entity disambiguation, and memory governance.

Paired test: raw-first vs. relevance ordering, +141/−67, p < 0.0001, with improvements in all four categories.

Three components we built and then removed:

About those 7.8 points from parent blocks: returning all 20 messages in the block containing a hit yields .711. But an equal-budget control ruled it out. Truncating the parent block to the baseline's character count gives only .572, 6.1 points below the baseline. Coarse-grained units themselves contribute negatively; all gains come from 5.4 times as much content. That volume can increase because the contract's top_k limits only item count, not size.

We also found that volume and granularity cannot be separated under this contract. With a 100-item limit, individual messages provide at most about 12.4k characters. To return more text, one must define a memory item more coarsely. The limit of this approach is a legal degenerate solution: a LoCoMo conversation contains about 40 blocks, so returning all of them hands over the entire conversation verbatim while using only 40 ≤ 100 items.

Thus, .711 measures not memory capability but how much reader context we are permitted to spend. Here is a concrete suggestion for AML: add a token limit alongside the item-count limit, or publish tokens/query together with accuracy. A precedent is TREC QA's reduction of answer strings from 250 bytes to 50 bytes, for exactly the same reason: every organization that submitted both lengths scored higher with 250 bytes.

The following is output from an actual run, not an illustration.

Input—the platform's Add request (LoCoMo conv-2, excerpt of 6 messages):

Note that extracted facts overlap heavily with the originals; many simply paraphrase first-person statements in the third person. This is the source of the ablation results discussed later: extraction is a lossy rewrite with a net contribution of only 0.9 points, at the cost of one LLM call per write.

Retrieval—the platform sends a question:

Already implemented, pending validation in the next cycle's Full evaluation: neighboring windows.

Each matched raw message is returned with one preceding and one following message. Neighbors consume the same pool of 100 slots, so this reallocates a fixed budget rather than packing in more content. The number of source blocks drops from 26.2 to 20.9, while character count rises by only 8%.

Measured performance: .6333 → .6802, +4.7 points (n=1540, paired +141/−67, p < 0.0001), including a 5.3-point gain in the temporal category. Radii 1, 2, and 3 are statistically indistinguishable. We choose 1 because it uses only 3 slots per seed and preserves the most breadth.

The same example after adding a window:

A single message is often not self-contained. Pronoun referents, the other speaker's response, and the conversation date occur in neighboring turns, and the reader cannot reconstruct things it was never given.

We owe InvMem thanks for this. Reading MEMORY_RESULT_WINDOW=1 in the public repository of this cycle's first-place Open-source Textual entry made us realize that we had incorrectly ruled out an entire family of techniques. We inferred that breadth is a hard constraint from the observation that parent blocks perform worse under an equal budget, then used that inference to exclude every form of expanded return units. The correct statement is that breadth constrains the extremes, not the middle range. Neighboring context is not unusual on this leaderboard: the 10th-place aml-memory-mvp uses adjacent-turn expansion, and the 8th-place FlowGrid boosts adjacent evidence.

Had this been a closed-source leaderboard publishing only scores, we would still not know what we had missed.

The next problem we really need to tackle is temporal reasoning.

Axis C (Temporal and Event Sequence Reasoning) is our weakest, at 18.35. The entire Open-source leaderboard is stuck around 18–24.6, while the top two Industry entries score 56–60. This is not just our problem; it is a shared boundary of this family of methods.

The public competing systems we have read handle time by prefixing timestamps to text, extracting years with regular expressions, or applying exponential decay by insertion time. None of these is temporal reasoning. We plan to resolve relative time expressions into absolute times at write time and version facts. “Last week” should be anchored to specific dates when written, and old and new values for the same state should have an explicit ordering. This addresses both C and A—wrong temporal anchors also cause wrong fact recall—and operates at the write layer, the layer we have never touched and that remains collectively undeveloped across the Open-source leaderboard.

Interestingly, this would move us toward the approach originally described in AML's email. The difference is that the description is not accurate yet; if it becomes accurate later, it will be because we actually built it.

Two other questions under evaluation:

First, should the extraction channel remain? It is worth only 0.9 points but adds a fixed cost to every write. If a lighter write path achieves the same score, it should replace it.

Second, the weight-level half of the project will continue. It cannot enter this leaderboard, but it remains the main focus of this research direction.

ActiveMemoryIndex was submitted by Xuhao Lin, an independent researcher, not a company or institution.

The actual working arrangement is one person plus an AI collaborator (Claude). Code, experimental design, and the first draft of this article were produced collaboratively. All experiments ran on the author's own hardware and were verified by the author. Every trade-off—what to deploy, what to abandon, and which scores not to claim—was decided by the author.

This division of labor is substantive. For example:

The author rejected the parent-block gain of 7.8 points because it felt like cheating.

Two mistakes documented here were made by the AI collaborator: extrapolating a conclusion valid at an extreme to the middle range, thereby missing neighboring windows; and introducing bias into instructions to subagents, producing contradictory noise estimates of 51% and 24% for the same data. The observation leading to the largest improvement—reading that configuration line in the first-place repository—came from the author. (Author: So my main contribution is getting the AI to learn?!)

A technical report claiming AI collaboration conveys little information if it does not explain where that collaboration went wrong and where a person corrected it.

References and acknowledgments: HyDE (arXiv:2212.10496), query2doc (arXiv:2303.07678), passage-level evidence (Callan, SIGIR 1994), LoCoMo (arXiv:2402.17753), AML's public evaluation code, and all peers on this cycle's Open-source leaderboard who released their code.

Scope of the numbers: The leaderboard score (44.84, 3rd on the Open-source leaderboard) reflects the submitted version without neighboring windows. That feature was added after leaderboard publication and will be reflected only in the next cycle's Full evaluation. All other numbers in this article come from our own offline reproductions: public locomo10 data, a local judge, n=1540, and 2–3 repetitions per arm. These and the AML evaluation use different yardsticks. They should guide comparisons among configurations, not be converted directly into leaderboard scores. All experimental data are in bench/results/ in the repository, and every figure can be reproduced using the repository's commands.