Teaching Memory to Reason

TLDR. A memory that keeps only conclusions forgets the most useful thing an agent produces: why it concluded what it did. So we made reasoning a stored type of its own. Storing it was the easy part. Reasoning stored like an ordinary fact comes back like an ordinary fact, and an agent that rereads its old deliberation as evidence ends up believing guesses it already rejected. Getting it right means visibility cannot belong to a kind of memory, because the same kind, reasoning, needs different visibility depending on who wrote it. Here is what we built, what broke, and what we still have not answered.

A few weeks ago I wrote about the memory agents write for themselves, and one line in it was a promise: keep reasoning as lineage, not as content. Store the why next to the conclusion, retrievable when the conclusion is challenged and invisible otherwise. This essay is about what happened when we built it, and part of what building it taught me is that the sentence was too clean.

Why keep the why

Here is the failure that makes reasoning worth storing. An agent decides that a flaky test is caused by a race in the setup code. It adds a lock, the test passes, and the conclusion goes into memory: the setup code has a race. Three weeks later the test fails again. Was the conclusion wrong? Was it right, and something new broke? Was the lock the fix, or did the test pass by luck that day? With only the conclusion on file, nobody can say. The agent has two choices and both are bad. Trust a conclusion it cannot check, or work the whole thing out again from scratch and maybe land somewhere different.

Keep the reasoning and the question becomes answerable. What did the agent observe, what did it consider, what did it rule out, and on what grounds? If something was ruled out because of a log line that has since changed, you know exactly which step to revisit.

So in SmartMemory, reasoning is a first-class memory type, not a blob of text with a tag on it. A stored trace has a shape: the steps it went through (thoughts, actions, observations, decisions, a conclusion), the task it was serving, an assessment of its own quality, and a link to the thing it produced. That last part is what makes it lineage. When a decision came out of a piece of reasoning, asking for the decision's history hands you the reasoning behind it. The why travels with the what.

What goes wrong if you store it like a fact

The tempting shortcut is to drop reasoning into the store as text and let retrieval sort it out. That fails in a specific way.

Reasoning is dense with the vocabulary of its subject. A trace about the flaky test mentions the test, the setup code, the race, the lock, the CI runner. So when anyone later asks about any of those things, the trace is an excellent match, and it comes back as context. Now the model is reading its own old deliberation, and deliberation is full of sentences that were never claims. "Maybe it is the cache." "Could be a timezone issue." Those were hypotheses, most of them rejected two steps later. Read back without the step that rejected them, they look like things the system remembers to be true. The agent did not learn anything false. It reread its own scratch paper and mistook it for the notebook.

The second problem is quality. Not every chain of reasoning deserves to be kept. Some go in circles, some repeat themselves, some reach a conclusion the steps do not support. So a trace carries a judgment of its own quality, and when a trace is recorded through the interface that checks quality and that judgment says it is poor, or flags a serious problem in it, the write is refused instead of quietly stored.

Both problems point the same way. If the store treats reasoning like a fact, retrieval will hand it back like one.

The question it forced: who gets to see it

The obvious move is to give reasoning a visibility rule of its own. Facts get recalled. Reasoning stays out of the way until someone asks for it.

It does not hold up, because reasoning arrives through more than one door, and the doors are not equal.

Sometimes a person or their agent records reasoning on purpose, and it is their own account of why. Sometimes reasoning is pulled out of material someone handed us, a write-up or a transcript that happens to contain the steps behind a decision. And sometimes a background process that makes judgment calls about the store itself writes up its reasoning for the call it made. Same type in every case. Very different authors.

Reasoning you recorded is your own testimony, and hiding it from you would be strange. Reasoning drawn out of your own material is close to that. Reasoning a background process produced about your memory is something else entirely. It is the system's own speculation, and as I argued when I wrote about retrieval as a trust boundary, speculation should be findable when you go looking and silent by default. It should never be recalled with the authority of something you said.

So visibility cannot hang on the type. It has to hang on the origin. In SmartMemory today, reasoning recorded through the main reasoning interface and reasoning drawn from your content are both eligible for ordinary recall. Reasoning a background process wrote shows up when you search, and stays out of what the agent recalls on its own. The type tells you what the record is. The origin tells you whose voice it is in, and only the second can tell you who should hear it. When we made reasoning a proper type, every door that writes it got its own visibility decision, and those decisions do not agree with each other. They are not supposed to. I have checked these routes end to end, not every route that can write reasoning.

What we got wrong

Three things, in rising order of how much they still bother me.

First, a door with no label. One of the paths that writes reasoning stamped it with an origin the visibility rules did not recognize. The safe default for an unrecognized origin is to hide it, so those traces were stored and then invisible to both recall and search. Nothing errored. The data was there and nothing could see it. We fixed the mapping in July, and an unrecognized origin now logs a warning instead of vanishing quietly. Hidden by default is the right failure, but a silent failure is still a failure.

Second, the quality check guards one door, not all of them. Reasoning recorded through the main reasoning interface goes through it. Reasoning extracted during ingestion is stored without it, which keeps how that path behaved before it moved onto the new type. Whether to apply the same check there is a real decision with a cost either way, since a strict check drops traces people may want, and we have not made it yet.

Third, and this is the one I have not solved: one label is doing two jobs. The origin of a memory answers "who wrote this" and also "who may see this." Those agree until a background process touches something you wrote. When a maintenance process adjusts the confidence on one of your decisions, it records its own name as the last writer, which is honest attribution. The visibility rules then read that name as the author, which is wrong. A decision of yours could drop out of recall because a maintenance job touched it. We patched the specific cases we found. The real fix is to keep who created a memory separate from who last changed it, and derive visibility only from the first. That is planned and not shipped.

Where this leaves the promise

I said reasoning should be lineage, not content, invisible until a conclusion is challenged. The system we built is less pure than that sentence. Reasoning recorded through the main reasoning interface, or drawn from your own material, can be recalled like anything else you said. I think that is right for now, because hiding people's own reasoning from them felt worse than the clutter. I am not certain, and the scratch-paper problem above is exactly what happens if that reasoning comes back at the wrong moment. The honest version of the principle is narrower than the one I published. Reasoning has to be reachable as lineage, and whether it also comes back as context depends on whose reasoning it is.

That is the part I would carry to any memory system, whoever builds it. Store the why, because an agent that only remembers conclusions has to relearn its judgment every time a conclusion is questioned. Give it a shape, linked to what it produced. And decide who sees it by asking who wrote it, not what it is. Reasoning is the type that taught us that. I doubt it will be the last.

---

I'm building SmartMemory, the expertise layer for AI agents: provenance-tagged, bi-temporal, graph-backed memory that knows not just what's true, but what to do, and when it stopped being true.

Try it: pip install smartmemory (docs) · hosted beta (private): smartmemory.ai/signup