Agents Write Their Own Memory

The five memory types I wrote about earlier in this series (episodic testimony, semantic fact, procedural knowledge, pending scratch, curated zettel) borrow honestly from human memory science, and the borrowing holds up better than it has any right to. Psychologists really do distinguish remembering that something happened from knowing that something is true from knowing how to do something, and agent memory really does inherit those distinctions, failure modes included.

This essay is about where the analogy stops. Watch an agent work for an hour and count what it produces. Chains of reasoning that led to conclusions. Judgments about which of its tools performed and which one quietly lied. Observations about the environment it is operating in. Plans, half-executed. Code. Opinions it formed along the way. And decisions: commitments about what to do that it will act on next session as if they were law. None of that came from you. The agent wrote it, and in any system doing real work, what the agent writes for itself quickly outgrows everything it was ever told.

Human memory never had to solve this as an engineering problem. We do not get to choose whether our conclusions are filed separately from our evidence. In fact most human reasoning failures (believing your own speculation, mistaking a stale plan for a current one, trusting an opinion because you have repeated it) come precisely from not having that separation. Agents give us the choice. The memory an agent writes for itself can be typed, tiered, and governed on purpose, which means for once the type system gets to fix a bug the human original shipped with.

Most stacks decline the offer. The agent's own output meets one of two default fates. Either it evaporates with the context window (the reasoning is gone, the conclusion survives, and nobody can ever again say why), or it gets dumped into the store as undifferentiated prose, where it begins its second career masquerading as knowledge.

The masquerade problem

The masquerade is the dangerous fate, and it has a specific mechanism worth naming: self-citation.

An agent forms a guess mid-task. Plausible, useful, unverified. Stored as prose in a flat index, that guess is now a string like any other string. Next week, a retrieval pulls it back as context, and the model reads its own speculation in the same voice it reads your verified facts. It acts on it, produces new conclusions conditioned on it, and stores those too. By the third generation the original guess has become load-bearing ancestry, cited by its own descendants, indistinguishable from ground truth, with nothing anywhere recording that the whole lineage traces to something the agent made up on a Tuesday.

No one poisoned this store. The agent poisoned itself, politely, one plausible write at a time. This is why origin is not optional metadata. In SmartMemory every memory carries where it came from, and origins sort into visibility tiers: content a human actually provided is not the same class of object as something a background process derived, and the speculative tier does not get recalled with user-content authority. It can surface when you search for it. It does not get to whisper into every session as if you had said it. An agent's own output is testimony from a witness with a known conflict of interest, and the store should treat it that way.

A tour of the family

Beyond the core five, SmartMemory types the agent's own cognitive output. The same rule from the five-types essay decides membership: when a kind of memory has its own lifecycle and its own rules for aging and trust, it gets to be a type.

Observations are the agent's field notes: something it noticed about the world or itself, tied to a moment. They are episodic in character (append-only, timestamped, never silently promoted to fact) but the witness is the agent, which means they start life on the speculative side of the trust ledger and have to earn their way up.

Reasoning is the why attached to a conclusion. As retrieval content it is nearly worthless (nobody wants a wall of chain-of-thought as context). As provenance it is priceless. When a conclusion turns out wrong, the reasoning that produced it is the difference between a fixable failure and a mystery. It is kept as lineage, not as reading material.

Opinions are stances the agent formed: this library is unreliable, this user prefers brevity, this approach tends to fail. Genuinely useful, revisable by construction, and never entitled to fact authority no matter how often they are repeated. An opinion that keeps getting confirmed should graduate through evidence, not through repetition.

Plans are commitment structures: an intent decomposed into tasks, each of which will complete, fail, or be abandoned. A plan is kin to pending scratch (its details should expire once discharged) but its skeleton is worth keeping, because what was planned and what actually happened is exactly the gap agents need to learn from.

Code the agent produced is an artifact with lineage: which session wrote it, in service of what, so that when it misbehaves later the question "why does this exist" has an answer.

Evaluations are the agent keeping score: how a tool, a model, or another agent has actually performed, broken out by dimension and domain, superseded over time as performance drifts. These are the memory of how good things are, derived from evidence rather than asserted, and they age the way reputations should: bi-temporally, with history.

Seven types, one common property. Every one of them is something the agent asserts about its own cognition, which means every one of them needs the trust machinery (origin, tier, lifecycle, supersession) more than the content you provided ever did. Your facts arrive with a source. The agent's self-produced memory is its source, and the store has to compensate.

One of them outranks the rest

The family is not flat, and this is the observation the next three essays are built on.

Look at where each type points. Observations are evidence, waiting to matter. Reasoning justifies something. Opinions incline toward something. Plans execute something. Evaluations judge how something went. Every arrow converges on the same object: the moment the agent commits to a course of action. The decision is where the agent's cognition cashes out into consequences, which makes it the type where a memory failure stops being embarrassing and starts being expensive. A lost observation costs you evidence. A lost decision costs you the ability to know why your agent does what it does.

Decisions also have the richest lifecycle of anything in the store (they are reinforced, contradicted, superseded, retracted) and the strongest claim to structure: what was chosen, what was rejected, what it was derived from, what would reopen it. If you type only one thing your agent writes for itself, type its decisions.

That is where this arc goes next. First, why a decision is not a fact, and what the flattened version costs. Then how a decision ages, which is nothing like how a fact ages. Then the payoff: why the decision record is the thing that turns a memory system into an expertise layer.

What to do if your agent's output all looks the same

Tag origin on every write. Human-provided, tool-returned, agent-derived. If you cannot tell whose voice a memory is in, neither can your agent.
Tier the trust. The agent's own speculation should be findable on purpose and silent by default. Never recall it with the authority of things you actually said.
Keep reasoning as lineage, not as content. Store the why attached to the conclusion, retrievable when the conclusion is challenged, invisible otherwise.
Let opinions graduate on evidence, not repetition. Repetition is how the masquerade works.
Expire plan residue, keep the plan skeleton. Planned-versus-happened is a signal worth keeping forever. The scratch is not.
Type your decisions first. They are where all the other arrows point, and they are next week's essay.

I'm building SmartMemory, the expertise layer for AI agents: provenance-tagged, bi-temporal, graph-backed memory that knows not just what's true, but what to do, and when it stopped being true.

Try it: pip install smartmemory (docs) · hosted beta (private): smartmemory.ai/signup