The Five Types of AI Memory

There is a sentence I keep having to repeat in design reviews, usually right after someone walks me through a memory architecture that turns out to be one embedding model, one vector index, and a lot of optimism: someone saying a thing once is not the same as the thing being true. It sounds too obvious to be worth saying. It is also the single most expensive distinction to get wrong in a memory system, and a flat vector store throws it away at the door.

The pattern underneath it is that storage is never neutral. The shape you store something in quietly decides what you are allowed to believe about it later. Drop a customer's offhand remark and a verified account fact into the same table, with the same columns and the same retrieval path, and you have not saved yourself any complexity. You have discarded the one piece of information that would have told the agent which of the two to trust. That is the mistake I see most often in agent memory right now, and it is the mistake SmartMemory is built to refuse.

The bucket problem

The default architecture for agent memory in 2026 is depressingly uniform. You take everything the agent encounters: user messages, tool outputs, the agent's own conclusions, documents it read. You embed all of it and drop the vectors into one index. At retrieval you do nearest-neighbor lookup and hand the top k hits to the model as context. It works in the demo, because the demo is short, single-session, and nobody has lied to the agent yet.

The trouble is that a vector is a similarity coordinate and nothing else. It does not record where the text came from, when it was true, or whether a human asserted it, the agent guessed it, or it was a step in some procedure the world has since invalidated. All of that information existed at write time. The flat index discards it, because the flat index has nowhere to put it. By the time you are computing cosine similarity, every memory is a point in space, equally authoritative, equally timeless, and the model reads them all back as the same kind of thing.

They are not the same kind of thing, and humans have known this for a long time. We don't keep one undifferentiated pile either. We remember that we were told something differently from how we hold a fact we believe, and differently again from how to do something. We keep scratch thoughts that evaporate by lunch alongside knowledge we deliberately filed and meant to keep. Collapse all of that into one store and you don't get a simpler agent. You get an agent that cannot tell testimony from truth. That failure stays invisible right up until the testimony turns out to be wrong.

So SmartMemory does not have one store. It has five core memory types, and each one ages and behaves differently on purpose.

Episodic: append-only testimony

Episodic memory is the record that something was said or happened at a particular time. The user told you, on Tuesday, that they cancelled their subscription last month. A tool returned, at 14:32, that the deploy finished. The agent observed something during a session. This is testimony, and testimony has exactly one honest behavior: it is append-only. You never edit it, you never overwrite it, and (the part teams keep getting wrong) you never quietly promote it to a general truth on its own.

The promotion mistake is the one that bites. Take a support agent and a customer who says, in passing, "I cancelled that subscription last month." Recorded as testimony, that line is true the instant it lands. The customer did say it. But "this customer's subscription is cancelled" is a claim about the billing system, not about what was uttered in the chat, and the two can disagree: the cancellation may have failed, or applied to a different plan, or never gone through at all. An episodic system keeps the utterance as an utterance, tagged with who said it and when, and leaves the question of whether billing agrees to a separate check. A flat store that has already laundered the remark into a fact will happily tell the next session the account is cancelled, and refund against it.

This is why episodic memory carries its origin and its timestamp as first-class fields, not as metadata you hope survives a round-trip through serialization. Who said it, when, in which session is not decoration on an episodic memory. It is the point of it.

Semantic: facts that must stay challengeable

Semantic memory is what the agent holds to be true about the world. The account tier. The customer's home region. The deployment topology. These are the memories the agent reasons from, which makes them the most useful category in the store and the most dangerous one.

The temptation is to treat a semantic fact as settled: write it once, read it back forever as ground truth. That is the wrong instinct, because a fact in an agent's memory is not a law of physics. It is a claim that was true given some evidence at some time, and both the evidence and the time can rot out from under it. The home region was right until the customer moved. The topology was right until the migration. Leave a fact in the index with no way to question it and it keeps radiating the same confidence long after it stopped being earned.

So in SmartMemory semantic facts stay challengeable. New evidence does not silently clobber the old fact. It supersedes it through an explicit link that keeps the prior state, so you can always ask what the agent believed, what changed it, and when. A fact is allowed to be retired, contradicted, or re-derived. What it is not allowed to do is sit there unquestionable because nobody built a path to question it. That gap between was said and is true (episodic versus semantic) is the whole wedge in miniature. You only get to know what's true by keeping the testimony separate from the verdict, so the verdict survives the moment the testimony turns out wrong.

Procedural: knowledge that drifts against the world

Procedural memory is how to do something. The runbook. The command. The sequence of steps that resolves a class of incident. It has a property the other types don't share as sharply: it drifts in silence, because the world it describes keeps moving while the memory holds still.

Here is a small one I watched happen. An onboarding agent learned the steps to stand up a local dev environment from a doc a senior engineer wrote: run the seed script, then point the service at config/dev.yaml. Correct, high relevance, retrieved cleanly every time. Then someone split that monolithic config into per-service files and deleted the old path. The seed script now runs against a config that no longer exists, finishes without error, and seeds nothing. The agent keeps confidently handing new hires a procedure that quietly does half of what it claims. The steps were never hallucinated. They came unmoored from the world they assumed, and a similarity score has no opinion about whether that world still exists.

This is the type where a flat vector index is not just lossy but unsafe, because the highest-scoring match can be the most dangerous thing it hands you. The right behavior is to check a procedure against the world it depends on instead of trusting the match. SmartMemory does schema-drift detection on exactly this surface: it snapshots the tools and schemas a procedure relies on, and when it later detects the underlying schema has drifted from that snapshot, it degrades the confidence of the match rather than serving it at face value, and persists the drift event so the change is visible and queryable rather than silent. A procedure does not get retrieved into action just because it looks relevant. It has to still describe a world that exists.

You cannot express that check in a vector. There is no embedding of "true as of the config layout before the split." Drift-checking needs the procedure and the schema it depends on to be distinct, related objects you can compare. That means procedural memory has to be a type, not another row in the same bucket as everything else.

Pending: scratch that should expire

Pending memory is the working scratchpad: the partial, in-flight, not-yet-anything material an agent piles up while it is mid-task. We renamed this from working deliberately, because "working memory" tempted people to treat it as durable, and "pending" is more honest about what it actually is. An intermediate extraction. A half-formed hypothesis. A note the agent left itself two steps ago.

Picture a procurement agent halfway through comparing vendors. It is holding a rough shortlist, three candidates it has half-ranked, a couple of "check this later" flags. That working state is genuinely useful while the task is open, but meaningless the moment the decision is made. The single most important property of pending memory is that it should not be permanent. Most of what an agent generates while working is not knowledge. It is the residue of thinking, and a memory system that files that residue at the same authority as a finished fact will slowly bury the signal under its own half-finished state. Pending is the type that is supposed to fade. It earns no permanence by default. It expires or gets compacted away unless something deliberately graduates it into a durable type.

A flat store has no concept of this. Everything you write to one index is forever by default, so the only way to manage scratch in a single-bucket system is to not write it. Then you lose the in-flight context you needed mid-task. A typed store lets you keep both: hold the shortlist while it is live, let it decay once the decision lands, and never confuse it with the things you meant to remember.

Zettel: curated, gated knowledge

The fifth core type is the zettel: curated knowledge, in the spirit of the Zettelkasten, deliberately filed, linked, and meant to last. It is the opposite of pending on every axis. Where pending is cheap, transient, and auto-generated, a zettel has earned a place in the permanent record, and it is gated on the way in.

The reason to have a curated type at all is that not everything deserves to flow into the durable store on equal footing. A fact the agent inferred in passing is not the same as a piece of knowledge someone decided was worth keeping and wiring into the rest. Making that entry an explicit act (curation, not a side effect of having retrieved something once) is how the high-trust core stays small and trustworthy instead of getting diluted by the firehose of everything the agent ever brushed against. That gate is precisely what a single bucket cannot offer: there is no admission step in a store whose only operation is write the vector and move on.

The extended types, briefly

Five core types are the spine, not the whole skeleton. SmartMemory also carries extended types for the structured things agents increasingly need to remember as first-class objects rather than as prose. A decision, for instance, is its own type, because a decision has a shape: what was chosen, what was rejected, on what basis, whether it was later contradicted or superseded. A free-text blob throws all of that away. There are others: code, plans, evaluations, structured tool calls, and the list grows as we find categories doing real work behind a generic label. The principle holds all the way down. When a kind of memory has its own lifecycle and its own rules for aging and trust, it gets to be a type. Flatten it back into "just text in the index" and you have signed up to relearn its failure modes the hard way.

Why this ends up a graph

There is a structural consequence the five types force, and it is the subject of a later essay, so I'll only plant the flag here. Once memories have distinct types that age differently, the relationships between them start carrying the weight: this testimony is evidence for that fact, this new fact supersedes that old one, this procedure depends on that schema snapshot, this curated note is part of that larger body of knowledge. None of those are similarity relationships, and a flat index speaks exactly one relationship: near. So SmartMemory is graph-centric: the types are nodes, provenance and temporal state are properties, and the relationships that make a memory safe to act on are first-class edges that travel with it instead of evaporating at write time. The full argument for why a graph is the minimum structure that can hold these distinctions is a piece of its own. For now it's enough that the five types don't fit in a bucket, and the reason is edges.

What to do if you're storing agent memory now

A few concrete moves:

Stop using one bucket. If user testimony, derived facts, runbooks, scratch, and curated notes all share one index and one retrieval path, the distinctions that decide what to trust are already gone. Type the store.
Keep episodic append-only, and never auto-promote it. "The user said X" and "X is true" are different memories. Record the first as testimony with full provenance. Require evidence and a deliberate step before you ever write the second.
Make semantic facts challengeable. A fact you can only read and never contradict is a stale fact in waiting. Supersede through explicit links. Never silently overwrite.
Drift-check procedural memory against the world. The highest-scoring runbook is the most dangerous thing you can serve if the world it assumes is gone. Snapshot what a procedure depends on and degrade the match when the dependency has moved.
Let pending expire. Scratch is not knowledge. Default in-flight material to fade unless something deliberately graduates it. Don't file working residue at the authority of finished facts.
Gate curated knowledge. Filing something into the permanent, high-trust core should be an explicit act, not a souvenir of having retrieved it once.
Reach for a graph the moment relationships matter. Supersedes, depends-on, evidence-for, part-of: none of those are similarity. If your memory needs them, and it will, it needs edges.

The reason any of this matters comes back to the sentence I opened with: someone saying a thing once is not the same as the thing being true. A flat vector store cannot hold that difference, and an agent that cannot hold it will act on testimony as if it were fact, on a stale runbook as if the world had stood still, on its own scratch as if it were knowledge. Knowing what's true is hard enough. Knowing what to do (which is what expertise actually is) starts with knowing which of your memories you are even allowed to believe. That is not a feature you bolt on later. It is the type system, and the type system is most of the work.

I'm building SmartMemory, the expertise layer for AI agents: provenance-tagged, bi-temporal, graph-backed memory that knows not just what's true, but what to do, and when it stopped being true.

Try it: pip install smartmemory (docs) · hosted beta (private): smartmemory.ai/signup