Identity Is Harder Than Memory

Partway through the first essay in this series, I dropped in two customers named Sam Chen, let the line do its work, and moved on. I want to come back to it now, because I buried the lede the first time. The two Sam Chens aren't a data-cleaning problem. They're an access-control problem, and that changes everything about how you have to build the part of the system that tells them apart.

Here are the two. One is on the free tier, keeps a private repo of weekend projects, and forgets the agent exists for weeks at a stretch. The other runs platform engineering at an enterprise account, and his every "remember this" is a load-bearing operational decision someone will eventually deploy against production. They share a name. They share nothing else. The entire promise of an agent memory system (that it knows who you are and what you've told it) rests on never confusing the two.

Confuse two customer records in a CRM and it's a contained embarrassment: someone notices the wrong billing address, you unmerge them, you apologize. Annoying, recoverable. What I did not appreciate until I started building SmartMemory, a memory infrastructure layer for AI agents, is that the same mistake in a memory system is not an embarrassment. It is a breach. And it is a breach that produces no error, no exception, no anomaly in any dashboard. It looks, from every angle the logs can see, completely normal.

So the thing I want to convince you of is narrow and specific. Entity resolution is not a nicety you bolt on after the interesting work is done. In a memory system it is an access-control mechanism, and treating it as anything less is how you ship a cross-tenant leak that nobody finds for months.

Memory without identity is just a pile of facts

Start from what memory is supposed to do. The whole reason you build one is so the agent can act on accumulated context instead of re-deriving the world every session. "Sam prefers Terraform over CloudFormation." "Sam has admin on the staging cluster." "Sam decided last quarter that we standardize on Postgres." Those are not trivia. They are inputs to decisions, permissions, and generated code. The agent reads them back and does something with them.

But every one of those facts is meaningless, worse than meaningless, dangerous, without a reliable answer to the question whose fact is this? "Has admin on the staging cluster" attributed to the wrong Sam is not a low-quality memory. It is an authorization statement pointing at the wrong principal. The fact is fine. The subject is wrong. And in a system that reads facts back as a basis for action, getting the subject wrong is the only kind of wrong that matters.

This is why I now believe identity is harder than memory, and more important. Storing a fact durably, tagging it with provenance and time, retrieving it by relevance: those are solved problems with known shapes. We borrowed them from databases and ledgers. But deciding that two mentions of "Sam Chen" refer to the same human being is a judgment call made under ambiguity, on partial information, by a process that would much rather guess than admit it doesn't know. Get that judgment wrong and every careful thing you did downstream, the provenance, the timestamps, the supersession links, is meticulously bookkeeping the wrong person's life.

The merge that looks like a memory bug but isn't

Picture how the failure actually arrives, because it never announces itself as an identity problem.

A user reports that the agent "knows things it shouldn't." It recommended a workflow that belongs to a different project. It granted access to a feature the user never had. It referenced a decision the user never made. The natural first hypothesis is a memory bug: bad retrieval, a hallucination, a stale fact. So you go look at retrieval. Retrieval is fine. The memory it surfaced is real, well-formed, properly stored, high relevance score. The provenance is intact. The timestamps are clean. Every individual piece of machinery is working exactly as designed.

The bug is upstream of all of it. At some earlier moment, the system decided that two entities were one entity, and merged them. From that instant forward, what one person told the agent in confidence, including the permissions they hold, lives in a node that another person also reads from. There is no corrupted record to find, because nothing is corrupted. There is one node where there should be two, and it is being served correctly to both of the people it was incorrectly fused from.

This is the failure mode that doesn't trip any of your instincts. A SQL injection looks like an attack. A leaked credential looks like an incident. A bad entity merge looks like the product working as intended. The agent is helpfully recalling context, the retrieval scores are high, the user is getting fast confident answers. The only thing wrong is that the "you" the agent is serving is actually two different people collapsed into one, and the logs will never tell you that, because they record retrievals against a node ID and that node ID is internally consistent. There is no corrupted-data alert for this. There is a customer who eventually notices.

Why naive identity is a leak generator

Here is where the architecture choice you made early, probably without thinking of it as an identity decision, comes back to bill you.

If your memory is keyed on a name, the two Sam Chens are one Sam Chen by construction. Every fact about either of them lands in the same bucket, and the bucket is the leak. This is the crudest version and also, in my experience, the most common, because "use the name as the key" is what you write in the first prototype and never revisit.

The subtler version is worse precisely because it feels sophisticated. You embed each entity mention into a vector and merge the ones that are close. Now identity is a similarity threshold, and a similarity threshold is a dial that trades two kinds of catastrophe against each other. Set it loose and you fold distinct people together. That's the cross-tenant leak. Set it tight and you shatter one person into a dozen fragments, each holding a slice of their history, so the agent forgets things it was explicitly told. There is no value of the dial that is correct, because the dial is answering an access-control question with a geometry heuristic. Two people named Sam Chen who both work in cloud infrastructure will sit closer in embedding space than the same Sam Chen discussing his kid's soccer schedule versus a Kubernetes upgrade. Semantic similarity is not identity. It correlates with identity right up until the moment it correlates you into a breach.

The deeper problem is that fuzzy merge is optimistic. It defaults to merging when it isn't sure, because a merge feels like progress: fewer nodes, more connected graph, richer recall. Optimism is exactly the wrong instinct for an access-control decision. The correct default, the only safe default, is to refuse to merge when you are not sure, and accept the cost of occasionally keeping two nodes that were really one. A system that errs toward splitting forgets a little. A system that errs toward merging leaks. I will take forgetting over leaking every single time, and so will any customer who understands what the alternative actually was.

How we built it, and why the boring part is the point

When we built alias disambiguation into SmartMemory, the temptation was to reach for the clever thing: train a model, tune an embedding space, learn a merge classifier. We did not, and the reason is the one above: a learned similarity score is a confident guesser, and confident guessing is the failure mode, not the feature.

What we shipped instead is deliberately conservative. When the same surface form ("Hudson," say) could plausibly attach to more than one canonical entity already resident in the graph ("Rock Hudson," "Rochelle Hudson"), the system does not pick the closest vector. It scores each candidate by structural evidence: how many typed entity neighbors the ambiguous mention shares with each canonical. Shared, specific, relational context (the films, the co-stars, the projects, the systems an entity is actually connected to in the graph) is a far harder thing to coincidentally match than a name or an embedding. And then the rule that matters: it only merges when one candidate wins by a real margin. When nothing wins clearly, it leaves the surface unresolved and moves on. That refusal is not the resolver giving up. It's the resolver declining to make an access-control decision it can't defend, which is the only correct thing to do with one.

That is the whole philosophy in one behavior. The interesting engineering went into knowing when not to merge. The capability is opt-in and off by default, because a merge is a consequential write, and consequential writes should be something you turn on deliberately, not something that happens to you. You run it (resolve_aliases(disambiguate=True)) when you want it, and you can run it as a dry run first to see what it would merge and, just as importantly, what it would refuse to. The things it refuses are the ones I read first.

The line where identity becomes tenancy

Now the part that turns a data problem into a security one, and it is the part I got viscerally educated on.

Every one of those neighbor lookups (who is this entity connected to, which canonicals could this surface belong to) is a graph query. And a graph query in a multi-tenant system must be scoped to the tenant, or it reads across tenants. When the disambiguation pass goes looking for candidate canonicals that an ambiguous "Sam Chen" might merge into, it had better only be looking at this workspace's Sam Chens. The moment that scope filter is missing, the resolver will happily consider canonicals belonging to a different tenant as merge candidates. And if it merges, you have wired one customer's entity to another customer's graph. That is not a bad merge anymore. That is a cross-tenant join, performed by your own helpful background process, with a high confidence score and a clean log line.

I want to be specific about how this fails, because it does not fail loudly. We caught a version of exactly this class of bug in our own code: a scope-resolution path that, on the unhappy branch, silently fell through to an unscoped read instead of raising. The merge logic was correct. The disambiguation was correct. The scoping quietly evaporated under an error nobody saw, and an unscoped read in a multi-tenant graph is a leak by definition. The fix was not cleverer disambiguation. The fix was to make the scope path fail closed. If we can't establish whose data this is, we refuse to read it, loudly, rather than read everyone's.

We wrote a test that I now think of as the canonical shape of this problem. Workspace A has a "Hudson" that means "Rock Hudson." Workspace B has a "Hudson" that means "Rochelle Hudson." Scoped to A, the resolver sees one candidate and resolves cleanly. Run without the scope filter, it sees two candidates, detects the collision, and abstains. The test asserts that the scoped run and the unscoped run produce different answers, that the workspace filter is doing real work, not decorating a query that would behave the same without it. If that test ever goes green-when-unscoped, the isolation has been silently removed, and we'd rather the build scream than the customer.

So here is the part I keep repeating to anyone who touches this code: a missing scope filter on an entity-resolution query is not a data-quality bug. It is a tenant-isolation bug. The same few dozen lines of code are both surfaces at once, and which one you think you're looking at depends entirely on whether you remembered the filter.

Expertise is per-entity, which is why the subject has to be right

This connects to the thing SmartMemory is actually for, and it is not "another memory system." Knowledge is what's true. Expertise is what to do. The expertise layer is the part that doesn't just store a fact but knows whose fact it is, in what context, and therefore what to do on whose behalf.

And expertise is irreducibly per-entity. The whole value proposition is that the agent learns what works for this person, this service, this project. Not what works in general. We attribute derived judgments to the actor they belong to: our evaluation records are keyed on the identity triple of (agent, dimension, domain), so a performance score about how a particular agent does a particular kind of task in a particular domain is pinned to that agent and not smeared across all of them. That precision is the product. It is also the liability. The instant identity resolution puts the wrong actor on the front of that triple, every "fact to act on" you've carefully accumulated is now attributed to the wrong principal. An expertise layer that confidently applies one entity's hard-won expertise on another entity's behalf isn't a smarter assistant. It's an authorization failure that happens to give good advice to the wrong person.

So the chain is short and unforgiving, and it runs in one direction. Memory is only useful if you know whose it is. Expertise is only useful if it's attributed to the right entity. That attribution is only as good as the entity resolution underneath it. And entity resolution is only safe if its queries are tenant-scoped and it would rather abstain than guess. Pull on the thread that says "we'll just match names for now" and the whole thing unravels into a leak you won't see in the logs.

What to do if you're building this now

If you take nothing else from this, take the six things below. The first one is the whole essay. The rest are how you live up to it.

Treat entity resolution as access control, not data hygiene. The owner of your merge logic should be whoever owns your tenant isolation, because they are the same surface. A bad merge is a cross-tenant leak, full stop.
Default to abstaining, not merging. Forgetting is recoverable. Leaking is not. When the evidence for "these two are one" doesn't clear a real margin, leave them separate and move on. Pay closest attention to what your resolver refuses to merge. That's where it's protecting you.
Don't make identity a similarity threshold. Embedding distance correlates with identity and then betrays you exactly when two real, distinct people are similar, which in any focused customer base is constantly. Use structural, relational evidence that's hard to match by coincidence, and require a margin, not just a maximum.
Scope every resolution query to the tenant, and fail closed. The disambiguation read is a graph query like any other. If the path that establishes whose data you're allowed to see can silently fall through to "everyone's," you have already shipped the leak. Make it raise.
Write the collision test. Two tenants, same surface, different referents. Assert that the scoped run resolves and the unscoped run abstains. Assert that they differ. If they ever stop differing, your isolation is gone and nothing else will tell you.
Pin every derived judgment to a subject. If a score, a preference, or a permission can't name the entity it belongs to, it doesn't belong in a memory an agent will act on.

The reason the title isn't hyperbole is this. Memory's failures eventually announce themselves: a stale fact gets contradicted, a wrong answer gets corrected, a missing memory gets noticed and re-supplied. Identity's central failure does the opposite. A confident bad merge gets more convincing over time, because every subsequent interaction adds context to the fused node and makes the fiction richer. The two Sam Chens don't drift apart in the system. They grow more inseparable, more corroborated, more trusted. Right up until one of them sees something that was never his.

I'd rather ship the agent that's certain whose data it's holding before the one that holds an impressive amount of it. Almost everyone builds those in the wrong order, and the order is the part that turns out to matter.

I'm building SmartMemory, the expertise layer for AI agents: provenance-tagged, bi-temporal, graph-backed memory that knows not just what's true, but what to do, and when it stopped being true.

Try it: pip install smartmemory (docs) · hosted beta (private): smartmemory.ai/signup