How agents remember
Ask people how an agent remembers and most will picture a vector database. The honest answer is duller and better: agent memory is mostly version-controlled text and code files that a human can read, diff, and reject in a pull request. The exotic part, a store the agent writes to on its own, is the smallest part of a working memory system. It is also the only part that carries genuinely new risk.
Get the boring part right first. Then decide, deliberately, how much of the risky part you actually need.
The four kinds of memory
Section titled “The four kinds of memory”The most useful taxonomy comes from cognitive-architecture research, the CoALA paper, which splits agent memory into four types. Working memory holds the current decision cycle; the other three persist beyond it.
| Type | Lifespan | Where it lives | Who can write it |
|---|---|---|---|
| Working | One session | The context window, plus the prompt cache | The agent, every turn |
| Semantic | Until edited | Instructions files and docs in the repo | Humans and agents, gated by review |
| Procedural | Until edited | Skills, scripts, hooks: code in the repo | Humans and agents, gated by review |
| Episodic | Indefinite | A persistent store the agent appends to | The agent, usually ungated |
Three of the four are things you already know how to govern. One is not.
Working memory: the current cycle
Section titled “Working memory: the current cycle”Working memory is the context window: everything the agent is considering right now, kept warm by the prompt cache. It holds the task, the files it has read, the output of what it just ran. It is ephemeral by design: when the session ends, it dies with it. That is a feature. A fresh window per task is what keeps an agent sharp; recall The context window.
Everything an agent “remembers” across sessions is something that was deliberately written down somewhere persistent, then loaded back into a future working memory. Memory, in practice, is a write followed by a later read. Both halves can go wrong.
Semantic and procedural: files under review
Section titled “Semantic and procedural: files under review”Semantic memory is what the agent knows to be true: your stack, your conventions, your constraints. In practice that is an instructions file and the docs beside it: plain text in the repo.
Procedural memory is what the agent knows how to do. CoALA splits it in two: the implicit part is the model’s weights, which you don’t control, and the explicit part is code (skills, scripts, hooks), which you fully control. Every procedure you move from “the model usually does this” into an explicit file is a behaviour you can now version, test, and review.
This is the thesis in one sentence: for semantic and procedural memory, git is the memory system. A change to what the agent believes or how it behaves arrives as a diff, gets reviewed like code, and can be reverted like code. No embedding pipeline gives you that. Most teams reaching for a vector database need a well-tended CLAUDE.md and a skills directory first.
One asymmetry inside this pair matters. Letting an agent write its own procedural code is the more dangerous write of the two: a wrong fact in semantic memory misleads the agent, but wrong procedural code changes what the agent does. It can introduce bugs or quietly work around the designer’s intent. CoALA’s own guidance is that procedural memory should be initialised by a human and changed deliberately. So the pull-request gate matters most here: “reviewable” is the safeguard, not a licence to let an agent append to its own skills unwatched.
Episodic: the risky one
Section titled “Episodic: the risky one”Episodic memory is the record of what happened: past sessions, decisions, learned facts, written by the agent into a persistent store. It is the one type where the writer is the agent itself and the write is usually ungated. That combination creates two failure modes that files-under-review don’t have:
- Drift. A fact that was true in March is loaded, unquestioned, into a session in July. Stale memory doesn’t announce itself; it just quietly steers.
- Poisoning. Anything an agent reads can end up in what it writes. Research on memory poisoning shows that a manipulated entry, once stored, can persist across sessions and keep influencing behaviour, succeeding on a large share of attempts in the studies that raised the alarm. A poisoned prompt hits one session; a poisoned memory hits every session after it.
The maintenance loop
Section titled “The maintenance loop”A memory system is a garden you tend, not a database you fill. The loop:
- Capture note the candidate fact cheaply
- Distill compress to the load-bearing sentence
- Store put it where its type belongs
- Retrieve load only what the task triggers
- Verify check it still holds before acting
- Prune delete what went stale
↺Prune feeds back into capture. A memory system is a garden you tend, not a database you fill.
- Capture. Note the candidate fact or lesson when it emerges, cheaply.
- Distill. Compress it to the load-bearing sentence. Chroma’s context-rot research found that a focused prompt of roughly 300 tokens outperformed handing the model the full ~113k-token history on LongMemEval. Distilled beats raw, by a wide margin.
- Store. Put it where its type belongs: convention → instructions file, procedure → skill or script, event → episodic store.
- Retrieve. Load only what the current task triggers, not the whole archive.
- Verify. On read, check the fact still holds before acting on it.
- Prune. Delete what went stale. A memory nobody prunes converges on being wrong.
Most teams do steps 1 and 3 and skip the rest. Capture without distillation fills the window with noise; storage without pruning fills it with lies.
The write gate is the design decision
Section titled “The write gate is the design decision”Deciding how an agent remembers reduces to one question per memory type: who reviews the write? Working memory needs no gate; it dies. Semantic and procedural memory get the strongest gate you already own: pull-request review. Episodic memory is where you must build the gate yourself, and where verification on read is the deterministic check that catches what the gate misses. That check belongs to the layer we cover next.