Chapter 97 · Memory and state

Storing memory: the options and their trade-offs

Prompt, rows, vectors or a graph. What each one makes easy, what it makes impossible, and the measured reason Atlas ended up with the boring combination.

8 min read

Chapter 96 assumed a table with an embedding column. This chapter is the argument for it, and the honest account of what it gives up.

Four options, and they are not alternatives so much as points on a line from "no infrastructure" to "a second system to operate".

The four

	Scales to	Recall is	Invalidation	Operational cost
In the prompt	~20 memories	Everything, always	Edit the prompt	None
Rows	Thousands	Exact filters only	A `where` clause	A table
Rows + vectors	Millions	Semantic	A `where` clause	An index
Graph	Millions	Multi-hop, relational	Hard	A second system

In the prompt

The option nobody writes down as a decision, and the right one more often than it gets credit for.

// All of a user's memories, concatenated into the system prompt. Under about
// twenty statements this is cheap, exhaustive and trivially debuggable: the
// whole memory state is visible in a single prompt dump.
const block = memories.map((m) => `- ${m.statement}`).join('\n')

It breaks at scale for the reason chapter 61 established — irrelevant context costs accuracy, not just tokens. Atlas measured the cliff by synthetically growing a user's memory set:

Memories in prompt	Groundedness	Preference followed
5	0.97	96%
15	0.96	94%
30	0.93	81%
60	0.88	54%

The right-hand column falls faster than the left, which is the finding worth carrying: with sixty notes in context, the model follows roughly half the stated preferences. Memory that is present and ignored is worse than absent, because the user watches it be ignored and concludes the feature is broken.

Twenty is a reasonable ceiling. Plenty of internal assistants never exceed it.

Rows

-- The whole store, with no vector column. Recall is by exact match on the
-- fields that were extracted at write time.
select * from agent_memory
 where user_id = $1
   and expires_at > now()
   and superseded_by is null
   and (kind = 'preference' or entity_id = any($2));

This is enough when memories are typed and attached to entities — "preferences always, plus anything about the customers named in this question". It is fast, exactly auditable, and it has no semantic recall at all: a memory about "the Northline account" will not surface for a question about "our biggest western customer", even when they are the same company.

Atlas ran this for three weeks. Recall against a hand-built set of 120 question-to-memory pairs was 0.61, and the misses were all of that shape.

Rows plus vectors

-- One column added, one index. The filter stays in the WHERE clause, where
-- chapter 45's rule puts it, and the vector search ranks what survives.
select id, statement, kind, written_at,
       1 - (embedding <=> $3) as score
  from agent_memory
 where user_id = $1
   and expires_at > now()
   and superseded_by is null
 order by embedding <=> $3
 limit 12;

Recall on the same 120 pairs: 0.89. One column, one index, and the system of record stays the same table, so chapter 96's sweep and chapter 94's row level security both apply without change.

That last point decided it. A separate vector store for memory would mean a second place to enforce the user filter, a second place to delete from on a forget request, and two stores that can disagree about what exists. The corpus already made that trade for scale reasons; memory is three thousand rows and does not have to.

// Hybrid helps here too, for the same reason as chapter 43: identifiers and
// account codes are lexical, and embeddings are poor at them.
const results = await hybrid({
  vector: await embed(question),
  keyword: question,
  weights: { vector: 0.7, keyword: 0.3 },
  filter: { userId, live: true },
})

Hybrid took the 120-pair recall from 0.89 to 0.94, and every one of the recovered misses contained an account code or an order number.

A graph

The appealing option, and the one that costs the most to be wrong about.

// Memory as entities and relations instead of statements.
//   (user:alice) -[covers]-> (region:west)
//   (customer:northline) -[terms]-> (net45) [since: 2026-06-01]
//   (alice) -[corrected]-> (fact:northline-terms)

What it genuinely buys is multi-hop recall — chapter 47's problem, answered by traversal rather than by iterated retrieval. "Which of my customers changed terms this year" is a query in a graph and a hard retrieval problem in a vector store.

What it costs:

// 1. Extraction. Every memory must be parsed into typed entities and
//    relations. The extractor is a model call with a precision problem,
//    and a wrong edge is worse than a wrong sentence — it is reachable
//    from queries that have nothing to do with the original conversation.

// 2. Schema. Every relation type is a decision someone has to make before
//    the memory can be stored, and the set grows forever.

// 3. Invalidation across edges. Superseding a statement is one row. The
//    graph version has to decide what else the removed edge invalidated,
//    and chapter 96's sweep has no obvious graph analogue.

// 4. A second system, with its own backups, its own access control, and
//    its own deletion path for chapter 94's forget request.

Atlas prototyped it and did not ship it. The measurement that settled it: of 1,340 eligible turns, 31 would have been answered better by multi-hop memory traversal. Two point three percent, against a second datastore and a four-way invalidation problem. The same questions were answerable by asking the user one clarifying question.

A graph is the right answer when relations between remembered facts are the product. For an assistant whose memory is a few thousand sentences about preferences and context, it is a second system bought with 2.3% of turns.

What Atlas ended up with

// Boring, and the boringness is the feature.
//   - one Postgres table, same database as everything else
//   - pgvector column + hnsw index, alongside a keyword index
//   - row level security on user_id
//   - hybrid recall, capped at 6
//   - preferences bypass ranking while there are fewer than 10

The last line is the small hybrid of options one and three: preferences are few, always relevant and cheap, so they go in the prompt unconditionally. Everything else is retrieved. That split took preference-followed from 81% to 97% at thirty total memories, because the preferences stopped competing with context notes for ranking slots.

Key points

Four options on a line from no infrastructure to a second system: prompt, rows, rows plus vectors, graph.
In-prompt memory is exhaustive and debuggable and breaks around twenty. At sixty, preference-following fell to 54% — present and ignored is worse than absent.
Rows alone gave 0.61 recall. Every miss was a question phrased differently from the stored statement.
Adding a vector column took it to 0.89, and hybrid to 0.94. The recovered misses all contained identifiers.
Keeping memory in the same database kept one filter, one deletion path and one invalidation sweep. A separate vector store doubles all three.
A graph buys multi-hop recall and costs extraction, schema, edge invalidation and a second system. It would have improved 2.3% of Atlas's eligible turns.
Split by kind: preferences unconditionally in the prompt while they are few, everything else retrieved. That took preference-following from 81% to 97%.

Next: letting a user see and edit what it remembers — the control that makes any of this acceptable.