Chapter 92 · Memory and state

Conversation state inside one session

The entity in focus, the filters in play, and the format they asked for. Three slots, resolved before retrieval, discarded at the end — and why replaying raw history is the expensive way to get it wrong.

8 min read

Chapter 91 measured what follow-up questions actually carry: an entity, a filter, a format preference. This chapter builds the thing that carries them, and it is deliberately small.

The obvious implementation is to append every turn to a list and send the list. It works at turn three and degrades from there, for reasons that are worth being precise about.

Why replaying raw history fails

// The version everyone writes first.
const messages = [
  { role: 'system', content: SYSTEM_PROMPT },
  ...session.turns.flatMap((t) => [
    { role: 'user' as const, content: t.question },
    { role: 'assistant' as const, content: t.answer },
  ]),
  { role: 'user' as const, content: question },
]

Three costs, measured on Atlas's six-turn sessions:

Turn	Prompt tokens	Latency p50	Groundedness
1	4,100	2.1 s	0.97
3	11,800	3.4 s	0.96
6	26,400	6.9 s	0.89

Cost and latency grow with the square of session length, because every turn resends every earlier turn. That part is predictable. The groundedness fall is the interesting one: by turn six the context holds five earlier answers, each full of retrieved passages that are no longer relevant, and the model draws from them. Chapter 61 showed the same effect with distractor chunks. Old answers are distractors with extra credibility, because the model wrote them.

An earlier answer in context is a passage the model already believes. Replaying history is the most expensive way to fill a context window with material that cannot be corrected by retrieval.

Three slots instead

// Everything the measured follow-ups needed, and nothing else. It is small
// enough to render into the prompt in about forty tokens.
interface SessionState {
  focus?: { type: 'customer' | 'order' | 'product' | 'rep'; id: string; label: string }
  filters: { period?: string; region?: string; status?: string }
  format?: 'short' | 'detailed' | 'table'
  recentQuestions: string[]   // last 3, text only, for rewriting
}

recentQuestions carries questions and no answers. That is the whole trick: a question is what the follow-up refers back to, and it costs twenty tokens; an answer is what pollutes the context, and it costs two thousand.

// Rendered into the system prompt as a compact block. Chapter 53 argued the
// prompt is an artifact — this is the one part of it that varies per turn, so
// it is fenced off where a diff will show it.
function renderState(s: SessionState): string {
  const lines: string[] = []
  if (s.focus) lines.push(`Currently discussing: ${s.focus.type} "${s.focus.label}" (${s.focus.id})`)
  const f = Object.entries(s.filters).filter(([, v]) => v)
  if (f.length) lines.push(`Filters in effect: ${f.map(([k, v]) => `${k}=${v}`).join(', ')}`)
  if (s.format) lines.push(`Answer style requested: ${s.format}`)
  if (s.recentQuestions.length) {
    lines.push(`Earlier in this session: ${s.recentQuestions.map((q) => `"${q}"`).join('; ')}`)
  }
  return lines.length ? `## Session context\n${lines.join('\n')}` : ''
}

Same six-turn session, rebuilt this way:

Turn	Prompt tokens	Latency p50	Groundedness
1	4,100	2.1 s	0.97
3	4,350	2.2 s	0.97
6	4,420	2.2 s	0.96

Flat, which is what a bounded state object buys. The groundedness at turn six is within noise of turn one.

Updating the slots

State changes in two ways: the user says something that changes it, and a tool returns something that changes it.

// From the user's question, before retrieval. This is chapter 46's query
// rewriting doing double duty — it already has to resolve "their" into a
// customer name, so it emits the resolution rather than discarding it.
interface Resolution {
  rewritten: string
  focus?: SessionState['focus']
  filters: Partial<SessionState['filters']>
  format?: SessionState['format']
}

// From a tool result. A successful lookup is the strongest possible signal
// about what the conversation is now about, and it costs nothing to capture.
function afterTool(state: SessionState, call: ToolCall, result: ToolResult): SessionState {
  if (call.name === 'lookup_customer' && result.ok) {
    return { ...state, focus: { type: 'customer', id: result.data.id, label: result.data.name } }
  }
  if (call.name === 'list_orders' && result.ok && call.args.period) {
    return { ...state, filters: { ...state.filters, period: call.args.period } }
  }
  return state
}

The asymmetry is deliberate. Tool results set focus confidently because an identifier came back from a system of record. User text sets focus tentatively, because "Northline" might be three customers, which is the next problem.

Ambiguity, and the one question worth asking

// Two customers match "Northline". The wrong move is to pick the more recent
// and carry it silently for the rest of the session, because every later
// answer inherits the error and none of them mention it.
if (candidates.length > 1) {
  return {
    kind: 'clarify',
    question: `Two customers match "Northline". Which one?`,
    options: candidates.map((c) => ({ id: c.id, label: `${c.name} — ${c.city}, acct ${c.account}` })),
  }
}

A clarification is a turn the user pays for, so the bar is that getting it wrong is expensive and silent. A wrong focus is exactly that: it corrupts every subsequent turn, and the user cannot see it. A wrong format preference is neither, so it is never worth a question.

Atlas asks one clarifying question in roughly 4% of sessions. Pushing that rate higher made satisfaction fall — confirmation flows found the same shape, for the same reason.

Letting state decay inside the session

State that survives a topic change is worse than no state.

// Focus decays when the new question is clearly about something else. The
// cheap detector: the question names a different entity, or shares almost no
// content words with the last three.
function shouldDropFocus(state: SessionState, question: string): boolean {
  if (!state.focus) return false
  if (mentionsOtherEntity(question, state.focus)) return true
  const overlap = contentWordOverlap(question, state.recentQuestions.join(' '))
  return overlap < 0.12
}

// Filters decay faster than focus, because "same thing for last quarter"
// means the period, and three turns later "and for Northline" almost never
// still means last quarter.
const FILTER_TURNS = 2

Both thresholds came from reading sessions where the answer was wrong and asking which stale slot caused it. The 0.12 overlap threshold was the difference between dropping focus on 3% of genuine follow-ups (annoying, recoverable, the user repeats the name) and carrying it into 11% of genuine topic changes (silent, wrong, unrecoverable). Failing in the recoverable direction is the same choice chapter 62 made about spans.

Where it lives

// Session state is small, hot, and worthless after an hour. It is the one
// thing in this series that genuinely wants a cache rather than a table.
await redis.set(`session:${sessionId}`, JSON.stringify(state), { EX: 3600 })

Losing it costs one repeated question. That is the correct risk posture for a one-hour object, and it is a different posture from resumability, where losing a suspended run loses real work. Two kinds of state, two storage decisions, and conflating them is how a session cache ends up with a durability requirement nobody needed.

Key points

Replaying raw history grows cost with the square of session length and falls to 0.89 groundedness by turn six, because old answers act as distractors the model already trusts.
Three slots covered the measured follow-ups: focus, filters, format. Plus the last three question texts, with no answers.
Bounded state held tokens, latency and groundedness flat from turn one to turn six.
Tool results set focus confidently; user text sets it tentatively. An identifier from a system of record is better evidence than a name.
Clarify only when a wrong guess is both expensive and invisible. A wrong focus is; a wrong format is not. Atlas clarifies in 4% of sessions.
Decay focus on topic change and filters after two turns. Fail toward dropping state rather than carrying it.
Session state belongs in a cache with an hour's TTL. Losing it costs one repeated question.

Next: long-term memory across sessions — where losing it costs something real.