Chapter 89 · Agent loops and control

Making a run's trace legible to a human

A trace nobody can read is a log. Three audiences with three different needs, the four questions a trace must answer in under a minute, and the detail that made it unreadable.

10 min read

Every chapter in this section has added to the trace. This one makes it readable, which is a different problem from making it complete.

The test is concrete: given a run that produced a bad answer, can someone find the cause in under a minute? On Atlas's first implementation the answer was no, and the trace contained everything needed.

Three audiences

Audience	Needs	Depth
The user, now	What was checked, what was not	One line per step
An engineer, later	Why a step did what it did	Everything
An auditor, much later	Who did what to which record	The write audit

One artefact cannot serve all three, and trying produced the unreadable version. Three renderings of one stored trace is the answer.

For the user: the activity line

Shown during the run and collapsible afterwards.

Looking into the Henderson account…
  ✓ Account summary                        0.3s
  ✓ Open invoices — 3 found                0.4s
  ✗ Payment history — system unavailable   2.5s
  ✓ Open tickets — 1 found                 0.4s

// Derived from the trace, one line per step, no arguments and no ids.
export function userTrace(trace: TraceEntry[]): UserLine[] {
  return trace.map((e) => ({
    label: TOOLS[e.tool].userLabel,          // "Payment history", not get_payment_history
    outcome: e.result.ok ? (isEmptyResult(e.result) ? 'none' : 'ok') : 'failed',
    // ⚠️ A count, not the data. "3 found" is useful; three invoice rows
    // in an activity line is noise.
    detail: summariseForUser(e.result),
    ms: e.latencyMs,
  }))
}

Two properties matter. It uses the tool's userLabel rather than its name, and a failed step is visible rather than hidden — which is what makes the partial answer's caveat corroborated rather than merely asserted.

For the engineer: the four questions

A trace must answer these four in under a minute, and the rendering should be organised around them rather than around chronology.

export interface EngineerTrace {
  // 1. What did it decide, and why that route?
  route: { kind: RouteKind; method: 'pattern' | 'neighbour' | 'model'; confidence?: number }

  // 2. What did it see at each step? ⚠️ The INPUT, which is what is
  //    usually missing — a trace of outputs cannot explain a decision.
  steps: Array<{
    n: number
    contextTokens: number
    toolsOffered: string[]           // chapter 67's per-request filter
    called: { tool: string; arguments: unknown }
    result: { ok: boolean; class?: string; rows?: number; digest: string }
    latencyMs: number
    costUsd: number
  }>

  // 3. What stopped it?
  termination: { reason: string; detector?: string }

  // 4. Where did the answer's claims come from?
  attribution: Array<{ claim: string; citation: number; source: string }>
}

Question 2 is the one first implementations get wrong. Recording what a tool returned explains what happened; recording what was offered and what the model could see explains why it chose. A trace showing check_inventory was called tells you nothing if you cannot see that lookup_discount_tiers was filtered out of the surface that turn.

// Which makes this one field unusually valuable.
toolsOffered: selectTools(question, scope, { maxTokens: BUDGET.tools }).map((t) => t.name),

On Atlas, toolsOffered resolved four "why didn't it use the right tool?" investigations in minutes each. The answer three times was that the filter had not offered it.

The rendering

run 01HQ3F…  user: rep-14  route: planning (model, 0.88)  4 steps  3.4s  $0.021
question: "is the Henderson discount being applied to their current orders?"

 1  tools offered: get_account_summary, get_open_invoices, check_inventory,
                   lookup_discount_tiers, search_tickets  (5 of 14)
    → get_account_summary { accountName: "Henderson" }
      ⚠ ambiguous: 2 candidates (4471 Henderson Industrial, 4480 Henderson Pacific)
    0.2s  $0.004  ctx 2,940

 2  → get_account_summary { accountId: "4471" }          [user clarified]
      ok  1 row  asOf 14:22:01
    0.3s  $0.005  ctx 4,180

 3  ∥ get_open_invoices { accountId: "4471" }     ok  3 rows  asOf 14:22:03
    ∥ get_contract_terms { accountId: "4471" }    ok  1 row   asOf 14:22:03
      ⚠ parallel set, spread 1.8s — within tolerance
    0.9s  $0.006  ctx 6,720

 4  → (no tool)  answered
    1.9s  $0.006  ctx 7,010

terminated: answered
attribution:
  "discount is 12% above $25,000"      → [1] contract terms, step 3
  "two of three orders have it"        → [2] open invoices, step 3
  "ORD-8821 is below the threshold"    → computed, not retrieved

Four things in that rendering earn their place.

The refusal at step 1 is shown, not hidden as a retry. A trace that only shows successful calls cannot explain why step 2 has a different argument.

Parallel calls are marked ∥ with their spread, which is chapter 88's consistency check made visible.

Context size per step shows the quadratic growth and is where an unexpectedly expensive run explains itself.

Attribution distinguishes computed from retrieved. "ORD-8821 is below the threshold" came from code, and knowing that is the difference between suspecting the model and checking the comparison.

What made it unreadable

Atlas's first trace stored full tool results and full message arrays per step. A four-step run was 40 KB of JSON, and finding the cause of a bad answer meant reading it.

// ❌ Everything, inline. Correct, complete, and nobody reads it.
trace.push({ step, request, fullResult: result, messages: [...run.messages] })

// ✅ A digest inline, the full payload addressable.
trace.push({
  step,
  called: { tool, arguments: redact(args) },
  result: { ok: result.ok, class: result.class, rows: rowCount(result),
            digest: sha256(result).slice(0, 12) },
  payloadRef: await storePayload(runId, step, result),    // ⚠️ fetched on demand
})

// And the payload store has its own retention, shorter than the trace's,
// because the digests remain useful long after the rows do not.
export const PAYLOAD_RETENTION_DAYS = 7
export const TRACE_RETENTION_DAYS = 30

The digest is what makes the compressed version sufficient for most investigations: two steps with the same digest is repetition, visible without reading either payload.

Redaction

// ⚠️ A trace contains retrieved content, account data and tool arguments.
// It is read by engineers, so it is subject to chapter 16's logging rules.
export function redact(value: unknown): unknown {
  // Identifiers and names are kept — they are what makes a trace useful.
  // Content is not: a digest and a row count suffice.
  return mapDeep(value, (v, key) =>
    CONTENT_FIELDS.has(key) ? `[${String(v).length} chars]` : v)
}

const CONTENT_FIELDS = new Set(['content', 'body', 'notes', 'text', 'resolution_notes'])

Diffing two runs

The investigative tool that pays for itself the first time it is used.

// "It worked yesterday." Two run ids, one diff.
export function diffRuns(a: EngineerTrace, b: EngineerTrace): RunDiff {
  return {
    route: a.route.kind !== b.route.kind ? { from: a.route.kind, to: b.route.kind } : null,
    // ⚠️ The most common cause, and the least obvious: the tool surface
    // changed because the filter ranked differently.
    toolsOffered: diffSets(a.steps[0]?.toolsOffered, b.steps[0]?.toolsOffered),
    path: diffSequences(a.steps.map((s) => s.called.tool), b.steps.map((s) => s.called.tool)),
    versions: {
      prompt: a.promptVersion !== b.promptVersion,
      toolSurface: a.toolSurfaceVersion !== b.toolSurfaceVersion,
      model: a.model !== b.model,
    },
  }
}

The versions block is why chapter 53 and chapter 68 hash their artefacts: without it, "it worked yesterday" is unanswerable.

The failure mode: the trace that was written for the happy path

Atlas's trace is good. Then a run produces a wrong answer and the trace shows four successful steps, a reasonable path, and nothing unusual.

The cause was at step 2: the model was offered eleven tools instead of the usual five, because the per-request filter fell back to the full surface when the relevance ranking errored. Correct-tool accuracy at eleven tools is 0.96 rather than 0.99, and this was one of the 4%.

The filter's error was caught and logged — in the application log, with no run id, in a different system from the trace.

// ⚠️ Anything that changes a decision belongs IN the trace, including
// the degradations that were handled successfully.
trace.push({
  step,
  toolsOffered: selected.map((t) => t.name),
  // The field that would have made this a two-minute investigation.
  degradations: [
    ...(filterFailed ? [{ kind: 'tool_filter_fallback', detail: 'relevance ranking failed' }] : []),
    ...(cacheMiss ? [{ kind: 'prompt_cache_miss' }] : []),
    ...(fellBackToSmallModel ? [{ kind: 'model_fallback' }] : []),
  ],
})

// And a rendering rule: a step with degradations is marked, so a reader
// scanning a trace sees it without looking for it.
//  2 ⚠ → get_account_summary { ... }      [tool_filter_fallback]

A handled degradation is invisible by design and changes behaviour anyway. Record every fallback in the trace, not only every failure — the run that silently took a worse path is the one you will spend an afternoon on.

Key points

A trace nobody can read is a log. Three renderings of one stored trace, for three audiences.
The user's line shows failed steps, which is what corroborates the answer's caveats.
Record what the model saw, not only what it returned. toolsOffered resolved four investigations on its own.
Show refusals and retries, mark parallel sets with their spread, show context size per step, and distinguish computed claims from retrieved ones.
Store a digest inline and the payload by reference, with shorter retention. Two identical digests is repetition, visible without reading either.
Redact content fields; keep identifiers, which are what make a trace useful.
Diff two runs, including the prompt, tool-surface and model versions. That is what makes "it worked yesterday" answerable.
Record handled degradations, not only failures. A silent fallback to a worse path is the expensive investigation.

Next, and the last chapter of this section: the control-flow bugs that only appear in production.