Chapter 90 · Agent loops and control

The control-flow bugs that only appear in production

Seven bugs that every test passed. What they have in common, why a test suite cannot find them, and the three production-only signals that can.

12 min read

The last chapter of section 7. Everything in it is a bug that shipped with a green test suite.

They share a structure worth naming up front: each requires a combination that a test fixture does not produce — real concurrency, real elapsed time, a real distribution of inputs, or a real second user.

1. The step limit that was per-process

// ❌ A module-level counter. Correct in a test, which runs one run at a
//    time. In production, concurrent runs share it.
let steps = 0
export function checkLimit() { return ++steps > MAX_STEPS }

Symptom: runs halting at step 1 under load, and never in testing. Every unit test passed; every integration test passed; the suite never ran two runs concurrently.

// ✅ State lives on the run. Chapter 81's Budget object, instantiated
//    per run, which is why it is a class rather than a module.
const budget = new Budget(run.limits)

Any mutable state outside the run object is a concurrency bug waiting for a second user. Grep for module-level let in the agent path.

2. The context that leaked between runs

// ❌ A default parameter evaluated once, at module load.
export function newRun(question: string, scope: Scope, ctx: RunContext = { called: [] }) {
  // ⚠️ Every run without an explicit ctx shares ONE array.
}

Symptom: preconditions passing when they should not, intermittently, more often under load. A ticket opened without a search because a different user's run had called search_tickets.

// ✅ A factory, so the default is constructed per call.
export function newRun(question: string, scope: Scope, ctx = freshContext()) {}
export const freshContext = (): RunContext => ({ called: [], toolResults: [], runId: randomUUID() })

// And a test that catches the class rather than the instance.
test('two runs do not share context', () => {
  const a = newRun('q1', SCOPE.insideSales)
  const b = newRun('q2', SCOPE.insideSales)
  a.context.called.push('search_tickets')
  expect(b.context.called).toEqual([])
})

3. The suspended run that resumed into a deploy

A run suspends awaiting confirmation. A deploy ships. The rep approves.

The stored messages reference a tool that no longer exists — renamed in the deploy — and the resumed loop sends the model a conversation containing a tool_use block for an unknown tool.

// ⚠️ Validate the stored conversation against the CURRENT tool surface
//    before resuming. A rename is a breaking change for suspended runs.
export function canResume(run: Run): ResumeCheck {
  const referenced = toolsReferencedIn(run.messages)
  const missing = referenced.filter((t) => !TOOLS[t])
  if (missing.length) {
    return { ok: false, reason: 'tool_surface_changed', missing }
  }
  // The surface version is recorded per chapter 68.
  if (run.toolSurfaceVersion !== TOOL_SURFACE_VERSION) {
    logger.warn('resume.surface_drift', { runId: run.id, from: run.toolSurfaceVersion })
  }
  return { ok: true }
}

// And the graceful answer, which is better than an error:
return { ok: false, error: 'stale_run',
  message: 'This request is out of date. Please ask again.' }

4. The idempotency key that included a mutable object

// ❌ `scope` is passed by reference and mutated during the run —
//    chapter 76's scope lock sets lockedAccount on it.
const key = sha256(JSON.stringify({ tool, args, scope }))

Symptom: the same call before and after the lock produced different keys, so idempotency stopped deduplicating exactly in the multi-step runs where retries happen.

// ✅ The key covers the intent, and nothing that changes during a run.
const key = sha256(JSON.stringify({ tool, arguments: canonicalise(args), runId }))

The deeper rule: anything in an idempotency key must be immutable for the life of the key. A test that computes the key twice in a row cannot find this; a test that computes it before and after a step can.

5. The timeout that was not inherited

// ❌ The run has 10 seconds. Step 1 takes 7. Step 2's tool uses its own
//    2.5s timeout and succeeds at 9.5s. Generation then runs unbounded.
const result = await tool.run(args, scope)

Symptom: p99 latency of 31 seconds against a 10-second limit, with the limit "working" in every test — because test runs never had a slow first step.

// ✅ Chapter 81's remainingMs, threaded through. A tool that cannot
//    finish in the time left should not start.
const allowance = budget.remainingMs() - GENERATION_RESERVE_MS
if (allowance < 500) return { ok: false, error: 'no_time_budget' }
const result = await tool.run(args, scope, { signal: AbortSignal.timeout(allowance) })

6. The retry storm behind a confirmation

A tool fails. The model retries once. The retry needs confirmation. The run suspends.

The rep approves. The resumed run retries, fails again, retries once more — because ctx.retries was not persisted with the run, so resumption reset it.

// ⚠️ Every counter that bounds behaviour must survive suspension, not
//    only the budget. Chapter 87 persisted spend; this is the same rule
//    applied to everything else that counts.
export interface RunContext {
  called: string[]
  retries: Record<string, number>      // persisted
  failedCalls: number                  // persisted
  nudgedOn: string[]                   // persisted — chapter 80
  replans: number                      // persisted — chapter 82
}

test('counters survive suspension', async () => {
  const run = await runUntilSuspended(withFailingTool())
  expect(run.context.retries['open_ticket']).toBe(1)
  const resumed = await resumeAfterConfirmation(run.pending_token, 'approve')
  expect(resumed.context.retries['open_ticket']).toBe(1)   // ⚠️ not 0
})

7. The trace that grew until the run failed

Long runs started failing at step 5 with a database error. The trace column is JSONB, and a run storing full tool payloads inline exceeded the row size on investigations that returned large result sets.

The bug is chapter 89's payload reference, and the reason it reached production is that test fixtures return three rows and production returns three hundred.

// ⚠️ A guard, because the fix can regress. The trace is bounded and a
//    run that would exceed it degrades rather than failing.
export function appendTrace(run: Run, entry: TraceEntry) {
  const size = JSON.stringify(entry).length
  if (run.traceBytes + size > MAX_TRACE_BYTES) {
    run.trace.push({ ...entry, result: { ...entry.result, truncated: true } })
    logger.warn('trace.truncated', { runId: run.id, step: entry.step })
    return
  }
  run.trace.push(entry)
  run.traceBytes += size
}

What they have in common

// None of the seven is findable by a test of a single run with fixture
// data. The four ingredients a test suite does not supply:
//
//   1. Concurrency        — bugs 1, 2
//   2. Elapsed real time  — bugs 3, 5
//   3. Real data volume   — bug 7
//   4. State across a     — bugs 4, 6
//      lifecycle boundary

Three of them can be partially bought back in testing.

// Concurrency: run the suite's integration tests in parallel, deliberately.
test.concurrent.each(CASES)('run %s', async (c) => { /* ... */ })

// Lifecycle boundaries: a test that suspends, restarts the process and
// resumes. Slow, worth having once.
test('survives a restart', async () => {
  const run = await runUntilSuspended(withConfirmation())
  await restartWorker()                    // ⚠️ actually restart it
  const resumed = await resumeAfterConfirmation(run.pending_token, 'approve')
  expect(resumed.state.kind).toBe('answered')
})

// Data volume: fixtures sized like production, not like a demo.
export const FIXTURES = { largeAccount: { orders: 340, invoices: 112, tickets: 58 } }

The fourth — elapsed real time — is the one testing cannot buy, and it is why the signals below matter.

Three production-only signals

-- 1. Step distribution. A shift is the earliest indicator of a control-
--    flow change, and it moves before correctness or cost. Chapter 81.
select date_trunc('hour', started_at) as hour,
       round(avg(steps), 2) as avg_steps,
       percentile_cont(0.95) within group (order by steps) as p95_steps
  from runs where started_at > now() - interval '48 hours'
 group by 1 order by 1 desc limit 12;

-- 2. Path diversity. A sudden collapse to one path, or an explosion of
--    distinct paths, both mean something changed.
select date_trunc('day', started_at) as day,
       count(distinct path) as distinct_paths,
       count(*) as runs,
       round(count(distinct path)::numeric / count(*), 3) as diversity
  from (select id, started_at,
               (select string_agg(tool, '>' order by step) from trace_steps t
                 where t.run_id = r.id) as path
          from runs r where started_at > now() - interval '14 days') p
 group by 1 order by 1;

-- 3. ⚠️ Terminal-state mix. The most sensitive of the three. A rise in
--    step_limit or failed is a control-flow regression; a rise in
--    suspended is usually a product change.
select date_trunc('day', started_at) as day, state, count(*)
  from runs where started_at > now() - interval '14 days'
 group by 1, 2 order by 1, 3 desc;

// And an alert on the mix rather than on any single state.
const baseline = { answered: 0.88, suspended: 0.07, halted: 0.045, failed: 0.005 }
for (const [state, expected] of entries(baseline)) {
  const actual = await stateShare(state, { hours: 2 })
  if (Math.abs(actual - expected) > expected * 0.5 && runsInWindow > 100) {
    alert(`terminal state '${state}' at ${pct(actual)}, baseline ${pct(expected)}`)
  }
}

Section 7 is done

	Start	End
Runs hitting the step limit	9%	0.2%
Median steps	4.1	2.4
p50 loop latency	6.2 s	3.4 s
Cost per loop run	$0.041	$0.019
Correct (loop runs)	0.86	0.95
Partial runs answered honestly	0%	9.7%

Section 8 is memory: what an agent should remember, for how long, and the attack surface that creates.

The failure mode: the bug that was fixed by a retry

The meta-failure, and the reason this chapter exists.

Each of the seven produced intermittent symptoms. Each was reported as "it sometimes does not work". For five of the seven, the first response from someone was to add a retry — and a retry made the symptom rarer without changing the cause.

Bug 2, the shared context, was "fixed" that way for three weeks. The duplicate-ticket rate fell from 11 a week to 3, which read as progress.

// ⚠️ A retry that reduces a symptom without a named cause is a bug
//    being hidden. The rule that caught the remaining six:
//
//    Before adding a retry, write down which of the five tool-error
//    classes this is. If it is not 'correctable' or 'unavailable', it
//    is not a retryable failure — it is a bug.

That rule is chapter 77's classification used as a diagnostic rather than as a handler, and it is the cheapest debugging discipline in this section: an intermittent failure that does not fit a class is a control-flow bug, and retrying it buys time at the cost of the evidence.

Key points

Seven production bugs, all with a green suite. Each needed concurrency, elapsed time, real data volume, or state crossing a lifecycle boundary.
Any mutable state outside the run object is a concurrency bug waiting for a second user.
A default parameter is evaluated once at module load. Use a factory.
Validate stored conversations against the current tool surface before resuming. A rename breaks suspended runs.
Anything in an idempotency key must be immutable for the key's life.
Thread the remaining time budget into every call, including generation.
Every bounding counter must survive suspension, not only the spend.
Bound the trace and degrade rather than failing. Fixtures return three rows; production returns three hundred.
Buy back concurrency, lifecycle boundaries and data volume in testing. Elapsed time you cannot.
Watch step distribution, path diversity and terminal-state mix. The last is the most sensitive.
A retry that reduces a symptom without a named error class is a bug being hidden.

Next, and the start of section 8: what an agent should remember, and for how long.