Chapter 87 · Agent loops and control

Resumability, and picking up an interrupted run

Three reasons a run stops and three different resumptions. The state that must be durable, the replay that must not re-execute, and the stale resume that acted on yesterday's world.

11 min read

Chapter 79 introduced suspended as a first-class run state and persisted every step. This chapter is what that persistence is for.

Three things interrupt a run, and they need different resumptions.

Interruption	State	Resume	Window
Awaiting confirmation	`suspended`	On human decision	Minutes to hours
Process crash or deploy	`running`, orphaned	Automatically	Seconds
Budget halt	`halted`	On explicit user request	Any

What must be durable

create table runs (
  id              text primary key,
  user_id         text not null,
  question        text not null,
  route           jsonb not null,
  state           text not null,        -- running|answered|halted|suspended|failed
  step            int  not null default 0,

  -- The conversation so far. ⚠️ The whole thing: resuming needs exactly
  -- what the model saw, not a summary of it.
  messages        jsonb not null,
  trace           jsonb not null default '[]',

  -- Budget consumed, so a resumed run cannot reset its limits.
  spend_usd       numeric(10,6) not null default 0,
  tool_calls      int not null default 0,
  elapsed_ms      int not null default 0,

  -- ⚠️ A snapshot of the scope, NOT a reference. See below.
  scope_snapshot  jsonb not null,
  scope_resolved_at timestamptz not null,

  -- Suspension detail.
  pending         jsonb,                -- the tool request awaiting approval
  pending_token   text unique,

  started_at      timestamptz not null default now(),
  updated_at      timestamptz not null default now(),
  heartbeat_at    timestamptz
);

create index runs_orphaned on runs (heartbeat_at) where state = 'running';
create index runs_suspended on runs (pending_token) where state = 'suspended';

Two fields need justifying.

spend_usd, tool_calls and elapsed_ms persist. A resumed run continues its budget rather than starting a new one. Otherwise suspension is a way to get unlimited steps by confirming repeatedly.

// Restored, not reset.
const budget = Budget.restore(run.limits, {
  spendUsd: run.spend_usd, toolCalls: run.tool_calls, elapsedMs: run.elapsed_ms,
})

The scope is a snapshot with a timestamp, which is the interesting decision and the subject of the failure mode below.

Resuming after confirmation

export async function resumeAfterConfirmation(token: string, decision: Decision) {
  const run = await loadBySuspensionToken(token)
  if (!run) return { ok: false, error: 'unknown_or_expired' }

  // ⚠️ Claim the resumption atomically. Two clicks on the approve button
  // must resume once — the same problem as chapter 73, one level up.
  const claimed = await db.query(
    `update runs set state = 'running', pending_token = null
      where id = $1 and state = 'suspended' returning *`, [run.id])
  if (claimed.rowCount === 0) return { ok: false, error: 'already_resumed' }

  // Re-validate the scope. Chapter 72 re-checked the preview; this
  // re-checks the person.
  const current = await resolveScope(await getUser(run.user_id))
  if (!stillPermitted(run.pending, current)) {
    return { ok: false, error: 'no_longer_permitted',
      message: 'Your permissions changed while this was waiting. Please ask again.' }
  }

  const result = decision === 'approve'
    ? await executeToolRequest(run.pending, current, { ...context(run), confirmed: true })
    : { ok: false, error: 'declined_by_user' }

  run.messages.push(toolResultMessage(run.pending, result))
  return runLoop({ ...run, scope: current })
}

Re-resolving the scope rather than using the snapshot is deliberate here: a confirmation can sit for an hour, and acting on an hour-old permission is the stale-grant problem with a human-shaped delay.

Resuming an orphaned run

// A heartbeat, written each step, identifies runs whose process died.
export async function heartbeat(runId: string) {
  await db.query(`update runs set heartbeat_at = now() where id = $1`, [runId])
}

// The reaper. Runs every minute.
export async function recoverOrphans() {
  const { rows } = await db.query(
    `select * from runs
      where state = 'running' and heartbeat_at < now() - interval '90 seconds'
      order by started_at limit 20`)

  for (const run of rows) {
    // ⚠️ The last step may have executed a write. Reconcile BEFORE
    // resuming, or the resumed run repeats it. Chapter 73.
    await reconcileUnknown({ runId: run.id })

    if (run.recovery_attempts >= 2) {
      await markFailed(run.id, 'recovery_exhausted')
      continue
    }
    await db.query(`update runs set recovery_attempts = recovery_attempts + 1 where id = $1`, [run.id])
    void runLoop(await loadRun(run.id))
  }
}

The reconciliation before resumption is the non-negotiable part. A run that crashed between executeToolRequest returning and the message being appended has performed the action and has no record of it in messages — so a naive resume asks the model to continue from a state where the tool was never called, and it calls it again.

// ⚠️ And the cap. A run that has crashed twice will crash a third time,
// and an uncapped reaper is an infinite loop with a database behind it.

Resuming a halted run

// A user-initiated continuation of a run that hit a budget bound.
export async function continueRun(runId: string, user: User) {
  const run = await loadRun(runId)
  if (run.state !== 'halted') return { ok: false, error: 'not_halted' }
  if (run.user_id !== user.id) return { ok: false, error: 'not_found' }

  // ⚠️ A fresh budget, and it is a deliberate grant rather than a reset.
  // The user is choosing to spend more; the system is not doing it.
  const extended = { ...run.limits,
    maxSteps: run.limits.maxSteps + 4,
    maxSpendUsd: run.limits.maxSpendUsd * 1.5 }

  logger.info('run.continued', { runId, by: user.id, previousReason: run.halt_reason })
  return runLoop({ ...run, limits: extended, state: 'running' })
}

Making continuation explicit rather than automatic is the point. An automatic retry of a halted run is a way of multiplying the budget bound by however many times it retries.

What must not be replayed

// Resumption replays the CONVERSATION, never the actions. The messages
// already contain the tool results; the tools are not called again.
//
// ❌ Re-running the trace to rebuild state.
for (const entry of run.trace) await executeToolRequest(entry.request, scope, ctx)
//
// ✅ The messages ARE the state. Hand them to the model and continue.
return runLoop({ ...run, messages: run.messages })

This is why messages is stored whole rather than reconstructed from the trace. The trace is for humans; the messages are for resumption, and deriving one from the other loses exactly the fidelity that makes resumption safe.

Expiry

// Suspended runs expire. An approval sitting for a day is approving a
// world that has moved.
export const SUSPENSION_TTL_MINUTES = 30      // chapter 72

export async function expireSuspended() {
  await db.query(
    `update runs set state = 'halted', halt_reason = 'suspension_expired'
      where state = 'suspended'
        and updated_at < now() - interval '${SUSPENSION_TTL_MINUTES} minutes'`)
}

// And halted runs are retained, not deleted — they are the input to
// chapter 90's control-flow debugging and to the routing improvement
// loop in chapter 85.
export const RUN_RETENTION_DAYS = 30

Testing it

test('a crash mid-write does not repeat the write', async () => {
  await resetSandbox()
  const run = await startRun('draft a quote for Henderson, 200 of SKU-4411', SCOPE.insideSales)

  // Crash after the tool executed and before the message was persisted.
  sandbox.injectFailure('POST /quotes', { kind: 'crash_after_commit' })
  await expect(runLoop(run)).rejects.toThrow()

  await recoverOrphans()
  await waitForRun(run.id, 'answered')

  expect(await sandbox.countQuotes({ accountId: '4471' })).toBe(1)
})

test('a resumed run keeps its spent budget', async () => {
  const run = await suspendedRunWithSpend(0.18)
  const resumed = await resumeAfterConfirmation(run.pending_token, 'approve')
  expect(resumed.budget.spentUsd).toBeGreaterThanOrEqual(0.18)
})

test('two approvals resume once', async () => {
  const run = await suspendedRun()
  const [a, b] = await Promise.all([
    resumeAfterConfirmation(run.pending_token, 'approve'),
    resumeAfterConfirmation(run.pending_token, 'approve'),
  ])
  expect([a.ok, b.ok].filter(Boolean)).toHaveLength(1)
})

The failure mode: the resume that acted on yesterday's world

A rep asks Atlas to draft and send a quote. The quote is drafted, the send is suspended for confirmation, and the rep goes into a meeting.

Forty minutes later — inside a 30-minute TTL that had been raised to 2 hours "for convenience" — they approve.

In between: the account was placed on credit hold, and the discount tier changed.

The run resumes. The scope is re-resolved, so permissions are current. The quote is not. It was composed against the old discount, the account is now on hold, and the quote goes out.

// Re-resolving the scope is necessary and not sufficient. The ARTEFACT
// has to be re-validated too, which chapter 72 did at the preview level
// and resumption must do again.
export async function resumeAfterConfirmation(token: string, decision: Decision) {
  // ... scope re-resolved ...

  // ⚠️ Re-run the preview and compare. Consent was to a specific artefact.
  const current = await TOOLS[run.pending.tool].preview(run.pending.arguments, scope)
  if (previewDigest(current) !== run.pending_preview_digest) {
    return { ok: false, error: 'changed',
      message: 'The details changed while this was waiting. Please review again.',
      diff: diffPreviews(run.pending_preview, current) }
  }

  // ⚠️ And re-run the tool's own guards, not just the schema. A credit
  // hold is a guard failure, not a preview difference.
  const guard = await TOOLS[run.pending.tool].guard?.(run.pending.arguments, scope, context(run))
  if (guard && !guard.ok) return guard
  // ...
}

Two checks, both cheap, and the second is the one that catches the credit hold — because the discount change shows up in the preview digest and the hold does not.

// And the TTL goes back to 30 minutes, with the reason recorded where
// the next person to find it inconvenient will read it.
export const SUSPENSION_TTL_MINUTES = 30
// ⚠️ Raised to 120 on 2026-11-14 for convenience and reverted on
// 2026-11-21 after a quote was sent against a stale discount on a
// credit-held account. The window is not a UX preference; it bounds how
// stale an approved artefact can be.

A resumption is an action taken on a world that has moved since the decision was made. Re-resolve the scope, re-render the artefact, and re-run the guards — all three, because each catches something the others do not.

Key points

Three interruptions, three resumptions: confirmation, crash, budget halt.
Persist the whole message list, not a summary. The messages are the state; the trace is for humans.
Persist consumed budget. Otherwise suspension is a way to get unlimited steps.
Claim a resumption atomically. Two clicks must resume once.
Re-resolve the scope on resume. An hour-old permission is a stale grant.
Reconcile before resuming an orphaned run, or the last action repeats. Cap recovery attempts.
Continuation of a halted run is user-initiated and grants a fresh budget deliberately.
Never replay actions. Hand back the conversation and continue.
Expire suspensions and retain halted runs — they are the input to debugging and routing improvement.
On resume, re-resolve the scope, re-render the artefact and re-run the guards. Each catches something the others miss.

Next: parallel tool calls, and when they are unsafe.