Chapter 86 · Agent loops and control

Handling partial failure mid-task

A run that got three of four things is the normal outcome, not an exception. Classifying what failed, deciding whether the answer survives it, and never presenting a partial result as complete.

10 min read

A single-call system succeeds or fails. A loop that makes five calls has thirty-two outcomes, and the interesting ones are in the middle.

Partial failure is the normal state of a multi-step run, and the question is not how to prevent it but how to answer honestly from what survived.

Classifying what failed

Not all missing pieces matter equally, and the distinction has to be declared rather than inferred.

// src/agent/partial.ts
export type StepCriticality =
  // Without it the answer is wrong, not just incomplete.
  | 'required'
  // The answer is narrower without it, and still correct.
  | 'enriching'
  // It was a check; its absence is a caveat.
  | 'confirming'

// Declared on the plan step, or derived from the tool for implicit runs.
export interface PlanStep {
  n: number
  tool: string
  criticality: StepCriticality
  // ...
}

export function assessPartial(plan: Plan, results: Map<number, ToolResult>): Assessment {
  const failed = plan.steps.filter((s) => !results.get(s.n)?.ok)

  const requiredFailed = failed.filter((s) => s.criticality === 'required')
  if (requiredFailed.length > 0) {
    // ⚠️ Not "answer with a caveat". The answer would be wrong.
    return { kind: 'cannot_answer', missing: requiredFailed.map((s) => s.tool) }
  }

  return {
    kind: 'partial',
    answerable: true,
    gaps: failed.map((s) => ({ tool: s.tool, criticality: s.criticality,
                               reason: results.get(s.n)?.error ?? 'not reached' })),
  }
}

The cannot_answer branch is the one that gets omitted, and omitting it produces the worst outcome in this chapter: an answer built from the enriching steps while the required one is missing, which reads complete and is wrong.

The distinction that matters: empty against failed

Chapter 77 made this point for one call. In a multi-step run it compounds.

// Three outcomes that look similar in a result map and mean different
// things to the answer.
//
//   ok, data: [...]     the step succeeded and found something
//   ok, data: []        the step succeeded and found nothing  ← a FACT
//   !ok                 the step did not run or did not answer ← a GAP
//
// ⚠️ Conflating the middle and the last is how "there are no open
// tickets" is reported when the ticket system was down.

export function describeOutcome(step: PlanStep, result: ToolResult): string {
  if (!result.ok) return `could not check ${describe(step)}`
  if (isEmptyResult(result)) return `checked ${describe(step)}: none found`
  return `checked ${describe(step)}`
}

Answering from what survived

export async function answerPartial(run: Run, assessment: Assessment) {
  if (assessment.kind === 'cannot_answer') {
    return {
      text: `I could not answer this. ${describeMissing(assessment.missing)}.`,
      found: false,
      // ⚠️ Still return what WAS gathered. The rep may be able to finish
      // from it, and discarding successful work is pure waste.
      gathered: successfulResults(run),
    }
  }

  const answer = await generate(run.question, successfulResults(run), {
    prompt: 'answer-partial',
    // The gaps go INTO the prompt, so the model writes around them rather
    // than being corrected afterwards.
    gaps: assessment.gaps.map((g) => describeGap(g)),
  })

  return { ...answer, partial: true, gaps: assessment.gaps }
}

<!-- src/generate/prompts/answer-partial.md -->
Some checks did not complete. Answer from what is available.

Rules:
- State what you could not check, in one sentence, at the end.
- Do not speculate about what the missing checks would have shown.
- If the missing information could change the answer, say so explicitly.

Could not check:
{{gaps}}

Available:
{{sources}}

Question: {{question}}

The third rule is the one that produces honest answers rather than merely caveated ones:

Henderson is on Net 45 per amendment 2 [1], and three invoices totalling
$23,100 are outstanding [2].

I could not check the payment history — the finance system did not respond —
so if a payment was rejected recently, that would change the picture.

That last clause is the difference between a caveat the reader skims and one they act on.

Deciding whether to retry mid-run

// Retrying inside a run is different from retrying a request. The budget
// is shared, so a retry spends the steps the rest of the plan needs.
export function shouldRetryStep(
  step: PlanStep, result: ToolResult, budget: Budget, plan: Plan,
): boolean {
  if (result.ok) return false
  if (result.class !== 'correctable' && result.class !== 'unavailable') return false

  // ⚠️ Only retry if the remaining budget covers the retry AND the rest
  // of the plan. A retry that strands the plan is worse than the gap.
  const remainingSteps = plan.steps.filter((s) => s.n > step.n).length
  if (budget.remainingSteps() < remainingSteps + 1) return false

  // Never retry an unavailable dependency on the request path. Chapter 75.
  if (result.class === 'unavailable') return false

  return step.criticality === 'required'
}

Retrying only required steps is the policy that falls out of the criticality classification, and it is the right one: an enriching step that failed costs a sentence of the answer, and the retry costs a step that the required steps may need.

Degrading the route

When a required step fails, the better move is often a different path rather than a retry.

// The ticket system is down. The investigation cannot complete — but
// the question may still be answerable from documents.
if (assessment.kind === 'cannot_answer' && canFallBack(run.question)) {
  logger.info('run.degraded', { from: 'planning', to: 'question',
                                reason: assessment.missing })
  const fallback = await questionPath(run.question, run.scope)
  return {
    ...fallback,
    // ⚠️ Say that this is the fallback. A document answer to a live-data
    // question is stale, and the reader must know which they got.
    caveats: [...fallback.caveats,
      'Live systems were unavailable, so this is from the documents and may be out of date.'],
  }
}

That is chapter 85's escalation running in reverse, and the caveat is mandatory: a document answer silently substituted for a live one is exactly the staleness failure the live route existed to avoid.

Measuring partial outcomes

-- Partial runs are not errors and should not be in the error rate. They
-- need their own row, and their own quality number.
select
  count(*) filter (where outcome = 'complete')      as complete,
  count(*) filter (where outcome = 'partial')       as partial,
  count(*) filter (where outcome = 'cannot_answer') as cannot_answer,
  count(*) filter (where outcome = 'failed')        as failed,
  round(avg(correct::int) filter (where outcome = 'complete'), 2) as correct_complete,
  round(avg(correct::int) filter (where outcome = 'partial'), 2)  as correct_partial
  from runs
 where started_at > now() - interval '30 days';

Atlas:

Outcome	Runs	Correct
complete	2,641	0.95
partial	294	0.91
cannot_answer	71	n/a — reported honestly
failed	9	n/a

Partial runs are 9.7% of the total and are correct 0.91 of the time — only four points below complete ones, because the gaps were mostly enriching rather than required. That number is the justification for answering partially at all: discarding those 294 runs would have cost 267 correct answers to avoid 27 incomplete ones.

-- And the query that drives improvement: which tool is causing the gaps?
select tool, count(*) as gaps,
       count(*) filter (where criticality = 'required') as blocking
  from run_gaps
 where created_at > now() - interval '30 days'
 group by 1 order by blocking desc, gaps desc;
-- get_payment_history   118   41   ⚠️ one dependency, most of the gaps

One flaky dependency accounting for 40% of all gaps is an infrastructure conversation, not an agent problem — and it is only visible because gaps are recorded per tool.

The failure mode: the gap that was summarised away

Atlas's partial answers are good. Then someone adds a final pass that tidies the answer for readability.

The tidying model receives the answer and the instruction to make it concise and well-structured. The caveat — "I could not check the payment history" — is one sentence at the end of a four-sentence answer, and conciseness removes it about 30% of the time.

The answers get shorter, cleaner, and silently complete-looking.

// ❌ A post-processing pass that can see and remove the caveat.
const tidied = await callModel(prompt('tidy', { answer: answer.text }))

// ✅ Caveats are structural, not prose. They are rendered by the
//    interface, outside anything the model can rewrite.
interface Answer {
  text: string
  caveats: string[]        // ⚠️ rendered separately, never in `text`
  gaps: Gap[]
}

// And the test, because this will be reintroduced by somebody trying to
// improve the writing.
test('caveats survive post-processing', async () => {
  const answer = await answerPartial(runWithGaps(), partialAssessment())
  const rendered = renderAnswer(answer)
  expect(rendered).toMatch(/could not check/i)
  expect(answer.caveats.length).toBeGreaterThan(0)
})

Anything that must reach the reader cannot be inside text a model is allowed to rewrite. Keep caveats as a field, render them outside the prose, and test that they survive.

That is the same rule as emitting the abstention sentence from code and keeping permissions out of the prompt, arriving a third time from a third direction.

Key points

Partial failure is the normal outcome of a multi-step run, not an exception.
Classify steps as required, enriching or confirming. Without required, a missing critical step produces a complete-looking wrong answer.
Keep empty distinct from failed all the way through. Conflating them reports absence of data as data about absence.
Put the gaps in the prompt so the model writes around them, and require it to say when a gap could change the answer.
Return gathered results even when the run cannot answer. Discarding successful work is waste.
Retry only required steps, and only when the budget covers the retry plus the rest of the plan.
Degrading to the document route is often better than retrying — with a mandatory caveat that the answer is not live.
Partial runs were 9.7% of Atlas's traffic and 0.91 correct. Discarding them would have cost 267 correct answers to avoid 27 incomplete ones.
Record gaps per tool. One flaky dependency was 40% of them.
Caveats are a field, rendered outside the prose. Anything inside rewritable text will eventually be rewritten away.

Next: resumability — picking up an interrupted run.