Chapter 77 · Tools and actions

Tool errors, and what the model should see when one fails

An error message becomes context, so it is a prompt. Five error classes, what each should say, and the stack trace that taught a model to invent an answer.

11 min read

Chapter 66 measured result interpretation at 0.96, which is the second-worst stage in the loop. A large share of that 4% is error handling.

The reason is structural: a tool error is not logged, it is read. Whatever the tool returns becomes the next thing in the model's context, so an error message is a prompt — and most error messages were written for a developer reading a log.

The five classes

Each needs a different response from the model, so each must be distinguishable.

Class	The model should	Example
Correctable	Retry with a fix	Bad SKU format
Clarifying	Ask the user	Ambiguous account
Empty	Report the absence	No tickets found
Unavailable	Report the gap, do not substitute	API down
Refused	Explain or offer	Permission, intent, precondition

// src/tools/errors.ts
export type ToolError =
  | { class: 'correctable'; error: string; message: string; suggestion?: Record<string, unknown> }
  | { class: 'clarifying';  error: string; message: string; candidates: Candidate[] }
  | { class: 'empty';       message: string; searched: string }
  | { class: 'unavailable'; error: string; message: string; retryable: boolean }
  | { class: 'refused';     error: string; message: string; offer?: ToolRequest }

Making the class part of the type rather than an inferred property of the message is the whole design. A model asked to deduce from prose whether to retry will sometimes retry a permission refusal, and that looks like an attack in an audit log.

Empty is not an error

The distinction that matters most, and the one most often collapsed.

// ❌ Both of these become "no tickets", and only one is true.
if (rows.length === 0) return { ok: false, error: 'not_found' }

// ✅ An empty result is a successful answer to the question asked.
if (rows.length === 0) {
  return {
    ok: true,
    data: [],
    class: 'empty',
    // ⚠️ Say what was searched, so the model can report the scope of the
    // absence accurately rather than overstating it.
    message: 'No open tickets for Henderson Industrial Supply in the last 90 days.',
    searched: 'open tickets, Henderson Industrial Supply, last 90 days',
  }
}

Including the search scope stops the answer generalising. "There are no tickets" and "there are no open tickets in the last 90 days" are different claims, and the model will make the first one if the result does not stop it.

Correctable errors carry the correction

Chapter 69 measured this: first-retry success went from 0.58 to 0.94 when the error carried a suggestion.

export function formatValidationError(issues: z.ZodIssue[], args: unknown): ToolError {
  const issue = issues[0]
  const suggestion = deriveSuggestion(issue, args)   // only when unambiguous

  return {
    class: 'correctable',
    error: 'invalid_arguments',
    message: suggestion
      ? `${issue.path.join('.')} ${issue.message}. You provided ` +
        `${JSON.stringify(get(args, issue.path))}; try ${JSON.stringify(Object.values(suggestion)[0])}.`
      : `${issue.path.join('.')} ${issue.message}.`,
    suggestion,
  }
}

// ⚠️ Only derive a suggestion where exactly one correction is possible.
// A guessed suggestion is worse than none: the model will take it.
function deriveSuggestion(issue: z.ZodIssue, args: unknown) {
  if (issue.path[0] === 'sku') {
    const raw = String(get(args, issue.path) ?? '')
    const digits = raw.match(/\d{4}/)?.[0]
    return digits ? { sku: `SKU-${digits}` } : undefined
  }
  return undefined
}

Unavailable must not be substitutable

Chapter 75 found 14 of 61 unavailable results papered over. The error itself can make that harder.

{
  class: 'unavailable',
  error: 'source_unavailable',
  // ⚠️ Written as the sentence you want relayed. A model given a phrase
  // it can use verbatim uses it; given a code, it paraphrases or omits.
  message: 'The ticket system did not respond, so I could not check for ' +
           'existing tickets. This is a gap in the answer, not a confirmation ' +
           'that none exist.',
  retryable: false,
}

That second sentence is doing the work. It pre-empts the specific wrong inference — absence of data read as data about absence — rather than relying on a general prompt rule to prevent it.

Refusals explain and offer

// A refusal is the system working. The message should not read like a
// malfunction, and where an action is available it should be offered.
{
  class: 'refused',
  error: 'outside_intent',
  message: 'You were asked to check something, not to change it. Tell the ' +
           'user what you found and offer the change.',
  offer: { tool: 'update_order', arguments: { orderId: 'ORD-8821', discountPercent: 12 } },
}

⚠️ Permission refusals are the exception and carry no offer and no detail, for chapter 63's reason:

{
  class: 'refused',
  error: 'not_found',          // ⚠️ deliberately not 'forbidden'
  message: 'No matching record is available to you.',
}

What never reaches the model

// ❌ Observed consequences of returning a stack trace: the model has
//    explained a database error to a sales rep, has suggested the user
//    "check the connection string", and — in one run — has invented a
//    plausible answer rather than relay something that looked like a bug.
return { ok: false, error: String(error) }

// ✅ Classify, log the detail, return a sentence.
catch (error) {
  const id = randomUUID()
  logger.error('tool.failed', { id, tool, error: serialiseError(error) })
  return {
    class: 'unavailable',
    error: 'internal_error',
    message: 'That check failed for a technical reason. It has been logged.',
    retryable: false,
    // For the interface and the support ticket, not for the model to read out.
    reference: id,
  }
}

The invented-answer behaviour is the one worth dwelling on. Given an error it could not interpret, a model under instruction to be helpful sometimes produced an answer from retrieval instead — so a technical failure became a confident, unsourced, stale answer.

// A sanitiser applied to every tool result, enforced centrally.
const LEAK = [
  /\b(?:at\s+\w+\.\w+\s+\(|Error:\s|\w+Error\b)/,         // stack traces
  /\b(?:select|insert|update|delete)\s+.*\bfrom\b/i,       // SQL
  /\b(?:postgres|redis|mongodb):\/\//,                     // connection strings
  /\b(?:\/Users\/|\/home\/|\/var\/)/,                      // paths
  /\b[A-Za-z0-9_-]{32,}\b/,                                // tokens
]

export function sanitiseToolMessage(message: string, tool: string): string {
  for (const pattern of LEAK) {
    if (pattern.test(message)) {
      logger.error('tool.message_leaked', { tool, pattern: String(pattern) })
      return 'That check failed for a technical reason.'
    }
  }
  return message
}

Logging the leak rather than silently replacing it is what makes the sanitiser a detector as well as a guard — it fires when somebody adds a tool that returns a raw error, and names the tool.

Retry policy belongs in your code

// ⚠️ Not in the model's hands. A model deciding whether to retry will
// retry the wrong classes and loop on the retryable ones.
const RETRY: Record<string, { attempts: number; backoffMs: number }> = {
  correctable: { attempts: 1, backoffMs: 0 },      // one corrected retry
  clarifying:  { attempts: 0, backoffMs: 0 },      // ask the user
  empty:       { attempts: 0, backoffMs: 0 },      // not a failure
  unavailable: { attempts: 0, backoffMs: 0 },      // chapter 75: never on
  refused:     { attempts: 0, backoffMs: 0 },      //   the request path
}

// And a per-run cap, because the model can always ask again with
// different arguments, which is not a retry but costs the same.
if (ctx.failedCalls >= MAX_FAILED_CALLS_PER_RUN) {
  return { class: 'refused', error: 'too_many_failures',
    message: 'Several tool calls have failed in this run. Summarise what you ' +
             'have and tell the user what could not be checked.' }
}

That message is deliberately an instruction rather than a refusal. It converts a dead end into a partial answer, which is the outcome a user prefers to an apology.

Measuring interpretation

// evals/error-handling.ts — inject each class and assert on the response.
const CASES = [
  { inject: EMPTY_RESULT,        expect: { mentions: /no .*tickets/i, doesNotMention: /error|fail/i } },
  { inject: UNAVAILABLE,         expect: { mentions: /could not check|did not respond/i,
                                           // ⚠️ must NOT answer the question anyway
                                           doesNotAnswer: true } },
  { inject: CORRECTABLE,         expect: { retriesWith: { sku: 'SKU-4411' } } },
  { inject: CLARIFYING,          expect: { asksUser: true, listsCandidates: 2 } },
  { inject: PERMISSION_REFUSED,  expect: { doesNotMention: /permission|forbidden|access/i } },
]

Atlas, before and after rewriting the error messages:

Class	Before	After
Empty reported correctly	0.91	0.98
Unavailable reported, not substituted	0.77	0.97
Correctable retried correctly	0.58	0.94
Clarifying asked the user	0.84	0.99
Permission refusal not disclosed	0.93	1.00

No model change and no prompt change beyond one rule. The gains are entirely in what the tools returned.

The failure mode: the error that was correct and unactionable

apply_credit rejects a call: { ok: false, error: 'limit_exceeded' }.

Accurate, logged, and useless. The model does not know whose limit, what it is, how far over the request was, or what to do — so it says "I was unable to apply the credit due to a limit." The rep does not know whether to reduce the amount, ask someone else, or whether Atlas is broken.

// The same refusal, actionable.
{
  class: 'refused',
  error: 'limit_exceeded',
  message: 'That credit is $4,200 and your approval limit is $2,500. ' +
           'Credits above your limit need a finance approver.',
  // ⚠️ The route forward, named. This is what turns a dead end into a
  // next step, and it is three fields.
  offer: {
    tool: 'request_approval',
    arguments: { capability: 'apply_credit', amount: 4_200, approverRole: 'finance' },
  },
}

I can't apply a $4,200 credit — your limit is $2,500. Finance can approve it.

  [ Request finance approval ]

The general rule, and it is the one to take from this chapter:

Write every error message as the sentence you want the user to read, with the next step in it. The model is a relay, and it relays what you give it — a code becomes an apology, and a sentence becomes an answer.

Key points

A tool error is read, not logged. It is a prompt and gets a prompt's care.
Five classes, carried as a typed field rather than inferred from prose: correctable, clarifying, empty, unavailable, refused.
Empty is a successful answer. Say what was searched, or the model overstates the absence.
Carry a correction only when exactly one is possible. A guessed suggestion will be taken.
Write the unavailable message to pre-empt the wrong inference, in words the model can relay verbatim.
Permission refusals carry no detail and no offer, and are phrased as not-found.
Never return a stack trace, SQL, a path or a token. Sanitise centrally and log every leak, naming the tool.
Retry policy is yours, not the model's, and there is a per-run cap that converts a dead end into a partial answer.
Rewriting error messages alone moved four of five interpretation metrics by 7 to 36 points.
Put the next step in the message. A code becomes an apology; a sentence becomes an answer.

Next, and the last chapter of this section: testing tools independently of the model.