Chapter 69 · Tools and actions

Arguments, validation and refusing a bad call

0.91 is the worst number in the loop, and identity resolution is most of it. Four validation layers, the ambiguity that must be escalated rather than guessed, and why a plausible argument is the dangerous one.

11 min read

Chapter 66 measured the loop: tool selection 0.99, arguments 0.91, interpretation 0.96, termination 0.98. Arguments dominate the product, and this chapter is why and what to do.

The core problem: an argument the model produces is a guess that looks like a fact. accountId: "4471" carries no indication of whether it was read from the conversation, inferred from a partial name, or invented because the schema required a string.

Where the 9% goes

Atlas's 60 tool cases, failures classified:

Cause	Share	Example
Identity resolution	48%	Two Henderson accounts; picked one
Missing optional argument	19%	Omitted `warehouse` when the question named one
Format	14%	`4411` instead of `SKU-4411`
Invented value	11%	An account id that does not exist
Wrong unit or scale	8%	`30` for a value in days when the field is hours

Nearly half is identity, and identity is the one the model cannot do correctly in principle — it does not have the data.

Layer 1: schema

Free, and it catches format and type.

// src/tools/schemas.ts
export const CheckInventoryArgs = z.object({
  // ⚠️ A pattern, not just a string. It converts an invented value into a
  // validation error instead of a database lookup that returns nothing.
  sku: z.string().regex(/^SKU-\d{4}$/, 'SKU must look like SKU-4411'),
  warehouse: z.enum(['dallas', 'reno', 'columbus']).optional(),
})

export const ApplyCreditArgs = z.object({
  accountId: z.string().regex(/^\d{4}$/),
  amount: z.number().positive().max(50_000),   // a cap is a cheap guard
  currency: z.literal('USD'),
  reason: z.string().min(10).max(500),
})

The max on amount is worth including even where the business has no such limit. It converts a misplaced decimal point from a credit of $250,000 into a validation failure, and the cost of being wrong about the cap is one refused call.

Layer 2: resolution, and the ambiguity escalation

The layer that matters. The model gives you what the person said; your code turns it into an identity, and it must be able to fail.

// src/tools/resolve.ts
export type Resolution<T> =
  | { kind: 'resolved'; value: T }
  | { kind: 'not_found'; query: string }
  | { kind: 'ambiguous'; query: string; candidates: Array<{ id: string; label: string }> }

export async function resolveAccount(name: string, scope: Scope): Promise<Resolution<Account>> {
  const { rows } = await db.query(
    `select id, name, city, status from accounts
      where name ilike $1 and owner_id = any($2)
      order by (lower(name) = lower($3)) desc, name
      limit 5`,
    [`%${name}%`, scope.ownedAccountIds, name],
  )

  if (rows.length === 0) return { kind: 'not_found', query: name }
  if (rows.length === 1) return { kind: 'resolved', value: rows[0] }

  // ⚠️ An exact case-insensitive match wins over partial matches. Without
  // this, "Henderson" against "Henderson" and "Henderson Pacific" is
  // ambiguous when it should not be.
  const exact = rows.filter((r) => r.name.toLowerCase() === name.toLowerCase())
  if (exact.length === 1) return { kind: 'resolved', value: exact[0] }

  return {
    kind: 'ambiguous',
    query: name,
    candidates: rows.map((r) => ({ id: r.id, label: `${r.name} — ${r.city} (${r.status})` })),
  }
}

// And the tool refuses rather than choosing.
async run(args: CheckInventoryArgs, scope: Scope) {
  const account = await resolveAccount(args.accountName, scope)

  if (account.kind === 'not_found') {
    return { ok: false, error: 'not_found',
      message: `No account matching "${account.query}" that you own.` }
  }
  if (account.kind === 'ambiguous') {
    // ⚠️ Returned to the model as a RESULT, so it asks the user. It is not
    // an error and it is not a guess.
    return { ok: false, error: 'ambiguous',
      message: `Several accounts match "${account.query}". Which did you mean?`,
      candidates: account.candidates }
  }
  // ...
}

An argument your code cannot resolve to exactly one thing is not a bad request to reject or a coin to flip. It is a question to ask.

The label in each candidate is doing real work: Henderson Industrial — Dallas (active) against Henderson Pacific — Reno (dormant) lets the person pick in one glance, where two bare ids would not.

Layer 3: cross-argument consistency

Arguments that are individually valid and jointly wrong.

export function validateConsistency(args: DraftQuoteArgs, ctx: ResolvedContext): Issue[] {
  const issues: Issue[] = []

  // The discount must be one the account is actually entitled to.
  if (args.discountPercent > ctx.account.maxDiscount) {
    issues.push({ kind: 'discount_above_entitlement',
      requested: args.discountPercent, permitted: ctx.account.maxDiscount })
  }
  // The SKU must be one this account can buy.
  if (!ctx.account.catalogues.includes(ctx.product.catalogue)) {
    issues.push({ kind: 'product_not_in_catalogue' })
  }
  // Quantity against a plausible band, derived from history rather than
  // from a constant somebody guessed.
  if (args.quantity > ctx.account.p99Quantity * 5) {
    issues.push({ kind: 'quantity_implausible',
      requested: args.quantity, typicalMax: ctx.account.p99Quantity })
  }
  return issues
}

The last check is the one that catches a transposition. A rep asking for 200 units and the model producing 2000 passes every type, range and entitlement check, and fails a comparison against what this account has ever ordered.

// Implausible is not invalid. It is a confirmation trigger, per chapter 72.
if (issues.some((i) => i.kind === 'quantity_implausible')) {
  return { ok: false, kind: 'needs_confirmation',
    preview: `Quote for ${args.quantity} units — this account's largest previous ` +
             `order was ${ctx.account.p99Quantity}. Confirm?` }
}

Layer 4: grounding the argument in the conversation

The check that catches invention, and it is cheap.

// Every identifier-shaped argument should trace to something the user said
// or a tool returned. One that appears from nowhere is fabricated.
export function validateProvenance(
  args: Record<string, unknown>, context: RunContext,
): Issue[] {
  const said = `${context.question} ${context.history.map((t) => t.question).join(' ')}`
  const returned = context.toolResults.map((r) => JSON.stringify(r)).join(' ')
  const seen = `${said} ${returned}`.toLowerCase()

  return Object.entries(args)
    .filter(([, v]) => typeof v === 'string' && IDENTIFIER.test(v))
    // ⚠️ A loose containment check. It is meant to catch invention, not to
    // be a strict provenance system — false positives here would block
    // legitimate normalisation like "4411" -> "SKU-4411".
    .filter(([, v]) => !seen.includes(normaliseIdentifier(String(v))))
    .map(([k, v]) => ({ kind: 'argument_not_grounded', field: k, value: String(v) }))
}

const IDENTIFIER = /^(?:[A-Z]{2,}-[\w.]+|\d{4,})$/

On Atlas this fired on 1.9% of tool calls. About half were legitimate — the model normalising a product name to a code it had seen in a retrieved chunk — so it logs and flags rather than blocking, and the flagged calls go to the same review queue as unverifiable figures.

Returning a failure the model can act on

A rejected call becomes context. Chapter 77 covers this generally; for argument failures the rule is specific.

// ❌ The model retries with the same arguments, or gives up and invents
//    an answer. Both are observed.
{ ok: false, error: 'validation_failed' }

// ✅ What was wrong, and what to do instead.
{
  ok: false,
  error: 'invalid_arguments',
  message: 'sku must look like SKU-4411. You provided "4411".',
  // ⚠️ The correction, where it is unambiguous. This turns a failed call
  // into a successful retry rather than into a second guess.
  suggestion: { sku: 'SKU-4411' },
}

Measured: adding suggestion to format failures took the success rate of the first retry from 0.58 to 0.94. Without it the model frequently "corrected" to a different wrong value.

Measuring argument accuracy

// evals/tool-arguments.ts — no execution. Deterministic and free.
export async function scoreArguments(cases: ToolCase[]) {
  const rows = await Promise.all(cases.map(async (c) => {
    const req = await planToolCall(c.question, { scope: scopeFor(c), history: c.history })
    if (req?.tool !== c.expect.tool) return { id: c.id, kind: 'wrong_tool' as const }

    const diff = diffArguments(req.arguments, c.expect.arguments)
    return { id: c.id, kind: diff.length === 0 ? 'ok' as const : 'wrong_args' as const,
             diff, cause: classifyArgumentFailure(diff, c) }
  }))

  return { accuracy: mean(rows.map((r) => +(r.kind === 'ok'))),
           byCause: countBy(rows.filter((r) => r.cause), (r) => r.cause!) }
}

Atlas, before and after the four layers:

	Before	After
Argument accuracy	0.91	0.98
Identity failures reaching execution	4.3%	0
Ambiguity escalated to the user	0%	3.1%
Format failures	1.3%	0.1%
End-to-end tool correctness	0.86	0.94

The second row is the one that matters for write tools: identity failures no longer reach execution, because resolution happens in code and refuses rather than guessing. The 3.1% escalation rate is the cost, and it is a question to a person rather than an action on the wrong account.

The failure mode: the argument that was plausible

A rep asks: "what's the stock on the usual Henderson part?"

The model has seen SKU-4411 in the conversation three turns earlier, in a different context. It produces sku: 'SKU-4411'. The SKU is real, it passes the pattern, it passes provenance because it genuinely appeared in the conversation, and it is not the part the rep meant.

Every layer passes. The answer is a correct stock figure for the wrong product.

// There is no validation fix for this. The argument is well-formed, real,
// and grounded. The only defence is that the ANSWER names what it acted on.
return {
  ok: true,
  data: rows,
  // ⚠️ Echo the resolved entity, so the answer can state it and a person
  // can catch the mismatch in one glance.
  resolved: { sku: 'SKU-4411', productName: 'Hydraulic seal, 40mm' },
}

<!-- And the prompt rule that uses it. One sentence, and it is the whole
     defence against a class nothing else catches. -->
When a tool result includes a `resolved` field, name the resolved entity in
your answer. Say "SKU-4411 (hydraulic seal, 40mm)" rather than "that part".

Atlas's measurement: with the echo, reps caught wrong-entity calls at a rate of about one per week and corrected them immediately. Without it, the same errors were found in weekly order reviews, or not at all.

Some wrong arguments are indistinguishable from right ones at validation time. For those, the defence is not a check — it is making the answer say what it acted on, so the person who knows can see it.

Key points

Arguments are the worst stage in the loop, and identity resolution is nearly half of the failures.
Patterns and caps in the schema turn invented values and misplaced decimals into validation errors.
Resolve identities in code with three outcomes: resolved, not found, ambiguous. Ambiguity is a question, not a coin flip.
Give candidates human labels, so the person picks in one glance.
Check cross-argument consistency against entitlements and against this account's own history. A transposed quantity passes every other check.
Implausible is a confirmation trigger, not a rejection.
Check provenance loosely: an identifier that appears from nowhere is fabricated. Flag, do not block.
Return the correction with the error. First-retry success went from 0.58 to 0.94.
Four layers took argument accuracy from 0.91 to 0.98 and identity failures reaching execution to zero.
A plausible wrong argument passes everything. Echo the resolved entity and have the answer name it.

Next: read tools against write tools — two boundaries, not one.