Chapter 78 · Tools and actions

Testing tools independently of the model

Four test layers, three of which need no model at all. The sandbox that has to fail the same way production does, and the fixture that made every test pass.

12 min read

The last chapter of section 6. Everything built here — resolution, permissions, idempotency, preconditions, error classes — is deterministic code, and almost all of it can be tested without a model.

That matters because chapter 58 established that model-involving tests are slow, non-deterministic and need majority runs. Pushing as much as possible below the model is what makes a tool suite fast enough to run on every commit.

Four layers

Layer	Model	Network	Runs	Time
Unit	No	No	Every commit	~2 s
Contract	No	Yes	Hourly, scheduled	~20 s
Selection	Yes	No	Every PR	~40 s
End to end	Yes	Yes (sandbox)	Pre-deploy	~4 min

Layers 1 and 2 cover roughly 80% of what goes wrong, and neither involves a model.

Layer 1: the tool function

// Everything in chapter 69's four validation layers, chapter 70's
// permissions, and chapter 73's idempotency is testable here.
describe('check_inventory', () => {
  test('rejects a malformed SKU with a correction', async () => {
    const r = await checkInventory.validate({ sku: '4411' })
    expect(r.ok).toBe(false)
    expect(r.class).toBe('correctable')
    expect(r.suggestion).toEqual({ sku: 'SKU-4411' })      // chapter 77
  })

  test('filters to permitted warehouses', async () => {
    const r = await checkInventory.run({ sku: 'SKU-4411' }, SCOPE.dallasOnly)
    expect(r.data.map((x) => x.warehouse)).toEqual(['dallas'])
  })

  test('distinguishes not-found from not-permitted', async () => {
    const missing = await checkInventory.run({ sku: 'SKU-0000' }, SCOPE.finance)
    expect(missing.error).toBe('not_found')

    // ⚠️ Exists, but not in this user's warehouses. Must NOT say not_found,
    // and must not reveal that stock exists elsewhere.
    const hidden = await checkInventory.run({ sku: 'SKU-4411' }, SCOPE.renoOnly)
    expect(hidden.ok).toBe(true)
    expect(hidden.data).toEqual([])
    expect(JSON.stringify(hidden)).not.toMatch(/dallas/)
  })
})

The third test is the one that catches a leak, and asserting on the serialised result rather than on a field is deliberate — a warehouse name leaking in a debug field or a message is the same disclosure.

The permission matrix

// Every tool against every role. Generated, so adding either is covered.
describe.each(ALL_TOOLS)('$name permissions', (tool) => {
  test.each(ALL_ROLES)('role %s', async (role) => {
    const scope = SCOPES[role]
    const expected = PERMISSION_MATRIX[tool.name][role]
    const result = await executeToolRequest(
      { tool: tool.name, arguments: VALID_ARGS[tool.name] }, scope, freshContext())

    expected === 'allow'
      ? expect(result.ok).toBe(true)
      : expect(result.error).toBe('not_found')   // never 'forbidden'
  })
})

// ⚠️ And the matrix is data, reviewed as data. A permission expectation
// buried in a test file is not reviewable by the person who owns the rule.
export const PERMISSION_MATRIX = {
  check_inventory: { inside_sales: 'allow', warehouse_supervisor: 'allow', finance: 'allow' },
  apply_credit:    { inside_sales: 'deny',  warehouse_supervisor: 'deny',  finance: 'allow' },
  send_quote:      { inside_sales: 'deny',  senior_sales: 'allow',         finance: 'allow' },
} as const

Eight tools by five roles is forty assertions written once. It is the cheapest security test in the project and it fails loudly when somebody adds a role.

Layer 2: contract tests

Covered in chapter 75: run them on a schedule against the real dependency, because the point is to catch their change.

// The addition worth making here: assert the error shapes too, not only
// the success shape. An API that changes its 404 body breaks the
// not-found branch, which is the branch nobody exercises.
test('a missing ticket returns the expected 404 shape', async () => {
  const response = await api.get('/tickets/does-not-exist').catch((e) => e.response)
  expect(response.status).toBe(404)
  expect(response.data).toMatchObject({ error: expect.any(String) })
})

Layer 3: selection and arguments

No execution, so it is fast and safe. Chapter 68 built the harness; the discipline here is what goes in the case set.

// Three case types, and the second and third are usually missing.
const SELECTION_CASES = [
  // 1. The tool should be chosen.
  { question: 'how many SKU-4411 in dallas', expect: { tool: 'check_inventory',
    arguments: { sku: 'SKU-4411', warehouse: 'dallas' } } },

  // 2. ⚠️ No tool should be chosen. A document question that triggers a
  //    tool call is a failure nobody writes cases for.
  { question: 'what is our returns policy for damaged goods', expect: { tool: null } },

  // 3. ⚠️ A near-miss that should choose a DIFFERENT tool. This is what
  //    the confusion matrix in chapter 68 turns into a permanent test.
  { question: 'can we ship 200 of these this week', expect: { tool: 'check_inventory' } },
]

// Run with n=3 and require a majority, per chapter 58. A tool chosen two
// times in three is chosen wrongly a third of the time in production.
test.each(SELECTION_CASES)('$question', async (c) => {
  await assertStable(3,
    () => planToolCall(c.question, { tools: ALL_TOOLS, scope: SCOPE.insideSales }),
    (r) => r?.tool === c.expect.tool &&
           (!c.expect.arguments || deepEqual(r.arguments, c.expect.arguments)))
})

Layer 4: end to end, in a sandbox

// The sandbox must fail the way production does, or it tests a system
// that does not exist.
export const SANDBOX_REQUIREMENTS = [
  'same schema, applied by the same migrations',
  'same API version, with the same error bodies',
  'same rate limits',                    // ⚠️ or the limiter is never exercised
  'same auth model',                     // a sandbox with auth disabled tests nothing
  'injectable failures: timeout, 500, slow response',
] as const

// Failure injection is what makes layer 4 worth its four minutes. The
// happy path is already covered by layers 1 and 3.
describe('end to end, with failures', () => {
  test('a timeout on a write does not double it', async () => {
    await resetSandbox()
    sandbox.injectFailure('POST /quotes', { kind: 'timeout_after_commit' })

    await runToolLoop('draft a quote for Henderson, 200 of SKU-4411',
                      SCOPE.insideSales, LIMITS).catch(() => {})
    await reconcileUnknown()                                  // chapter 73

    expect(await sandbox.countQuotes({ accountId: '4471' })).toBe(1)
  })

  test('an unavailable dependency is reported, not papered over', async () => {
    sandbox.injectFailure('GET /tickets', { kind: 'unavailable' })
    const result = await runToolLoop('any open tickets for Henderson?',
                                     SCOPE.insideSales, LIMITS)
    expect(result.text).toMatch(/could not check|did not respond/i)
    // ⚠️ And it must not answer anyway from indexed content. Chapter 75.
    expect(result.text).not.toMatch(/there are no (open )?tickets/i)
  })
})

timeout_after_commit is the injection that matters most, and most sandboxes cannot produce it. Without it, the idempotency path is never exercised and chapter 73's reconciliation is untested code running in production.

What to assert on a write

// ❌ The return value. A tool that executed twice and returned the last
//    result passes this.
expect(result.ok).toBe(true)

// ✅ The side effect, counted. Chapter 70's point, and it catches
//    double-execution for free.
expect(await sandbox.countQuotes({ accountId: '4471' })).toBe(1)
expect(await sandbox.countEmails({ to: 'j.oyelaran@...' })).toBe(0)   // draft, not sent

The second assertion is the one worth copying: asserting that something did not happen is how you catch a tool that does more than its name.

The coverage that matters

// Line coverage is not the useful number here. These five are.
export const TOOL_COVERAGE_REQUIREMENTS = {
  everyToolHasPermissionTests: true,     // the matrix above
  everyWriteToolHasIdempotencyTest: true,
  everyWriteToolHasReconcileTest: true,  // chapter 73's mandatory interface
  everyToolHasSelectionCases: 3,         // chosen, not chosen, near miss
  everyErrorClassHasInterpretationCase: true,   // chapter 77
} as const

// Enforced, so a new tool cannot ship without them.
test('every tool meets the coverage requirements', () => {
  for (const tool of ALL_TOOLS) {
    expect(PERMISSION_MATRIX).toHaveProperty(tool.name)
    expect(selectionCasesFor(tool.name).length).toBeGreaterThanOrEqual(3)
    if (tool.kind === 'write') {
      expect(idempotencyTestFor(tool.name)).toBeDefined()
      expect(tool.reconcile).toBeDefined()
    }
  }
})

The failure mode: the fixture that made everything pass

Atlas's sandbox is seeded with a fixture account: Henderson Industrial Supply, id 4471, owned by every test user, with three orders and one open ticket.

Every test passes. The suite is green for four months.

Then production produces a run of wrong-account calls, and the cause is chapter 69's ambiguity: two Henderson accounts. The sandbox has one, so resolveAccount never returned ambiguous in any test, and the entire escalation path — the branch, the candidate labels, the model's handling of it — was never executed.

// The fixture has to contain the hard cases, and they have to be named
// as the reason each one exists.
export const FIXTURES = {
  accounts: [
    { id: '4471', name: 'Henderson Industrial Supply', owner: 'rep-1' },
    // ⚠️ Ambiguity. Without a second Henderson, the clarifying path is dead code.
    { id: '4480', name: 'Henderson Pacific', owner: 'rep-1' },
    // ⚠️ Permission. An account no test user owns, so not-found is exercised.
    { id: '4492', name: 'Acme Fabrication', owner: 'rep-9' },
    // ⚠️ Edge: an exact-match substring of another name.
    { id: '4501', name: 'Henderson', owner: 'rep-1' },
  ],
  // ⚠️ An account with NO orders, so the empty branch from chapter 77 runs.
  ordersByAccount: { '4471': 3, '4480': 1, '4492': 2, '4501': 0 },
}

// And the test that prevents the fixture from being simplified later by
// somebody who finds it confusing.
test('fixtures contain the cases the branches need', () => {
  const names = FIXTURES.accounts.map((a) => a.name)
  expect(names.filter((n) => n.startsWith('Henderson')).length).toBeGreaterThan(1)
  expect(FIXTURES.accounts.some((a) => a.owner === 'rep-9')).toBe(true)
  expect(Object.values(FIXTURES.ordersByAccount)).toContain(0)
})

A test suite is only as good as the fixture's worst case. A clean fixture tests the happy path thoroughly and leaves every defensive branch as dead code that first runs in production.

Section 6 is done

Atlas now acts. Eight tools, five gates before execution, and the numbers:

	Start of section	End
Correct tool	0.89	0.99
Correct arguments	0.91	0.98
Result interpreted correctly	0.96	0.99
Terminated correctly	0.98	0.99
End to end	0.86	0.95
Identity failures reaching execution	4.3%	0
Duplicate writes	~4/week	0

Section 7 is the loop itself: planning, termination, subagents and the control-flow bugs that only appear in production.

Key points

Four layers, and the two that catch ~80% need no model.
Assert on the serialised result when testing a leak. A warehouse name in a debug field is the same disclosure.
Keep the permission matrix as reviewable data, not as assertions buried in a test file.
Contract-test the error shapes, not only the success shape. The not-found branch is the one nobody exercises.
Selection cases need three types: should be chosen, should not be chosen, and a near-miss that belongs to another tool.
A sandbox must fail the way production does, including rate limits, auth, and timeout_after_commit.
Assert on counted side effects and on effects that must not occur.
Enforce coverage requirements per tool so a new tool cannot ship without them.
A clean fixture leaves every defensive branch as dead code. Put the ambiguity, the permission case and the empty case in the fixture, with a comment saying why.

Next, and the start of section 7: the loop — plan, act, observe, repeat, and making it stop.