Chapter 67 · Tools and actions

Designing a tool surface a model can actually use

Granularity, overlap and count. Why fourteen tools scored worse than four, the five rules that fix a surface, and the per-request filter that is both a cost and a quality lever.

11 min read

Chapter 66 found ten points of tool-selection accuracy in the surface rather than the model. This chapter is how that was recovered.

The framing: a tool surface is a menu you are asking someone to choose from with no training and no ability to ask what the items are. Most surfaces are designed as an API — complete, orthogonal, composable — and those are the wrong properties.

Granularity

The first decision, and the one with the widest reasonable range.

// ❌ Too fine. Eight tools where the model must know the sequence, and
//    every step is a model call. Chapter 66's cost structure makes this
//    expensive as well as unreliable.
get_account, get_account_contracts, get_contract_terms, get_contract_discount,
get_orders, get_order_lines, calculate_discount, compare_values

// ❌ Too coarse. One tool whose behaviour depends on a free-text argument,
//    which is a second prompt the model must get right with no schema help.
query_business_data({ request: 'Henderson discount vs current orders' })

// ✅ One tool per question a person would ask.
check_inventory, get_order_status, lookup_discount_tiers, get_account_summary,
search_tickets, draft_quote, open_ticket

Size a tool to a question somebody asks, not to an endpoint you have. The right granularity is the one where a single call answers a real question.

The test is concrete: take the 34 questions and count calls per question. More than two for a common question means the tools are too fine; a tool that appears in every question means it is too coarse.

The five rules

1. No two tools answer the same question

The largest single gain on Atlas. search_tickets and get_ticket_history both plausibly answered "has this come up before?", and selection between them was close to a coin flip.

// Detect overlap mechanically rather than by reading. Embed the
// descriptions and look for near-neighbours — the same technique the
// retrieval section uses, pointed at your own surface.
export async function findOverlaps(tools: Tool[], threshold = 0.82) {
  const vectors = await embed(tools.map((t) => `${t.name}: ${t.description}`))
  const pairs: Array<[string, string, number]> = []
  for (let i = 0; i < tools.length; i++) {
    for (let j = i + 1; j < tools.length; j++) {
      const sim = cosineSimilarity(vectors[i], vectors[j])
      if (sim > threshold) pairs.push([tools[i].name, tools[j].name, sim])
    }
  }
  return pairs.sort((a, b) => b[2] - a[2])
}
// Atlas: search_tickets / get_ticket_history  0.91  ⚠️ merged
//        check_inventory / get_stock_level    0.88  ⚠️ merged
//        draft_quote / create_quote           0.94  ⚠️ one was dead code

Three overlapping pairs in fourteen tools, and one of them was a tool nobody had removed when it was superseded.

2. A tool's name says what it does, not what it wraps

// ❌ Named after the system. The model has no idea what the system holds.
query_erp, call_wms, fetch_crm_record

// ✅ Named after the question.
check_inventory, get_order_status, get_account_summary

3. Required arguments are things the model can actually know

// ❌ The model does not have an internal account id. It will invent one,
//    or guess from the question, which is chapter 69's failure.
get_account_summary({ accountId: 'ACC-4471-INT' })

// ✅ Take what the person said, and resolve it in your code.
get_account_summary({ accountName: 'Henderson' })
// ⚠️ Resolution — including the two-Henderson case — happens in the tool,
//    where it can ask for clarification. Chapter 69.

4. Enums wherever the set is known

// Every free string is a chance to produce a value your code must handle.
warehouse: { type: 'string', enum: ['dallas', 'reno', 'columbus'] }
status: { type: 'string', enum: ['open', 'in_progress', 'resolved', 'closed'] }

On Atlas, converting three free-string parameters to enums removed 40% of argument-validation failures in one change. The model cannot produce Dallas, TX if the schema does not allow it.

5. One tool, one effect

// ❌ A flag that changes what the action IS. Confirmation, audit and
//    permission all differ between the two behaviours, and one definition
//    cannot carry both.
manage_quote({ action: 'draft' | 'send' | 'cancel', ... })

// ✅ Three tools. draft_quote is reversible; send_quote is not and
//    requires confirmation. Chapter 71.
draft_quote, send_quote, cancel_quote

How many is too many

Measured on Atlas, same cases and model throughout:

Tools in context	Correct tool	Tokens	Note
4 (filtered)	0.99	520	—
7	0.98	910	—
11	0.96	1,430	—
14	0.94	1,820	After overlap removal
14 (with overlap)	0.89	1,820	Before
22	0.88	2,900	Hypothetical, with ticket sub-tools

Two separate effects, and they are worth distinguishing. Overlap costs five points at a fixed count (0.89 → 0.94 at fourteen). Count costs another five on top (0.94 at fourteen → 0.99 at four). Fixing overlap is permanent; fixing count is per request.

Filtering per request

// src/agent/select-tools.ts
export function selectTools(
  question: string, scope: Scope, options: { maxTokens: number },
): Tool[] {
  // 1. Permission first. A tool the asker cannot use is never shown —
  //    which is also why the model never explains that it exists.
  const permitted = ALL_TOOLS.filter((t) => t.permitted(scope))

  // 2. Always-available core. A handful that are relevant to most
  //    questions and whose absence would be worse than their cost.
  const core = permitted.filter((t) => t.core)

  // 3. Relevance, by the same hybrid retrieval as chapter 43 — the tool
  //    descriptions are a tiny corpus and the machinery already exists.
  const scored = rankToolsByRelevance(question, permitted.filter((t) => !t.core))

  const selected = [...core]
  let tokens = sum(core.map((t) => t.tokens))
  for (const tool of scored) {
    if (tokens + tool.tokens > options.maxTokens) break
    // ⚠️ A relevance floor. Padding to the budget with marginal tools is
    // exactly the count problem the filter exists to avoid.
    if (tool.score < 0.25) break
    selected.push(tool); tokens += tool.tokens
  }
  return selected
}

Three things that make this safe rather than a new failure source.

Permission filtering is first, so an unavailable tool is invisible rather than refused — the same reasoning as the abstention phrasing for permissions.

A core set is always present. Filtering can be wrong, and a question that needed check_inventory and did not get it is a worse failure than four extra tool definitions.

A relevance floor, not a budget fill. Stopping at the floor is what delivers the four-tool case rather than always sending eleven.

// And the measurement that proves the filter is not losing tools.
test('the filter does not drop a needed tool', async () => {
  for (const c of TOOL_CASES) {
    const selected = selectTools(c.question, scopeFor(c), { maxTokens: 1_200 })
    expect(selected.map((t) => t.name)).toContain(c.expect.tool)
  }
})

That test is non-negotiable. A filter that improves selection accuracy by hiding the right tool is improving a metric by removing the cases it would have failed.

Tools and retrieval are one surface

The decision chapter 66 ended on: a tool competes with indexed content for the same question.

// So design them together, per source, in the registry.
{
  id: 'pricing_workbook',
  route: 'index',
  // ⚠️ Rows are suppressed; only the summary is retrievable. The tool owns
  // specific values and comparisons. Without this they compete and the
  // index wins, because it is always "good enough".
  indexedContent: 'summary_only',
  tool: 'lookup_discount_tiers',
}

Question	Answered by
"What pricing information is there?"	Index — the summary chunk
"What is the north tier-3 discount?"	Tool
"Which region has the best tier-3 rate?"	Tool — a comparison across rows
"Does tier 3 need approval?"	Index — the notes region

Each question has exactly one correct route, and the registry entry is what makes that true rather than aspirational.

The failure mode: the surface that grew to match the API

Six months in, Atlas has 31 tools. Each was added for a real request. Nobody designed a 31-tool surface; it accreted.

Correct-tool accuracy is 0.84, down from 0.99, and the per-request filter is hiding it in the aggregate — the filter sends four tools and the four are frequently the wrong four, because with 31 candidates the relevance ranking has far more near-neighbours to confuse.

// The audit, run quarterly. Three questions per tool, and most tools
// fail at least one.
export async function auditSurface(tools: Tool[], log: CallLog) {
  return tools.map((t) => ({
    name: t.name,
    // 1. Is it used? Chapter 66's underuse check.
    calls30d: log.countFor(t.name, 30),
    // 2. Does anything else answer the same question?
    overlaps: overlapsFor(t.name),
    // 3. When it IS the right tool, is it chosen?
    recall: log.recallFor(t.name),
  }))
}

tool surface audit — 31 tools

  7 tools: 0 calls in 30 days          -> remove
  5 tools: overlap > 0.85 with another -> merge
  4 tools: recall < 0.80 when correct  -> rewrite the description
 15 tools: healthy

after: 19 tools, correct-tool accuracy 0.84 -> 0.96

Seven tools with zero calls is the number that makes this audit worth scheduling. Each one costs tokens on every request it is selected into, adds a near-neighbour to every relevance ranking, and answers nothing.

A tool surface needs pruning the way a prompt does. Every tool that is never called is making the ones that are called harder to choose.

Key points

A tool surface is a menu for someone who cannot ask what the items are. API properties — complete, orthogonal — are the wrong goals.
Size tools to questions people ask, not to endpoints you have.
Detect overlap mechanically by embedding descriptions. Atlas had three overlapping pairs in fourteen tools, one of them dead code.
Name tools after the question, not the system they wrap.
Required arguments must be things the model can know. Resolve internal identifiers inside the tool.
Enums wherever the set is known: three conversions removed 40% of argument failures.
One tool, one effect. A flag that changes what the action is cannot carry one set of confirmation and audit rules.
Overlap and count are separate costs: five points each on Atlas.
Filter per request with permission first, a core set always present, and a relevance floor — and test that the filter never drops a needed tool.
Design tools and indexed content together. Each question should have exactly one correct route.
Audit quarterly. Seven zero-call tools were making every other tool harder to choose.

Next: tool descriptions, and the writing that makes them work.