Chapter 68 · Tools and actions

Tool descriptions, and the writing that makes them work

Sixty tokens decide whether a tool is chosen correctly. The five-part structure, the words that measurably help, and how to A/B a description against a test set.

10 min read

Chapter 66 established that a tool description is the entire basis for selection. Chapter 67 found three points of accuracy in rewriting them. This chapter is how.

A tool description is roughly sixty tokens that must answer four questions: what this does, when to use it, when not to, and what comes back. Most descriptions answer only the first.

The five-part structure

export const checkInventory: Tool = {
  name: 'check_inventory',
  description: [
    // 1. WHAT, in one sentence, concrete and present tense.
    'Returns current on-hand and allocated quantities for a SKU.',

    // 2. WHEN. Phrased with the words a person would use, because the
    //    question is in the context and lexical overlap genuinely helps.
    'Use for any question about stock, availability, how many are on hand, ' +
    'or whether something can ship.',

    // 3. WHEN NOT. The part that is almost always missing and that does
    //    the most to prevent wrong selection.
    'Do not use for pricing, lead times, or whether an order has shipped — ' +
    'use lookup_discount_tiers or get_order_status.',

    // 4. WHAT COMES BACK, so the model can judge whether it is done.
    'Returns one row per warehouse with on_hand, allocated and updated_at.',

    // 5. CAVEATS that change behaviour, and only those.
    'Quantities are live and change throughout the day.',
  ].join(' '),
  input_schema: { /* ... */ },
}

Part 3 is the highest-value sentence and it is the one nobody writes. On Atlas, adding explicit negative guidance to five tools moved correct-tool accuracy from 0.94 to 0.97 — more than any other single change to the descriptions.

A description that only says what a tool does leaves the model to infer what it does not do, by elimination, from the other descriptions. Saying it directly is three words and removes the inference.

Parameter descriptions matter more than the tool description

Selection is one failure point; argument construction is the one at 0.91. Parameter descriptions are where that is fought.

input_schema: {
  type: 'object',
  properties: {
    sku: {
      type: 'string',
      // ⚠️ Format, example, and what to do when it is not available. A
      // parameter description with an example reduced malformed SKUs by
      // about two thirds on Atlas.
      description:
        'The SKU code, formatted like SKU-4411. If the user gives a product ' +
        'name instead of a code, use search_products first to find the code.',
    },
    warehouse: {
      type: 'string',
      enum: ['dallas', 'reno', 'columbus'],
      // Say what omitting it does. Otherwise the model invents a default
      // or asks the user unnecessarily.
      description: 'Optional. Omit to return all three warehouses.',
    },
  },
  required: ['sku'],
}

Three things that measurably help in a parameter description:

An example of the format. SKU-4411 rather than "the SKU code".

What to do when the value is not available. Naming the tool that finds it turns a guessed argument into a two-step chain.

What omission means. An optional parameter with no stated default is filled with a guess roughly a third of the time.

Words that help and words that do not

Measured by ablation on Atlas's 60 tool cases, holding everything else fixed.

Change	Δ correct tool	Δ correct args
Add "Do not use for…"	+0.03	0.00
Add a format example to each parameter	0.00	+0.05
Add the user's own vocabulary to "Use for…"	+0.02	0.00
State what the tool returns	+0.01	+0.01
Add "IMPORTANT:" / "ALWAYS" / "NEVER"	0.00	0.00
Add a usage example as JSON	0.00	+0.01
Make the description twice as long	−0.02	−0.01

Two rows deserve comment.

Emphasis markers do nothing. IMPORTANT, ALWAYS, NEVER in capitals are a reflex borrowed from prompt writing and they measured at exactly zero. The content is what carries; the shouting does not.

Longer is worse. Doubling the description cost two points, for the same reason fourteen prompt rules scored worse than six: the surface is a budget and every sentence competes.

Writing the "use for" clause from real questions

The "when to use" sentence should contain the words people actually type, not the words the system uses internally.

-- Harvest the vocabulary. Questions that SHOULD have called this tool,
-- from the log, grouped by the words they used.
select question
  from question_log
 where asked_at > now() - interval '30 days'
   and (question ~* '\y(stock|on hand|available|availability|inventory|in the warehouse)\y')
 order by random() limit 40;

have we got any 4411 left
how many SKU-4411 in dallas
is 4411 available
can we ship 200 of these this week          <- "ship" wasn't in the description
whats the stock position on 4411
do we have enough 4411 for the henderson order

"Can we ship" was not in the description and it is a natural way to ask a stock question. Adding or whether something can ship to the "use for" clause fixed a class of misroute that no amount of reasoning about the tool would have surfaced.

Testing a description

A description change is a behaviour change and gets a test, exactly like a prompt rule.

// evals/tool-selection.ts — no execution, just the choice. Fast and cheap.
export async function scoreSelection(cases: ToolCase[], tools: Tool[]) {
  const rows = await Promise.all(cases.map(async (c) => {
    const request = await planToolCall(c.question, { tools, scope: scopeFor(c) })
    return {
      id: c.id,
      correctTool: request?.tool === c.expect.tool,
      // ⚠️ Expecting NO tool is a case too. A read question that triggers
      // a tool call is a failure, and it is the one nobody writes cases for.
      correctAbstain: c.expect.tool === null && request === null,
      chose: request?.tool ?? null,
    }
  }))
  return { rows, accuracy: mean(rows.map((r) => +(r.correctTool || r.correctAbstain))),
           confusion: confusionMatrix(rows) }
}

tool selection — 60 cases, 14 tools

accuracy 0.97

confusion (expected -> chose):
  check_inventory    -> get_order_status     2   ⚠️ "can we ship" cases
  lookup_discount    -> (none, answered)     1
  search_tickets     -> open_ticket          1   ⚠️ chapter 65's duplicate risk

The confusion matrix is the artefact to work from. Two cases confusing check_inventory with get_order_status names the exact sentence to change, and one case choosing open_ticket where search_tickets was correct is the duplicate-ticket failure visible before it happens.

// A/B a description against the test set. Thirty seconds per variant.
for (const variant of DESCRIPTION_VARIANTS) {
  const tools = withDescription(ALL_TOOLS, 'check_inventory', variant.text)
  const scored = await scoreSelection(cases, tools)
  console.log(variant.id, round(scored.accuracy), scored.confusion)
}

Descriptions are versioned with the code

// Same discipline as chapter 53: hash them, log the hash, so a selection
// regression can be attributed.
export const TOOL_SURFACE_VERSION = createHash('sha256')
  .update(JSON.stringify(ALL_TOOLS.map((t) => [t.name, t.description, t.input_schema])))
  .digest('hex').slice(0, 8)

logger.info('tool_loop.step', { runId, step, toolSurfaceVersion: TOOL_SURFACE_VERSION, ... })

⚠️ Hash the schema as well as the description. A parameter changing from free string to enum is a behaviour change, and it is the kind that improves things — which makes it just as important to be able to attribute.

The failure mode: the description that documented the implementation

get_account_summary is written by whoever built the endpoint:

// ❌ Accurate, and useless for selection. It describes a join.
description:
  'Retrieves the account record with related contacts, the current contract ' +
  'reference, credit terms and the aggregated 12-month order value from the ' +
  'accounts, contacts, contracts and order_summary tables.'

The model calls it rarely, and when it does it is often for the wrong reason. The description names four tables and no question.

// ✅ The same tool, described by what a person would ask it.
description:
  'Returns an overview of one customer account: who the contacts are, the ' +
  'payment terms, the credit limit, and how much they have ordered in the ' +
  'last 12 months. Use when someone asks about a customer generally, their ' +
  'terms, or their recent volume. Do not use for a specific order — use ' +
  'get_order_status — or for the contract wording, which is in the documents.'

Recall for that tool went from 0.61 to 0.94, with no change to the implementation.

The general version of the mistake is worth naming because it is structural rather than careless: the person who writes the tool knows the implementation and writes from it. The description needs to be written from the question side, which usually means by someone looking at the question log rather than at the code.

// A cheap forcing function: the description must name no table, endpoint
// or internal system. Enforced in a test.
test('descriptions do not name internals', () => {
  const INTERNALS = /\b(table|endpoint|API|join|column|schema|record set)\b/i
  for (const tool of ALL_TOOLS) {
    expect(tool.description).not.toMatch(INTERNALS)
  }
})

Write the description from the question, not from the implementation. The model is choosing from a menu, and a menu item that describes the kitchen does not help anyone order.

Key points

Five parts: what, when to use, when not to, what comes back, behaviour-changing caveats.
"Do not use for…" is the highest-value sentence and almost always missing. It was worth three points on Atlas.
Parameter descriptions fight the argument-accuracy problem: give a format example, say what to do when the value is unavailable, and state what omission means.
Emphasis markers measured at exactly zero. Doubling the length cost two points.
Harvest the "use for" vocabulary from the question log. "Can we ship" was a stock question nobody had anticipated.
Score selection without execution, and read the confusion matrix — it names the sentence to change.
Include cases where the correct answer is no tool at all.
Hash descriptions and schemas together, and log the version.
The person who built the tool writes from the implementation. The description has to be written from the question side.

Next: arguments, validation and refusing a bad call — the worst number in the loop.