Chapter 75 · Tools and actions

Calling an internal API as a tool

Someone else's uptime on your request path. Authentication as the agent or as the user, the degradation that must not look like an answer, and the contract test that catches a breaking change before your users do.

11 min read

Chapter 74 wrapped a database you control. This chapter wraps a system you do not, which changes three things: authentication, availability, and the contract.

Chapter 26 covered pulling from an API for ingestion, where slowness is tolerable. Here the call is on the request path inside a four-second budget.

Authenticating as the agent or as the user

The first decision, and it determines the entire permission story.

// ✅ Preferred: act as the user, so the API applies its own rules.
const response = await api.get(`/tickets/${id}`, {
  headers: { authorization: `Bearer ${await delegatedToken(scope.userId)}` },
})

// ⚠️ Common: a service account. Then Atlas is a confused deputy unless it
// enforces the user's permissions itself, in the tool, every time.
const response = await api.get(`/tickets/${id}`, {
  headers: { authorization: `Bearer ${secrets.serviceToken}` },
})
// ...and the check that makes it safe, which is easy to omit:
if (!scope.accountIds.includes(response.data.accountId)) {
  return { ok: false, error: 'not_found' }   // not 'forbidden'. Chapter 63.
}

The confused-deputy problem is chapter 120's subject in full. The operational rule here: a service account must be scoped as narrowly as the broadest user, and the tool must filter down from there. A service token with company-wide read access, used by a tool that forgets its filter, is every permission control in this series bypassed in one line.

// Make the filter structural rather than remembered — the same move as
// chapter 13's scoped query builder.
export async function callAsUser<T>(
  path: string, scope: Scope, extract: (data: T) => { accountId?: string },
): Promise<ToolResult<T>> {
  const data = await apiCall<T>(path)
  const owner = extract(data).accountId
  // ⚠️ A resource with no owner is not automatically public.
  if (owner === undefined) return { ok: false, error: 'unclassifiable_resource' }
  if (!scope.accountIds.includes(owner)) return { ok: false, error: 'not_found' }
  return { ok: true, data }
}

Degrading without lying

Chapter 12 made the point and it is worth the full treatment here, because it is the failure with the worst consequence-to-likelihood ratio.

// ❌ An empty list becomes "there are no open tickets", said with total
//    confidence, when the truth is "I could not check".
catch (error) { return [] }

// ✅ An explicit unavailable result, which the model must report.
catch (error) {
  if (isTimeout(error) || isUnavailable(error)) {
    return {
      ok: false,
      error: 'source_unavailable',
      // ⚠️ Phrased for the model to relay, because it will be relayed.
      message: 'The ticket system did not respond. I could not check whether ' +
               'there are existing tickets for this.',
      retryable: true,
    }
  }
  throw error
}

<!-- And the prompt rule that makes it stick. Without it, a model given an
     unavailable result sometimes answers from retrieval instead and does
     not mention the gap. -->
If a tool reports that a system was unavailable, say so explicitly in your
answer. Do not substitute information from documents for a live check.

Atlas's measurement before that rule existed: of 61 unavailable results over a month, 14 produced answers that did not mention the failure. Nine of those answered the question anyway from indexed content, which is a stale answer presented as a live one.

The circuit breaker

A failing dependency should fail fast rather than consuming the whole latency budget on every request.

// src/tools/circuit.ts
export class CircuitBreaker {
  private failures = 0
  private openedAt: number | null = null

  constructor(private readonly threshold = 5, private readonly cooldownMs = 30_000) {}

  async run<T>(fn: () => Promise<T>): Promise<T> {
    if (this.openedAt !== null) {
      if (Date.now() - this.openedAt < this.cooldownMs) throw new CircuitOpen()
      // Half open: let one request through to test.
      this.openedAt = null
    }
    try {
      const result = await fn()
      this.failures = 0
      return result
    } catch (error) {
      // ⚠️ Only infrastructure failures open the circuit. A 404 is a
      // correct answer and must not count toward it.
      if (isInfrastructureFailure(error)) {
        if (++this.failures >= this.threshold) {
          this.openedAt = Date.now()
          logger.error('circuit.opened', { threshold: this.threshold })
        }
      }
      throw error
    }
  }
}

The distinction in that comment is the one usually missed. Counting 4xx responses toward the breaker means a run of legitimate not-founds disables a healthy integration.

// An open circuit is a known state, surfaced rather than retried.
if (breaker.isOpen()) {
  return { ok: false, error: 'source_unavailable',
    message: 'The ticket system is currently unavailable.', retryable: false }
}

Timeouts inside the budget

// The tool's timeout must be well inside the request's remaining budget,
// and the budget is what remains, not the total.
export async function callWithBudget<T>(fn: (signal: AbortSignal) => Promise<T>, ctx: RunContext) {
  const elapsed = Date.now() - ctx.startedAt
  const remaining = ctx.limits.maxMs - elapsed
  // ⚠️ Leave room for generation. A tool allowed the whole remainder
  // returns just in time for the request to time out anyway.
  const allowance = Math.min(remaining - GENERATION_RESERVE_MS, MAX_TOOL_MS)

  if (allowance < 500) {
    return { ok: false, error: 'no_time_budget',
             message: 'Not enough time remained to check that system.' }
  }
  return fn(AbortSignal.timeout(allowance))
}

Returning a distinct no_time_budget rather than attempting the call is worth the branch: it is honest about why, and it does not spend 400ms confirming that 400ms was not enough.

Contract tests

An internal API changes without telling you. The tool breaks in production, and the symptom is a model apologising.

// Run against the real API, on a schedule, not only in CI — the point is
// to catch THEIR change, not yours.
describe('ticket API contract', () => {
  test('get ticket returns the fields the tool reads', async () => {
    const ticket = await api.get(`/tickets/${FIXTURE_TICKET_ID}`)
    // ⚠️ Assert on exactly the fields the tool uses, and their types.
    // Asserting on the whole shape fails on every harmless addition.
    expect(ticket).toMatchObject({
      id: expect.any(String),
      status: expect.stringMatching(/^(open|in_progress|resolved|closed)$/),
      accountId: expect.any(String),
      createdAt: expect.any(String),
    })
  })

  test('status enum has not gained values', async () => {
    const { statuses } = await api.get('/tickets/meta')
    // A new status the tool does not handle is a silent misclassification.
    expect(new Set(statuses)).toEqual(new Set(KNOWN_STATUSES))
  })
})

The second test is the one that pays. Atlas's ticket system added a pending_customer status, which the tool mapped to "unknown" and the model reported as "the ticket is in an unrecognised state" for nine days.

// Run it every hour against the real API and alert on failure.
// ⚠️ A contract test is a monitor. In CI only, it tells you about your
// changes; on a schedule, it tells you about theirs.

Caching, and when not to

// A short cache absorbs the common case of the same question twice in one
// conversation. The TTL comes from the source's tolerance, chapter 19.
const CACHE_TTL: Record<string, number> = {
  get_ticket: 30_000,          // changes often; 30s is enough for one session
  get_account_summary: 300_000,
  check_inventory: 0,          // ⚠️ never. A stale stock figure is the
                               //    chapter 12 failure, with a tool in front.
}

Caching a live-queried source for longer than its tolerance reintroduces exactly the staleness the live query was chosen to avoid. The entry of 0 is a decision recorded in code rather than an omission.

The failure mode: the retry that filled the queue

The ticket API slows down — not failing, just taking eight seconds instead of 300ms.

Atlas's tool has a 2.5-second timeout and retries twice. Each user request now makes three ticket-API calls, all of which time out. The API is already struggling, and Atlas has tripled its load.

Forty reps are asking questions. The API, which was slow, becomes unavailable. The team that owns it sees a traffic spike from the assistant and asks for it to be turned off.

// Three fixes, and the second is the one that was missing.

// 1. The circuit breaker above. Five failures and Atlas stops calling.

// 2. ⚠️ Do not retry a timeout on the request path. A retry helps a
//    transient network error and harms an overloaded dependency, and you
//    cannot tell them apart from the client. Retry at most once, and only
//    on a connection error — never on a timeout.
const RETRYABLE_ON_REQUEST_PATH = new Set(['ECONNRESET', 'ECONNREFUSED'])

// 3. A concurrency bound across the whole process, so forty simultaneous
//    users cannot become forty simultaneous calls.
const ticketApiConcurrency = pLimit(6)

// And the measurement that makes the whole thing visible to both teams.
logger.info('tool.external_call', {
  tool, outcome, latencyMs, circuitState: breaker.state,
  // ⚠️ Report calls per user request, not just total calls. The tripling
  // was invisible in a total-calls graph that was already rising.
  callsThisRequest: ctx.externalCalls,
})

On the request path, a retry is a load multiplier applied exactly when the dependency can least afford it. Retries belong in the ingestion pipeline, where nobody is waiting; here, fail fast and say so.

That is the inverse of chapter 26's advice, and deliberately — the same dependency, the same failure, and the opposite correct response, because one path has a person waiting and the other does not.

Key points

Authenticate as the user where delegated auth exists. A service account makes Atlas a confused deputy unless the tool filters, every time.
Scope a service account as narrowly as the broadest user, and make the ownership filter structural.
A resource with no owner is not automatically public.
Never return an empty result for a failed call. An explicit unavailable result, plus a prompt rule requiring it to be reported — 14 of 61 failures were silently papered over before that rule.
Open the circuit on infrastructure failures only. Counting 4xx disables a healthy integration.
Size the timeout from the remaining budget with generation reserved, and return a distinct result when there is no time left.
Contract-test on a schedule, not only in CI. In CI it catches your changes; on a schedule it catches theirs.
Assert on the fields you read and on enum closure. A new status value is a silent misclassification.
Cache to the source's tolerance, and record a zero TTL as a decision.
Do not retry timeouts on the request path. A retry is a load multiplier exactly when the dependency can least afford it.

Next: composing tools — letting the model chain them, and keeping the chain on task.