Chapter 73 · Tools and actions

Idempotency for agent actions

Four ways an action happens twice, only one of which is a retry. Deriving a key that is stable across retries and distinct across genuine repeats — which is harder than it sounds.

12 min read

Chapter 65 listed "the action taken twice" as one of four new failure classes. It is the one with a known solution and the one where the known solution is usually applied wrongly.

Idempotency in an ordinary API is well understood. What is different here is that an agent generates the request, so the usual key — a client-supplied identifier — is generated by something non-deterministic.

Four ways it happens twice

Cause	Looks like	Standard idempotency helps?
Network retry after timeout	One request, two deliveries	✅ Yes
Your code retries	Two deliberate attempts	✅ Yes
The model calls it twice in one run	Two tool_use blocks	⚠️ Only with the right key
The user asks twice	Two separate runs	❌ Must not be deduplicated

The last two are the agent-specific ones and they pull in opposite directions. A model that calls open_ticket twice in one run because the first result was ambiguous should produce one ticket. A rep who opens a ticket on Monday and a second on Tuesday for a recurring fault should get two.

The key must be stable across retries of the same intent and distinct across genuinely separate intents. Getting that boundary wrong fails in one of two bad directions.

Deriving the key

// src/tools/idempotency.ts
export function idempotencyKey(request: ToolRequest, context: RunContext): string {
  return createHash('sha256').update(JSON.stringify({
    tool: request.tool,
    // ⚠️ Canonicalised: key order and whitespace must not change the hash,
    // or a model that emits the same arguments in a different order
    // produces a different key and the deduplication silently stops.
    arguments: canonicalise(request.arguments),
    // The run, NOT the user or the session. Scopes deduplication to one
    // intent: the same run retrying is one action, a new run is a new one.
    runId: context.runId,
  })).digest('hex')
}

function canonicalise(value: unknown): unknown {
  if (Array.isArray(value)) return value.map(canonicalise)
  if (value && typeof value === 'object') {
    return Object.fromEntries(
      Object.entries(value as Record<string, unknown>)
        .filter(([, v]) => v !== undefined)      // undefined and absent are the same
        .sort(([a], [b]) => a.localeCompare(b))
        .map(([k, v]) => [k, canonicalise(v)]),
    )
  }
  if (typeof value === 'string') return value.trim()
  return value
}

Scoping to runId is the decision that resolves the tension above. Within one run the same call is one action; a new question starts a new run and a new key, so the Tuesday ticket is created.

⚠️ Canonicalisation is not optional. Without it, Atlas produced two drafts from one run three times in the first month — the model retried with {sku, quantity} where the first attempt had {quantity, sku}.

Enforcing it in the database

The only place it can be enforced correctly, because it has to be atomic with the effect.

-- The audit table from chapter 70, now also the idempotency ledger.
create table tool_audit (
  id               bigserial primary key,
  idempotency_key  text not null unique,     -- ⚠️ the constraint IS the control
  user_id          text not null,
  tool             text not null,
  arguments        jsonb not null,
  status           text not null,            -- attempting | succeeded | failed | unknown
  result           jsonb,
  error            text,
  started_at       timestamptz not null default now(),
  finished_at      timestamptz
);

// Claim the key first. The insert either wins or it does not, and the
// loser reads the winner's result.
export async function withIdempotency<T>(
  key: string, scope: Scope, request: ToolRequest, run: () => Promise<T>,
): Promise<ToolResult<T>> {
  const claim = await db.query(
    `insert into tool_audit (idempotency_key, user_id, tool, arguments, status)
     values ($1, $2, $3, $4, 'attempting')
     on conflict (idempotency_key) do nothing
     returning id`,
    [key, scope.userId, request.tool, redact(request.arguments)],
  )

  if (claim.rowCount === 0) {
    // Somebody else holds this key. Three cases, and they differ.
    const { rows: [existing] } = await db.query(
      `select status, result, started_at from tool_audit where idempotency_key = $1`, [key])

    if (existing.status === 'succeeded') return { ok: true, data: existing.result, replayed: true }
    if (existing.status === 'failed') return { ok: false, error: 'previous_attempt_failed' }

    // ⚠️ 'attempting' or 'unknown': in flight, or crashed mid-write. Do
    // NOT execute. Do not assume failure either — chapter 70's unknown state.
    return { ok: false, kind: 'in_progress', since: existing.started_at }
  }

  // ... execute, and update the row
}

The in_progress branch is the one that is usually missing. Treating a concurrent attempt as a failure and retrying is how one timeout becomes three tickets.

Idempotency across the boundary

Your key stops at your process. The external system needs its own.

// Pass the key through, so the external system deduplicates too.
export async function sendQuoteExternal(payload: QuotePayload, key: string) {
  return api.post('/quotes/send', payload, {
    headers: { 'Idempotency-Key': key },
  })
}

// Where the API has no such support — common for internal systems — the
// fallback is a natural key and a reconciliation read.
export async function openTicketSafely(args: OpenTicketArgs, key: string) {
  // A deterministic external reference derived from the key.
  const externalRef = `atlas-${key.slice(0, 16)}`

  // ⚠️ Check-then-create is a race. It is still much better than nothing,
  // and the window is milliseconds against a failure mode measured in
  // retries seconds apart.
  const existing = await api.findTicketByReference(externalRef)
  if (existing) return existing

  return api.createTicket({ ...args, externalReference: externalRef })
}

Being explicit that check-then-create is a race, and why it is still worth doing, is the honest version. The alternative — nothing — produced the duplicate tickets.

Reconciling the unknown state

A timeout leaves a row in attempting and the external system in a state you do not know.

// A job, not a request-path concern. It runs every few minutes and
// resolves rows that have been attempting for too long.
export async function reconcileUnknown() {
  const { rows } = await db.query(
    `select * from tool_audit
      where status in ('attempting', 'unknown')
        and started_at < now() - interval '2 minutes'
      order by started_at limit 100`,
  )

  for (const row of rows) {
    const tool = TOOLS[row.tool]
    // ⚠️ Every write tool must be able to answer "did this happen?".
    // A tool that cannot be reconciled cannot be safely retried, and
    // registerTool enforces that it has a reconcile function.
    const actual = await tool.reconcile!(row.idempotency_key, row.arguments)

    await db.query(
      `update tool_audit set status = $2, result = $3, finished_at = now() where id = $1`,
      [row.id, actual.found ? 'succeeded' : 'failed', actual.result ?? null],
    )
    logger.info('reconcile.resolved', { tool: row.tool, key: row.idempotency_key,
                                        outcome: actual.found ? 'succeeded' : 'failed' })
  }
}

// Which makes reconcile a required part of the tool interface for writes.
export interface WriteTool extends Tool {
  reconcile(key: string, args: unknown): Promise<{ found: boolean; result?: unknown }>
}

Making reconcile mandatory in the interface is the structural move. It forces the question "how would I know whether this happened?" at design time, and for two of Atlas's tools the answer was initially "I could not" — which was a design problem, not an operational one.

The model calling the same tool twice, legitimately

Not every repeat is a bug.

"Check stock for SKU-4411 and SKU-4480."

Two calls to check_inventory with different arguments. Different keys, both execute, correct.

"Open a ticket for the seal failure and one for the pump noise."

Two calls to open_ticket, different arguments, both correct. The key includes the arguments, so both proceed.

// The case to catch is identical arguments within one run.
// ⚠️ And it should be surfaced to the model, not silently swallowed —
// a model that believes it opened two tickets will say so.
if (replayed) {
  return { ok: true, data: existing.result, replayed: true,
    message: 'This action was already performed in this run; returning the earlier result.' }
}

Returning replayed: true with a message is what stops the answer saying "I have opened two tickets" when one exists.

Testing it

test('a retried call produces one effect', async () => {
  await resetSandbox()
  const request = { tool: 'draft_quote', arguments: { accountName: 'Henderson Industrial', sku: 'SKU-4411', quantity: 200 } }
  const context = freshContext()

  await executeToolRequest(request, SCOPE.insideSales, context)
  await executeToolRequest(request, SCOPE.insideSales, context)   // same run

  expect(await sandbox.countQuotes({ accountId: '4471' })).toBe(1)
})

test('argument order does not change the key', () => {
  const a = idempotencyKey({ tool: 't', arguments: { sku: 'SKU-4411', quantity: 200 } }, CTX)
  const b = idempotencyKey({ tool: 't', arguments: { quantity: 200, sku: 'SKU-4411' } }, CTX)
  expect(a).toBe(b)
})

test('a new run is a new intent', () => {
  const args = { accountName: 'Henderson Industrial', issue: 'seal failure' }
  expect(idempotencyKey({ tool: 'open_ticket', arguments: args }, { runId: 'r1' }))
    .not.toBe(idempotencyKey({ tool: 'open_ticket', arguments: args }, { runId: 'r2' }))
})

The third test is the one that documents the design decision, and it is the one somebody will want to change. Keeping it with a comment explaining the Monday/Tuesday case is cheaper than re-deriving the reasoning.

The failure mode: the key that included a timestamp

Atlas's first implementation hashed the tool, the arguments, and Date.now().

It was written in ten seconds and it guaranteed a unique key, which felt like the point. Every retry produced a new key, every key was unclaimed, and every retry executed.

The deduplication rate was 0%. Nobody noticed for five weeks, because the audit table filled correctly, the constraint never fired, and duplicates were rare enough — three or four a week — to be attributed to users.

// ❌ Unique, and therefore useless.
createHash('sha256').update(`${tool}:${JSON.stringify(args)}:${Date.now()}`).digest('hex')

// ✅ Deterministic over the intent.
createHash('sha256').update(JSON.stringify({ tool, arguments: canonicalise(args), runId })).digest('hex')

// The test that would have caught it on day one, and the monitor that
// caught it in week five.
test('the same request produces the same key', () => {
  const r = { tool: 'draft_quote', arguments: { sku: 'SKU-4411' } }
  expect(idempotencyKey(r, CTX)).toBe(idempotencyKey(r, CTX))
})

-- ⚠️ A deduplication rate of exactly zero is the signal. Any working
-- implementation catches something, because timeouts happen.
select count(*) filter (where replayed) as replays,
       count(*) as total,
       round(100.0 * count(*) filter (where replayed) / count(*), 2) as replay_pct
  from tool_audit
 where started_at > now() - interval '30 days';
-- Expect 0.1-1%. Exactly 0.00 means the mechanism is not working.

A safety mechanism that has never fired is indistinguishable from one that does not work. Monitor the rate at which it fires, and treat zero as a failure rather than as success.

That is the same rule as the validator that was quietly disabled, arriving from a different direction.

Key points

Four ways an action repeats. Two are agent-specific and pull in opposite directions: a model retrying within a run is one intent; a user asking again is two.
Scope the key to the run, not the user or the session.
Canonicalise arguments. Key order and whitespace changing the hash silently disables deduplication.
Enforce with a unique constraint in the same table as the audit, claimed before the attempt.
Handle three outcomes on a conflict: succeeded, failed, and in progress. Treating in-progress as failure turns one timeout into three tickets.
Pass the key to external systems; where they have no support, use a deterministic reference and a check-then-create, knowing it is a race.
Make reconcile mandatory on the write-tool interface. It forces "how would I know whether this happened?" at design time.
Surface a replay to the model, or the answer claims two actions where one occurred.
Test that the same request produces the same key, and that a new run does not.
Monitor the replay rate. Exactly zero means the mechanism is broken.

Next: querying the database as a tool.