Chapter 139 · Production

On-call for an agent: what to alert on

Nine pages and eleven tickets, each with an action. Why quality metrics must never page, the three signals specific to this system, and the rotation design for a two-person team.

8 min read

Every chapter in this section has produced a metric. This one decides which of them wake someone up, and the answer is fewer than the chapter count suggests.

One rule governs the whole list: a page needs an action the person on call can take at 3am. Everything else is a ticket. That rule is what keeps the rotation answerable, and the hard part is applying it to quality signals, which feel urgent and have no 3am action.

Quality must not page

// The tempting alert, and the reason it is wrong.
//   ❌ "groundedness below 0.90 for 30 minutes"
//
// It needs a judge (minutes of latency and a model call), the threshold
// is noisy at small volumes, and the action at 3am is... nothing. You
// cannot fix a prompt safely without the chapter 108 paired run, which
// takes six minutes and a slice table and a clear head.

Quality degradation is real and it is a next-morning problem. Atlas's quality signals go to the weekly drift run and a daily digest, and the one quality-adjacent thing that does page is citation validation failure — because that is a deterministic check, it indicates something structurally broken, and the action is to roll the prompt flag back.

Nine pages

const PAGES = [
  { on: 'error rate >5% for 5 min', action: 'check the provider; degrade if theirs' },
  { on: 'canary failing 3 consecutive runs', action: 'the runbook decision tree' },
  { on: 'p95 latency >20 s for 10 min', action: 'degrade at the threshold; find the slow dependency' },
  { on: 'citation validation failures >1%', action: 'roll the prompt flag back' },
  { on: 'any cross-user disclosure detected', action: 'full stop; chapter 125' },
  { on: 'any unconfirmed write executed', action: 'disable writes; chapter 125' },
  { on: 'global spend >3x trailing median over 1 h', action: 'check terminated_by; cap or stop' },
  { on: 'circuit breaker open >10 min', action: 'confirm degraded mode is serving' },
  { on: 'vector index unreachable', action: 'read-only-corpus; page the data owner' },
]

Nine. Atlas's actual page rate is about one every three weeks, which is the number that makes the rotation sustainable for a small team.

Two of the nine have no threshold: cross-user disclosure and an unconfirmed write. A rate implies a tolerable number, and chapter 127 set both to zero.

Eleven tickets

const TICKETS = [
  'ingestion lag >4 h',                        // chapter 138
  'parse failure rate >2% on any source',
  'chunks-per-document median shifted >30%',
  'refusal rate moved >5 points with no deploy',   // chapter 111 — a silent model roll
  'escalation rate outside 12-35%',            // chapter 131
  'prompt cache hit rate <60%',                // chapter 132 — the eleven-day bug
  'resolved model id changed',                 // chapter 112
  'judge model id changed',
  'weekly drift run moved >3 points',
  'reconciliation guard fired',                // chapter 138
  'out-of-profile citation rate doubled',      // chapter 128
]

The fourth is worth singling out. A refusal rate moving with no deploy is almost always the provider rolling a model version forward, and it shows there days before it shows anywhere else. It is a ticket rather than a page because the correct response is chapter 109's comparison, which is a morning's work.

Three signals specific to this system

The alerts above that have no analogue in an ordinary service, and which an on-call rotation inherited from web operations will not have.

// 1. terminated_by distribution. Chapter 128 called it the highest-value
//    attribute. A shift from 94% natural to 81% natural, with the
//    difference going to step_limit, is a control-flow regression that
//    no error rate shows — every one of those runs returned a 200.
const naturalShare = byTerminatedBy.natural / total
if (naturalShare < 0.88) ticket('termination distribution shifted')

// 2. Spend per answer, as a reliability signal. Chapter 134 wired it to
//    the breaker. A provider returning valid output that now needs nine
//    steps is a degradation with 100% availability.

// 3. Out-of-profile citations. Chapter 125's incident ran five days
//    because nothing watched the rate of answers citing documents
//    outside the caller's usual groups. It is not a permission failure
//    — it is a change in pattern, which is what a misconfiguration
//    looks like from outside.

All three share a property: the system is returning 200s and the users are, for a while, not complaining. They are the signals that distinguish operating an agent from operating an API.

The rotation, for a team of two

// An honest design for the size of team that usually builds this.
const ROTATION = {
  hours: 'business hours, primary only. Nothing here is life-critical.',
  overnight: 'pages queue; the degraded mode serves. Chapter 135 is the night shift.',
  escalation: 'one named secondary, for the two zero-threshold security pages only',
  handover: 'a weekly 10-minute read of the digest, not a meeting',
}

// Which is only defensible because of chapter 135. An internal
// assistant that degrades to permission-correct document search at 2am
// does not need a human at 2am. Building the degraded mode is what buys
// the rotation, and that is the strongest practical argument for it.

The two security pages are the exception and they wake the secondary, because disclosure cannot be undone and every hour of delay widens the blast radius.

The daily digest

// One message, every morning, read in two minutes. It is where the
// eleven tickets actually surface, and where a human notices the thing
// no threshold was set for.
const DIGEST = [
  'yesterday: questions, p50/p95, error rate, spend',
  'terminated_by distribution against the trailing week',
  'refusal rate by reason, against the trailing week',
  'top 3 slowest runs, with links',
  'top 3 most expensive runs, with links',
  'any threshold crossed, with the metric',
  'ingestion: lag, parse failures, chunks-per-document',
  'open tickets from this list',
]

The "top 3 slowest, with links" line has produced more findings than any alert in the chapter. Reading three real traces a day builds the intuition that makes someone able to answer a page, and it costs two minutes.

What on-call needs before the first page

const PREREQUISITES = [
  'The runbook, symptom-indexed. Chapter 140.',
  'The kill switches, and who has access. Chapter 125.',
  'The three dashboards. Chapter 128.',
  'A link from every alert to the exact query or dashboard panel.',
  'The composite version, visible. Chapter 137.',
  'One practice incident, run deliberately.',
]

The fourth is the cheapest and most often missing. An alert that says "error rate >5%" and links to nothing costs the responder five minutes of navigation while the incident runs. Every Atlas alert carries the dashboard URL and the SQL, which is a one-line change per alert made once.

Key points

A page needs an action the responder can take at 3am. Everything else is a ticket.
Quality must not page. It needs a judge, it is noisy, and the fix needs a paired eval run and a clear head.
Nine pages, two of them with zero threshold: cross-user disclosure and an unconfirmed write. Atlas pages about once every three weeks.
Eleven tickets, including the resolved model id changing — which is the cheapest alert in the system and catches the hardest-to-diagnose cause.
A refusal rate moving with no deploy is usually a silent model roll, days before anything else shows it.
Three signals have no analogue in ordinary operations: the terminated_by distribution, spend per answer as a reliability signal, and out-of-profile citations. All three occur while the system returns 200s.
For a team of two: business hours primary, pages queue overnight, degraded mode is the night shift, and one secondary for the two security pages.
Building the degraded mode is what buys the rotation. That is its strongest practical argument.
Send a daily digest. "Top 3 slowest and most expensive runs, with links" has produced more findings than any alert.
Link every alert to the exact dashboard panel and query. It is one line per alert and saves five minutes per incident.

Next: the runbook for the three most likely failures — what the responder opens.