Chapter 140 · Production

The runbook for the three most likely failures

Symptom-indexed, decision-tree shaped, written for someone who did not build it. The three failures that account for most incidents, with the exact commands — and the rules that keep a runbook true.

8 min read

A runbook is read by a tired person under pressure who did not build the system. That reader determines everything about its shape: indexed by symptom rather than by cause, decision trees rather than prose, exact commands rather than descriptions of commands.

Atlas's covers three failures. Not because there are only three, but because these three account for 71% of incidents, and a runbook covering twenty is a document nobody has read.

The index

SYMPTOM                                    -> SECTION
Answers are wrong or stale                 -> 1
Everything is slow, or timing out          -> 2
Errors, or the canary is failing           -> 3
The bill jumped                            -> 2 (usually), then 139's spend page
Someone saw data they should not           -> STOP. Chapter 125, not this.

The last line is deliberate. A disclosure is not a runbook problem; it is an incident with a different procedure and the runbook's job is to say so in one line rather than to offer a tempting partial response.

Failure 1: answers are wrong or stale

The most common, and the one least likely to arrive as an alert.

START: one report, or several?

ONE REPORT
  -> chapter 129's procedure. Not an incident. Investigate, write the eval case.

SEVERAL, SAME TOPIC
  -> likely a corpus problem. Go to 1a.

SEVERAL, UNRELATED TOPICS
  -> likely a config change. Go to 1b.

# 1a. Corpus. Three checks, in order of likelihood.

# Is ingestion behind?
curl -s "$METRICS/ingestion.lag_seconds" | jq .value
# >14400 -> the backlog IS the problem. Section: ingestion lag.

# Did the source change shape?
psql -c "select source, round(avg(chunks)) as now,
         round(avg(chunks) filter (where d < now() - interval '7 days')) as before
         from ingestion_daily where d > now() - interval '14 days' group by 1;"
# a >30% shift -> extraction broke. Pause that source, page the data owner.

# Is the document actually there and current?
psql -c "select id, title, updated_at, acl_group from documents
         where title ilike '%<topic>%' order by updated_at desc limit 10;"

# 1b. Config. What changed, from the one deployment stream.
psql -c "select at, kind, \"from\", \"to\", actor from deployment_event
         where at > now() - interval '48 hours' order by at desc;"

# If kind='prompt' -> roll back. One flag, ~10 s.
flagctl set prompt.version <previous>

# If kind='model' and nobody did it -> the provider rolled. Chapter 109.
#   Do NOT attempt a prompt fix at 3am. Record it, raise the ticket,
#   and degrade to the previous model id if it is still servable.

# If nothing changed -> compare against a corpus snapshot. Chapter 112.
evalctl run --config production --corpus snapshot-$(date -d '1 month ago' +%Y-%m-01)
# score recovers -> the corpus moved. Not a rollback; a retrieval fix.
# score does not  -> the model or judge moved. Check the composite version.

Failure 2: everything is slow

START: is it the provider or us?

  Open the latency dashboard. The provider's reported p95 is plotted
  against ours on the same panel (chapter 134).

  BOTH HIGH   -> theirs. Go to 2a.
  OURS ONLY   -> ours.   Go to 2b.

# 2a. Theirs. Degrade rather than wait.
flagctl set mode reduced          # cheap model, no subagents, 4-step cap
# still bad after 5 min:
flagctl set mode retrieval-only   # no model at all; chapter 135

# Confirm degraded mode is actually serving, which is the step people skip:
curl -s "$APP/health" | jq '.readiness, .mode'

# 2b. Ours. Three causes, in order of frequency.

# i. Runs got longer. The terminated_by distribution is the tell.
psql -c "select terminated_by, count(*),
         round(100.0*count(*)/sum(count(*)) over (),1) as pct
         from audit_run where at > now() - interval '1 hour' group by 1;"
# natural share below ~88% -> a control-flow regression. Roll the
# prompt back; chapter 90 is the reading.

# ii. A tool backend is slow. Tool spans are children of their step.
#     Find the slow one, disable it, say so in the answer.
flagctl set tool.<name>.enabled false

# iii. Retries eating the deadline. Chapter 134.
curl -s "$METRICS/provider.retry_rate" | jq .value      # >5% is the signal

Failure 3: errors, or the canary is failing

START: read the canary's failure reason. It is one of four.

Canary says	Cause	Action
auth / 401	Credential expired or rotated	Rotate; this is the one that needs the owner
timeout / 5xx	Provider	Failure 2a
no results	Index empty or unreachable	Below
schema invalid	Prompt or model produces bad structure	Roll the prompt flag back

# "No results" is the frightening one and usually has a mundane cause:
# a reconciliation or a re-index emptied or replaced the index.
psql -c "select count(*) from chunks;"
# near zero -> STOP. Do not re-ingest; that is 1.5 days (chapter 138).
#              Point at the previous index by flag and investigate.
flagctl set index.active <previous-index-id>

# And check the guard that should have prevented it:
psql -c "select at, deleted, guard_margin from reconciliation_run
         order by at desc limit 5;"

The instruction not to re-ingest is the single most valuable line in Atlas's runbook. The instinct under pressure is to rebuild, the rebuild takes a day and a half, and the previous index is sitting there behind a flag.

Rules that keep a runbook true

const RULES = [
  'Symptom-indexed. The responder has a symptom, never a cause.',
  'Exact commands, copy-pasteable, with the env var named.',
  'Every branch ends in an action or an explicit escalation.',
  'State what NOT to do, where the instinct is wrong.',
  'Date every section and name an owner.',
  'Exercise it quarterly. An unexercised runbook is fiction.',
]

// The quarterly exercise is a drill from chapter 135's list, run by
// someone who did not write the runbook. Atlas's three exercises found:
//   - two commands referencing a renamed metric
//   - a flag name changed in a refactor
//   - a psql query against a dropped column
// Three of three exercises found something. That is the expected rate,
// and it is why the exercise is the rule that matters most.

The fourth rule earns its place twice in this chapter: do not re-ingest, and do not attempt a prompt fix at 3am. Both are places where a competent engineer's instinct makes the incident longer, and a runbook that only lists correct actions leaves them unaddressed.

A runbook is not documentation of the system. It is a set of decisions made in advance, by people with time, for people without it.

That closes section 11. The system is now built, measured, secured and operable.

Next, and the start of section 12: interface and adoption. Everything so far assumes people use the thing. The last section is about whether they do — how the answer is presented, how trust is built and lost, onboarding, the feedback loop that makes the eval suite grow, measuring adoption honestly, and what to do when the people it was built for go back to email.

Key points

Index by symptom. The responder has a symptom and never a cause.
Cover the few failures that account for most incidents. Atlas's three cover 71%; a runbook covering twenty is unread.
Send disclosure straight to the incident procedure in one line, rather than offering a tempting partial response.
For wrong answers, branch on one report against several, and on same-topic against unrelated. That one split separates a corpus problem from a config problem.
Read the one deployment stream first. Prompt rollbacks are one flag and ten seconds.
Plot the provider's latency against yours on one panel. It answers "theirs or ours" immediately.
The terminated_by distribution is the tell for a control-flow regression: natural share below 88% with everything returning 200s.
An empty index means point at the previous one by flag. Do not re-ingest; that is a day and a half.
State what not to do. Two of Atlas's most valuable lines are prohibitions against a competent engineer's instinct.
Exercise quarterly, with someone who did not write it. Three of three exercises found a stale command.