Chapter 129 · Production
Debugging a bad answer after the fact
A decision procedure that localises a wrong answer in under ten minutes, in the order that eliminates the most ground first — and the measured distribution that says why the model is the last thing to look at.
8 min read
A report arrives: "Atlas told me Northline is on 30-day terms and that's wrong." One sentence, no run id if you are unlucky, and an expectation of an explanation.
The instinct is to read the answer and think about the prompt. Chapter 107 measured where failures actually are, and the prompt is near the bottom:
| Cause | Share |
|---|---|
| Retrieval — it never had the right passage | 51% |
| Generation — it had it and answered wrong | 22% |
| Composition — layers fine, whole wrong | 18% |
| Action — wrong tool or arguments | 9% |
So the procedure goes in that order, and most investigations stop at the first step.
Step 0: find the run
-- With a run id it is a lookup. Without one, the question text in the
-- audit log is what makes this possible at all -- chapter 124's decision
-- to store it in full.
select run_id, at, actor->>'userId', answer_hash, config
from audit_run
where actor->>'userId' = $1
and at between $2 and $3
and question ilike '%northline%'
order by at desc limit 20;If this returns nothing, the investigation is already over and the answer is interesting: the user is misremembering, or asked a different system. Atlas sees that in roughly one report in eight.
Step 1: did it have the answer?
// The audit record holds the retrieved chunk ids. Two questions, both
// mechanical.
const retrieved = await chunks(auditRun.retrieved.map((r) => r.chunkId))
// a. Was the correct passage among them?
const hadIt = retrieved.some((c) => c.documentId === 'supply-agreement-northline-2026')
// b. If not, does it exist and would the filter have allowed it?
const gold = await findChunk('Northline', 'payment terms')Four outcomes, and each names a different owner.
| Diagnosis | Owner | |
|---|---|---|
| Not in the corpus at all | An ingestion gap | Data |
| Present, excluded by the filter | A permission or classification issue | Access |
| Present, allowed, ranked below the cut | A retrieval problem | Retrieval |
| Present and retrieved | Continue to step 2 |
Row two is the one that looks like a model failure and is not. The system refused or answered thinly because the document was invisible to that user, which is correct behaviour reported as a bug — and the fix is a classification change, not a prompt change.
// The replay that settles row three, and it is exact. Retrieval is
// deterministic given the same index and filter (chapter 127).
const replayed = await retrieve(auditRun.question, {
corpus: auditRun.config.corpusSnapshotId,
groups: auditRun.actor.groups,
})
const rank = replayed.findIndex((c) => c.documentId === gold.documentId)
// rank 47 -> the retriever had it and ranked it out. A real finding with
// a reproducible test case.Step 2: did the context contain it, and did the answer use it?
// Retrieved and in-context are different. The chapter 61 budget drops
// chunks, and a passage retrieved at rank 11 may not have survived.
const inContext = auditRun.retrieved.filter((r) => r.usedInAnswer || r.inContext)// Then the near-miss check, which is the single most common step-2 cause.
// Chapter 62 described it: the right document, the wrong clause.
const cited = auditRun.citations.map((n) => inContext[n - 1])
// cited[0] is the 2024 agreement, section "Late returns".
// The 2026 agreement was rank 3 and also in context.That is the Northline case, and it recurs because adjacent clauses share almost all their vocabulary. The diagnosis is specific — two competing documents, the superseded one ranked higher — and the fixes are known: the heading path in context, a freshness signal on the superseded document, or removing it from the corpus.
Step 3: only now, the model
// Re-run with the exact context from the audit record. Not the current
// context -- the corpus has moved -- which is why the snapshot id is
// stored.
const rerun = await generate(auditRun.question, {
context: await chunks(auditRun.retrieved.map((r) => r.chunkId)),
promptVersion: auditRun.config.promptHash,
modelId: auditRun.config.modelId,
})| Rerun result | Diagnosis |
|---|---|
| Same wrong answer | Deterministic generation failure. Reproducible, so it becomes an eval case. |
| Correct | The 0.4% sampling variance, or something not captured. Keep looking. |
| Correct, and the stored config differs from current | A config change already fixed it. Confirm and close. |
// Run it five times rather than once. A failure that reproduces 5/5 is a
// different thing from one that reproduces 1/5, and the second is the
// one that gets closed as "could not reproduce" when it is actually a
// near-tie in the model's decision -- which a prompt change can fix.The artifact every investigation produces
// Three outputs, and the third is the one that makes this worth doing.
interface Investigation {
runId: string
layer: 'retrieval' | 'generation' | 'composition' | 'action' | 'not-a-bug'
cause: string
evalCaseId: string // ⚠️ mandatory
}Every closed investigation adds a case to the suite. That is how Atlas's set went from 40 to 240 without anyone sitting down to write cases, and it is the rule that stops the same defect returning in a different shape.
// The case is written from the audit record, which means it carries the
// real question, the real context and the real failure -- not a
// reconstruction from memory three days later.
const c = await caseFromAuditRun(auditRun, { expect: { mustContain: '45', mustNotContain: '30' } })When there is no run
// The report is a screenshot, or a paraphrase from a meeting. Two moves.
//
// 1. Search the audit log by question shape and user, widening the window.
// Chapter 123's digit-normalised shape is what makes this work when
// the exact wording is lost.
//
// 2. If it is genuinely unfindable, reproduce the SITUATION rather than
// the run: same user, same question, current config. If it recurs,
// it is a live defect and the run id now exists.Atlas puts the run id in the interface footer precisely so this path is rare. It costs nothing and converts a search into a lookup.
Timing
// Atlas's median investigation, since the procedure was written down.
const MEDIAN = {
findRun: '1 min',
step1_retrieval: '4 min', // resolves 51% of cases, and stops here
step2_context: '3 min',
step3_model: '8 min', // only reached by ~30%
writeEvalCase: '6 min',
}
// Median total: 11 minutes. Before the procedure: 40-90 minutes, most of
// it spent reading the prompt.The order is the whole technique. Reading the prompt first means an hour spent in the 22% before checking the 51%.
Key points
- Debug in order of where failures are: retrieval 51%, generation 22%, composition 18%, action 9%. Reading the prompt first inverts it.
- Find the run first. The audit log stores question text in full, which is what makes a report without a run id findable. One report in eight has no run at all.
- Step 1 has four outcomes with four different owners: missing from the corpus, excluded by the filter, ranked out, or retrieved.
- A document excluded by the permission filter looks like a model failure and is correct behaviour. The fix is classification.
- Retrieval replays exactly against the stored corpus snapshot. A gold chunk at rank 47 is a reproducible finding.
- Distinguish retrieved from in-context. The token budget drops chunks between the two.
- The commonest step-2 cause is the near-miss: right document, wrong clause, superseded version ranked higher.
- Re-run generation with the stored context and config, five times. 5/5 and 1/5 are different diagnoses, and 1/5 is the one wrongly closed as unreproducible.
- Every investigation produces an eval case, written from the audit record. That is how the suite grows without anyone writing cases.
- Put the run id in the interface. Median investigation went from 40-90 minutes to 11.
Next: cost: where the money actually goes — measured, before optimising anything.