Chapter 112 · Evaluation

Tracking quality over time as the data moves under you

Nothing was deployed and the score fell. Five sources of drift, the provenance a stored result needs to stay comparable, and the alerts that distinguish a moving corpus from a broken system.

7 min read

Every comparison so far has been two configurations in one run. This chapter is the other axis: the same configuration, months apart, and the score is different.

That is the characteristic failure of a system built on a corpus somebody else maintains. Nothing in the repository changed and the system got worse, which no amount of code review finds and no CI pipeline reports.

Five sources of drift

Source	Cadence	Detected by
The corpus changed	Continuous	Freshness, case failures on stable cases
The model moved under a floating version	Unannounced	Refusal rate, output length
The judge moved	Unannounced	Historical re-run
The question distribution shifted	Seasonal, or a new team onboarded	Slice volumes in production
The eval set rotted	Slow	Cases asserting facts that changed

The last one is the uncomfortable one. A suite is supposed to be the fixed point, and a set of 240 cases asserting real facts about a real company contains facts that stop being true.

Provenance on every stored result

A score without its context cannot be compared with anything.

// Everything that could make two runs incomparable, recorded with the run.
// This is the artifact that makes drift analysis possible at all, and it
// is cheap to write and impossible to reconstruct later.
interface RunRecord {
  runId: string
  at: Date
  gitSha: string
  promptHash: string
  modelId: string          // the resolved version, not the alias
  judgeModelId: string
  judgeRubricHash: string
  embeddingModelId: string
  corpusSnapshotId: string // chapter 34's sync produces this
  caseSetVersion: string
  results: CaseResult[]
}

// The resolved model version is the field people omit and then need. An
// alias like "latest" resolves differently on two dates, and a run stored
// against the alias cannot be compared with anything.
const modelId = response.model   // what the provider actually served

Atlas learned this the way most teams do. Six weeks of results stored against an alias, a provider version roll, and a four-point step in the aggregate that took two days to attribute and could only be confirmed by re-running an old configuration against the new judge.

The weekly re-run

// Same cases, same config, scheduled. Its only job is to move when nothing
// should have moved.
const DRIFT_RUN = {
  schedule: 'weekly, Sunday',
  config: 'production, exactly',
  cases: 'the full set, including the held-out tenth',
  compare: 'against the previous four runs, not only the last one',
}

Comparing against four runs rather than one is what separates a trend from a blip. A single week-on-week fall is usually noise; four consecutive small falls is drift, and the sum of them is a number nobody noticed because each step was within the interval.

Atlas's cited criterion over two quarters: 0.94, 0.93, 0.93, 0.91, 0.90, 0.89, 0.88. No single step was significant. The total is six points, and the cause was a growing share of the corpus coming from a source whose documents had no stable section headings — so citations landed at document level, which chapter 62 measured as barely better than nothing.

Drift is a series of changes each too small to act on. The only instrument that sees it is one that looks at the series.

Separating corpus drift from system drift

// The diagnostic: re-run the old config against a pinned corpus snapshot.
// If the score recovers, the corpus moved. If it does not, the system did.
const [onCurrent, onSnapshot] = await Promise.all([
  run(PRODUCTION_CONFIG, { corpus: 'current' }),
  run(PRODUCTION_CONFIG, { corpus: 'snapshot-2026-07-01' }),
])

Result	Diagnosis
Both low	The system drifted — model, judge or embedding
Snapshot high, current low	The corpus drifted
Both high	The case set rotted, or the earlier number was wrong

Keeping a corpus snapshot is the prerequisite, and it is cheaper than it sounds: the chunk table with its embeddings, dumped monthly, is a few hundred megabytes for Atlas. It is the single most useful artifact for this analysis and it does not exist unless somebody decided in advance to keep it.

Case rot, and how to find it without editing cases

// A rotted case fails because the world changed, not because the system
// did. The tell is that it fails on the current corpus and passes on an
// old snapshot, AND the expectation's stated fact is no longer true.
async function auditRot(c: EvalCase): Promise<'rotted' | 'regression' | 'fine'> {
  const cur = await run(c, { corpus: 'current' })
  if (cur.passed) return 'fine'
  const old = await run(c, { corpus: 'snapshot' })
  if (!old.passed) return 'regression'          // broken in both: a real one
  return 'rotted'                               // worked then, not now
}

A rotted case is retired with its reason recorded and replaced with a fresh one asserting the current fact. It is not edited — the distinction matters, because an edited case loses the history of what it used to assert, and the retirement record is what stops the same fact being re-asserted next quarter.

Atlas retired 19 of 240 cases over two quarters, all from two sources: a price list and a contract set, both of which legitimately change.

Alerts worth having

// Four, each with an owner and a stated action. An alert without an action
// gets muted within a month.
const ALERTS = [
  { on: 'aggregate falls >3 points week-on-week', action: 'run the snapshot diagnostic' },
  { on: 'resolved model id changes', action: 'run the chapter 109 comparison before anything else' },
  { on: 'judge model id changes', action: 'recalibrate; mark the history boundary' },
  { on: 'refusal rate moves >5 points with no deploy', action: 'check for a silent model roll' },
] as const

The second and third fire on a field changing, not on a score. They are the cheapest alerts in the system and they catch the two causes that are hardest to diagnose after the fact, because by the time the score moves the old version is no longer servable.

Charting it honestly

// Two rules for the quality chart that hangs in the team's view.
//
// 1. Mark the boundaries. Every judge change, model change and case-set
//    version gets a vertical line. A continuous line across a judge change
//    is a lie about comparability.
//
// 2. Plot the slices, not just the aggregate. Chapter 108's lesson holds
//    over time as well: the aggregate is the average of the thing that
//    improved and the thing that decayed.

Key points

A system with no deploys gets worse. Five sources: the corpus, the model, the judge, the question distribution, and the case set itself.
Store full provenance with every run: resolved model id, judge id, rubric hash, embedding model, corpus snapshot, case-set version.
Record the resolved model version, never the alias. Six weeks of Atlas's results were uncomparable for this reason.
Re-run weekly against the previous four runs. Atlas's citation score fell six points in seven steps, none individually significant.
Keep monthly corpus snapshots. Re-running against one separates corpus drift from system drift in a single comparison.
Retire rotted cases with a recorded reason rather than editing them. Atlas retired 19 of 240 in two quarters, all from a price list and a contract set.
Alert on the resolved model id and judge id changing, not only on scores. They are the cheapest alerts and catch the hardest-to-diagnose causes.
Mark every judge, model and case-set boundary on the chart. A continuous line across a judge change is a lie.

Next: A/B testing agent changes with real users — the question the suite has never been able to answer.