Chapter 108 · Evaluation

Regression testing a prompt change

A one-line prompt edit is a change to every answer the system will ever give. The paired comparison, the interval that stops you shipping noise, and the slice table that finds what the aggregate hides.

7 min read

Chapter 53 argued that the prompt is an artifact with change control. This chapter is what that change control actually consists of, and it is the single most-used procedure in the section — Atlas's prompt changed 41 times in two quarters.

The shape of the problem: a prompt edit is made to fix one observed failure, and it silently alters behaviour on every question in the distribution. The fix almost always works. The question is what else moved.

The paired run

// Both configurations, same cases, same seed, same judge version, in the
// same run. Comparing against a stored baseline from last week is the
// mistake that wastes an afternoon: the model, the corpus and the judge
// may all have moved since.
export async function compareConfigs(a: Config, b: Config, cases: EvalCase[]) {
  const results = await Promise.all(
    cases.map(async (c) => ({
      case: c,
      a: await run(c, a),
      b: await run(c, b),
    })),
  )
  return summarise(results)
}

// Paired, which matters statistically. Each case is its own control, so
// the variance that comes from cases being different from each other
// drops out entirely. On Atlas's 240 cases, pairing roughly halves the
// interval width against treating the two runs as independent samples.

The interval, and shipping on noise

The number that stops most bad decisions.

// Bootstrap over paired differences. 2,000 resamples is instant on 240
// cases and needs no distributional assumption, which matters because
// per-case scores are bounded and lumpy.
export function pairedInterval(diffs: number[], resamples = 2000): [number, number] {
  const means: number[] = []
  for (let i = 0; i < resamples; i++) {
    let sum = 0
    for (let j = 0; j < diffs.length; j++) sum += diffs[(Math.random() * diffs.length) | 0]
    means.push(sum / diffs.length)
  }
  means.sort((x, y) => x - y)
  return [means[Math.floor(resamples * 0.025)], means[Math.floor(resamples * 0.975)]]
}

Atlas's rule, agreed once:

Interval	Decision
Entirely above 0	Ship
Entirely below 0	Do not ship
Spans 0	No evidence. Ship on other grounds, or add cases

The third row is the common one, and naming it as a legitimate outcome is what keeps the rule enforceable. Of Atlas's 41 prompt changes, 23 produced an interval spanning zero. Most of those shipped anyway — on the grounds that they fixed a specific reported failure and did no measurable harm, which is a perfectly good reason, stated honestly.

// What the suite prints. The width is as important as the midpoint and is
// usually omitted from reports, which is how a +1.2 gets treated as a win.
// "overall +1.2 points [-2.4, +4.9] over 240 cases -- no evidence of change"

The slice table is where the finding is

An aggregate is a weighted average over strata that behave differently. The aggregate moves half a point; one stratum moves eleven.

// Sliced by the tags chapter 102 put on every case. Any slice with fewer
// than 15 cases prints its count and no interval, because an interval on
// six cases is theatre.
const SLICES = ['intent', 'sources', 'difficulty'] as const

A real example. Atlas added one sentence to the system prompt — "If the sources do not contain the answer, say so plainly" — to fix a complaint about hedging.

Slice	Before	After	Change	Interval
Overall	0.847	0.859	+1.2	[-1.1, +3.4]
lookup	0.91	0.91	+0.3	[-2.0, +2.6]
policy	0.82	0.84	+1.6	[-2.2, +5.3]
comparison	0.78	0.79	+0.9	[-4.1, +6.0]
out-of-scope	0.74	0.89	+15.1	[+8.2, +21.7]
action	0.88	0.79	-9.4	[-16.1, -2.8]

The overall number says nothing happened. Two slices moved hard in opposite directions and nearly cancelled.

The out-of-scope gain is the intended fix. The action regression is the cost: told to say plainly when sources lack an answer, the model began declining to call tools whose results would have supplied it. That is a nine-point regression on 13% of traffic, and it was invisible in the aggregate.

An aggregate is the average of the thing you fixed and the thing you broke. The slice table is where the second one lives.

The eventual prompt scoped the instruction to answering rather than to acting, and the next run showed +14.8 on out-of-scope with action flat.

Diffing responses, not just scores

// Scores say how much changed. The responses say what. The suite writes
// every case whose score moved by more than a threshold, both versions,
// side by side, to an artifact a human reads in ten minutes.
const changed = results
  .filter((r) => Math.abs(r.b.partial - r.a.partial) > 0.2)
  .map((r) => ({ id: r.case.id, question: r.case.question, before: r.a.text, after: r.b.text }))

Reading twelve diffs is where the action regression above was actually understood. The numbers said action fell nine points; the diffs showed the model writing "I don't have that information in the provided sources" on questions where a tool call would have fetched it. No metric would have said that.

What not to do

// Four habits that produce confident wrong conclusions.

// 1. Comparing against a baseline from a previous day. The judge, the
//    model and the corpus all move.

// 2. Evaluating on the cases that motivated the change. They are
//    guaranteed to improve. Report them separately and excluded.

// 3. Shipping on the aggregate without reading the slice table.

// 4. Making two prompt edits in one run. The suite can tell you the pair
//    is net positive and never which half carried it.

The fourth is the most tempting under time pressure and is the one that compounds: a prompt accumulating paired changes, each justified by a joint measurement, ends up with instructions nobody can attribute to a result.

Key points

A prompt edit changes every answer the system will give. Atlas's changed 41 times in two quarters.
Run both configurations in the same run, paired per case. Pairing roughly halves the interval width.
Bootstrap a 95% interval over paired differences. Three outcomes: ship, do not ship, and no evidence.
"No evidence" was 23 of 41 changes. Most shipped anyway, on stated grounds, which is honest and fine.
Always print the interval. A +1.2 with a [-1.1, +3.4] interval is not a win.
Read the slice table. One change moved out-of-scope +15 and action -9, netting +1.2 overall.
Suppress intervals on slices under 15 cases rather than printing meaningless ones.
Diff the responses whose scores moved. The nine-point action regression was only understandable by reading twelve of them.
Never compare against yesterday's baseline, never evaluate on the motivating cases, never change two things at once.

Next: regression testing a model upgrade — the same procedure where everything moves at once.