Chapter 109 · Evaluation

Regression testing a model upgrade

A new model changes everything at once, including the things your prompt was compensating for. The four-stage procedure, the prompt that has to be re-tuned, and what Atlas's upgrade actually cost.

8 min read

Chapter 54 wrote the prompt so an upgrade would not break it. Chapter 64 compared models before the system existed. This chapter is the operation itself, on a running system with users, and it is the clearest demonstration of what a suite is for.

The difference from a prompt change is scope. A prompt edit moves one thing deliberately. A model change moves tokenisation, instruction-following, refusal behaviour, tool-calling format, output length, latency and cost simultaneously — and it silently removes whatever the old prompt was compensating for.

Four stages

const UPGRADE = [
  '1. Mechanical compatibility — does it run at all?',
  '2. Unchanged-prompt comparison — the honest baseline',
  '3. Re-tune the prompt for the new model — where most of the gain is',
  '4. Shadow on real traffic — what the suite cannot see',
] as const

Skipping stage two is the common error. A team that re-tunes first reports a gain that is partly the new model and partly a better prompt, and cannot say how much of either. Stage two is unflattering on purpose.

Stage 1: compatibility

// Cheap, deterministic, and it fails fast on things that are not quality
// questions at all.
const COMPAT = [
  'tool-call schema parses',          // the format changes between families
  'structured output honours the schema',  // chapter 57
  'the full prompt fits the window',       // chapter 52
  'streaming chunks arrive as expected',   // chapter 59
  'stop sequences behave',
  'the token count per request is within budget',  // chapter 60
]

Atlas's upgrade failed two of these on the first attempt. Tool-call arguments arrived as a JSON string rather than an object, and the structured-output schema rejected an added field. Both were an hour's work and neither was visible in any quality metric — they were total failures that would have looked like a catastrophic regression if stage one had been skipped.

Stage 2: the unchanged-prompt comparison

// Same prompt, same cases, same judge, both models, one run. Chapter 108's
// paired comparison, with the config difference being the model.

Atlas, old model against new, prompt untouched:

Slice	Old	New	Change	Interval
Overall	0.847	0.861	+1.4	[-0.9, +3.8]
lookup	0.91	0.93	+2.1	[+0.2, +4.1]
policy	0.82	0.86	+3.9	[+0.8, +7.1]
comparison	0.78	0.85	+6.8	[+2.1, +11.4]
out-of-scope	0.74	0.66	-8.2	[-14.0, -2.5]
action	0.88	0.87	-0.9	[-4.2, +2.4]

Better at everything except refusing, where it is eight points worse. That shape is common enough to expect: a stronger model is more willing to attempt an answer, and the old prompt's refusal instructions were tuned against a model that needed less convincing.

On cost and latency, measured over the same run:

	Old	New
Cost per answer	$0.0141	$0.0084
p50 latency	2.4 s	1.9 s
p95 latency	6.1 s	5.4 s
Output tokens, median	210	284

The output-length increase is worth noticing. It is 35% more tokens for an answer to the same question, which users read as the assistant becoming wordier, and no quality metric captures it. It became a prompt target in stage three.

Stage 3: re-tuning

// The upgrade's real value is usually here, and it has to be measured
// against stage 2's new-model baseline rather than against the old model.
// Measuring against the old model lets a prompt fix take credit for the
// model, and the model take credit for the prompt fix.

Three changes, each measured separately:

Change	Target	Result
Strengthen the abstention instruction with two examples	out-of-scope	+11.3 [+5.8, +17.0]
Add an explicit length target	median output tokens	284 → 196
Remove a workaround for the old model's list formatting	nothing	no change, 40 fewer prompt tokens

The third is the one worth dwelling on. The old prompt carried three sentences compensating for a formatting quirk the new model does not have. Removing them changed no score and made the prompt shorter and more readable — and finding them required reading the prompt line by line and asking, of each line, which model behaviour is this for?

An upgrade is the only good occasion to do that, and a prompt that never gets that pass accumulates instructions for models that no longer run.

Final position after re-tuning, against the original old-model system:

Slice	Old	New + tuned	Change
Overall	0.847	0.892	+4.5 [+2.1, +6.9]
out-of-scope	0.74	0.78	+3.1 [-1.2, +7.5]
comparison	0.78	0.86	+7.9 [+3.0, +12.6]

Out-of-scope recovered past the old model but with an interval spanning zero, which is recorded honestly rather than claimed as a win.

Stage 4: shadow traffic

// The suite is 240 cases. Production asks 1,300 questions a week, and the
// tail is where an upgrade bites. Run the new config on real questions,
// serve the old answers, compare offline.
export async function shadow(question: string, userId: string) {
  const live = await answer(question, userId, OLD)
  void (async () => {
    const candidate = await answer(question, userId, NEW)
    await recordShadow({ question, live, candidate, divergence: diffScore(live, candidate) })
  })()
  return live
}

Two weeks of shadow on Atlas, 2,600 questions, surfaced two things the suite did not:

// 1. A tool-argument format difference on a tool the suite covers with
//    three cases and production exercises with 40 distinct argument
//    shapes. Chapter 107's action harness had 0.88 argument validity; on
//    real traffic the new model's was 0.71 for one tool.

// 2. A latency tail. p95 improved, p99 went from 9 s to 19 s on long
//    multi-step runs, because the new model took more steps before
//    terminating. Chapter 80's step limits caught them; users saw a wait.

Neither is a quality regression and both would have been a bad first week.

A 240-case suite is a sample. Shadow traffic is the distribution. The suite decides whether to attempt the upgrade; shadow decides whether to ship it.

What the upgrade cost and bought


Elapsed	11 days, including two weeks of overlapping shadow
Engineering	~3 days of one person
Eval spend	$61
Outcome	+4.5 points overall, 40% cheaper per answer, 0.5 s faster at p50

Forty percent off the inference bill is the number that justified the exercise to everyone outside the team. Three days of work and sixty dollars is the number that makes it repeatable — which matters more, because a provider ships a better model roughly twice a year and a team that cannot do this cheaply stops doing it.

Key points

A model change moves everything at once, including whatever the prompt was silently compensating for.
Four stages: compatibility, unchanged-prompt comparison, re-tuning, shadow traffic.
Run compatibility first. Atlas's upgrade failed two checks that would have looked like a catastrophic quality regression.
Stage two is unflattering on purpose. Re-tuning first makes the model and the prompt take credit for each other.
Expect refusal to regress. A stronger model is more willing to attempt, and the old abstention instructions were tuned against a weaker one: -8.2 points.
Watch output length. The new model wrote 35% more tokens for the same questions and no quality metric saw it.
Use the upgrade to read the prompt line by line and ask which model behaviour each line is for. Atlas removed three obsolete sentences.
Shadow real traffic for two weeks. It found an argument-format failure on one tool at 0.71 and a p99 latency doubling, neither visible in 240 cases.
Total: 11 days, 3 days of work, $61 of evals, +4.5 points and 40% off the inference bill.

Next: measuring groundedness and catching confident invention — the hardest property to score, and the one that matters most.