Chapter 101 · Evaluation

Why you cannot ship this without evals

Every component so far was tuned against a number produced for that question alone. This is the argument for a standing harness, and the specific failures that happen without one.

7 min read

A hundred chapters of numbers: recall@12 at 0.94, groundedness at 0.96, a seven-point memory margin, 1.3% of citations unsupported. Every one of them was produced to answer the question in front of it and then discarded.

That is how a system gets built and it is not how one gets maintained. An ad hoc measurement answers a question once; an eval suite answers it every time anything changes — and in a system with a prompt, a retriever, a model version, a tool surface and a corpus that moves underneath all of them, something changes most days.

What goes wrong without one

Four failures, all of which happen to competent teams, none of which a conventional test suite catches.

// 1. The silent regression. A prompt edit to fix one complaint degrades a
//    class of questions nobody tried. It ships, and nothing fails.

// 2. The unfalsifiable improvement. "This feels better." Shipped on a demo
//    of four questions, all of which were used to write the change.

// 3. The model upgrade nobody dares do. With no suite, swapping a model is
//    an unbounded risk, so the system stays on a model that is eighteen
//    months old and three times the price.

// 4. Drift. Nothing changed in the repo and quality fell, because the corpus
//    changed. Chapter 112 is entirely about this one.

The third is the expensive one in practice. Atlas's provider released a model that was 40% cheaper and measurably stronger. Without a suite the upgrade is a judgement call made nervously; with one it is chapter 109, a four-hour exercise ending in a table.

An eval suite is not a test suite

They look similar and they answer different questions.

	Test suite	Eval suite
Output	Pass or fail	A score, and a distribution
On a failure	The code is wrong	Maybe nothing is wrong
Determinism	Required	Not available
What it protects	Behaviour you specified	Quality you can only observe
Run on	Every commit	Every change to a model-facing surface

The second row is the one that trips teams used to conventional testing. A single eval case going from pass to fail is often noise. The suite's unit of meaning is the aggregate over a set, with a comparison and an interval — which is why chapter 114 spends its time on making a 200-case run cheap rather than on making a single case precise.

Both exist. The tool tests from chapter 78 are deterministic and belong in CI on every commit. The eval suite is a different artifact with a different cadence.

Four questions a suite has to answer

// Everything in this section serves one of these.
const QUESTIONS = [
  'Is it right?',          // groundedness, correctness, citations
  'Is it right about the right things?',   // retrieval, before generation
  'Does it know when to stop?',            // refusal, abstention, confirmation
  'Did this change make it better or worse?',  // the comparison
] as const

The fourth is the only one that justifies the infrastructure. Absolute quality numbers are interesting once; the comparison is what gets used weekly, and it is what the suite has to be designed around. A harness that produces a beautiful absolute score and cannot cheaply compare two configurations has solved the easy half.

The yellow box is the honest outcome most of the time, and a suite that never produces it is a suite whose intervals are wrong.

What it costs, stated plainly

The reason teams skip this is cost, so the number is worth putting up front. Atlas's suite:


Cases	240
Full run, wall clock	6 minutes
Full run, model spend	$2.10
Runs per week	~25
Monthly	~$210
Build cost	about three weeks of one person, spread over a quarter

Two hundred and ten dollars a month to be able to change anything with evidence. Against that: one silent regression found before release, in a system answering 1,300 questions a week to people making commitments to customers.

The build cost is the real one, and it is why the next chapter starts with 40 cases rather than 240. A suite that exists and is small beats a suite that is being designed.

Starting is the hard part, and it is smaller than it looks

// The minimum viable suite. It is worth having on day one.
interface EvalCase {
  id: string
  question: string
  expectations: Expectation[]     // chapter 103 makes these precise
  tags: string[]                  // for slicing: by source, by difficulty
}

Forty real questions with checkable expectations, run by a script, producing a number. No judge, no interval, no CI integration. It catches the catastrophic class immediately — a retrieval change that breaks a whole source, a prompt edit that drops citations — and it is the thing the rest of this section grows.

The suite you will not finish designing is worth less than the forty cases you ran this afternoon. Every property in this section was added to Atlas's suite after it already existed and was already finding things.

What this section builds

Chapter	Adds
102–103	The case set, and what makes a case worth having
104–106	Scoring: mechanical, then a judge, then the judge calibrated
107	Separating retrieval, generation and action so a score localises
108–109	The two comparisons that get run most: a prompt change, a model change
110–111	The two properties hardest to score: groundedness and refusal
112–113	Production: drift over time, and real users
114	Making it fast enough to be run, which decides whether any of it matters

Key points

Every number in this series so far was produced once for one question. A suite is what makes them repeatable.
Four failures without one: silent regressions, unfalsifiable improvements, model upgrades nobody dares attempt, and drift from a moving corpus.
An eval suite is not a test suite. It scores rather than passes, its unit of meaning is the aggregate, and a single case flipping is usually noise.
Keep the deterministic tool tests in CI as well. They are a different artifact at a different cadence.
The question that justifies the infrastructure is the comparison, not the absolute score. Design for cheap A-against-B.
"No evidence either way" is the honest outcome most of the time. A suite that never says it has broken intervals.
Atlas: 240 cases, 6 minutes, $2.10 a run, about $210 a month. Three weeks to build, spread over a quarter.
Start with 40 real questions and a script. The catastrophic class is caught immediately.

Next: building the first eval set from real questions — where those forty come from.