Chapter 106 · Evaluation

Calibrating a judge against human ratings

An uncalibrated judge produces numbers with unknown meaning. The 100-case exercise, the agreement statistic that is not accuracy, and what to do when humans disagree with each other.

7 min read

Chapter 105 built a judge that avoids six known biases. That makes it better; it does not make it correct. A judge is an instrument, and an instrument that has never been compared against a reference produces readings of unknown meaning.

This chapter is referenced from chapter 62 and from several places in section 8, because every one of those numbers depends on it.

The exercise

It is a day of work and it is the difference between a suite that informs decisions and one that generates numbers.

// 1. Sample 100 responses, stratified across the same strata as the case
//    set. Not the 100 most interesting — the interesting ones are where
//    agreement is lowest, which biases the estimate downward.
// 2. Two humans rate each, independently, against the same rubric the
//    judge uses. Same binary criteria, same wording.
// 3. Resolve human disagreements by discussion, and record how many there
//    were. That number matters as much as the judge's.
// 4. Run the judge on the same 100.
// 5. Compute agreement per criterion, not overall.

Step three is the step that is usually skipped, and it produces the most useful number in the exercise.

Humans disagree, and that is the ceiling

Atlas's two raters, before discussion:

Criterion	Human-to-human agreement
grounded	0.94
no-invention	0.92
cited	0.97
answers-the-question	0.84
appropriate-scope	0.79

A judge cannot usefully exceed the agreement humans reach with each other. On appropriate-scope — whether the system should have refused — two careful people agreed 79% of the time, which means a judge scoring 0.79 against the consensus is performing at the ceiling, and chasing 0.90 there is chasing noise.

Measure the humans first. Without the ceiling, a judge's agreement score has no scale, and teams spend weeks improving a prompt past the point where the underlying question has a definite answer.

The low score on appropriate-scope is itself a finding: it means the refusal policy is underspecified. Chapter 111 is largely about turning that into something two people can agree on.

The statistic is not accuracy

// Raw agreement is misleading when a criterion is mostly true. If 92% of
// responses are grounded, a judge that always says "grounded" scores 0.92.
function cohensKappa(a: boolean[], b: boolean[]): number {
  const n = a.length
  const observed = a.filter((v, i) => v === b[i]).length / n
  const pa = a.filter(Boolean).length / n
  const pb = b.filter(Boolean).length / n
  const expected = pa * pb + (1 - pa) * (1 - pb)
  return (observed - expected) / (1 - expected)
}

Atlas's judge, both statistics:

Criterion	Raw agreement	Kappa	Human ceiling (kappa)
grounded	0.93	0.71	0.82
no-invention	0.95	0.48	0.79
cited	0.97	0.89	0.93
answers-the-question	0.88	0.69	0.74
appropriate-scope	0.81	0.61	0.66

The no-invention row is the whole argument for kappa. Raw agreement of 0.95 looks excellent and the kappa of 0.48 says the judge is barely better than always answering "no invention" — which it nearly always is, because invention is rare. A suite trusting the 0.95 would have been blind to exactly the failure it most needed to catch.

The fix was not a better prompt. It was giving the judge the sources and asking it to list each specific figure, date and name in the response and mark whether each appears in a source — turning a yes/no judgement into an enumeration. Kappa went from 0.48 to 0.76. Chapter 110 builds that out.

Where a judge is wrong, not just how often

// Agreement is a summary. The confusion matrix is the actionable artifact,
// because the two error directions have completely different consequences.
interface JudgeCalibration {
  criterion: string
  falsePass: number    // judge said pass, humans said fail -- the dangerous one
  falseFail: number    // judge said fail, humans said pass -- noise
  kappa: number
  humanCeiling: number
}

Atlas's grounded criterion: 4% false-pass, 3% false-fail. Roughly balanced, which is fine for tracking a trend.

Atlas's appropriate-scope: 11% false-pass, 8% false-fail. The 11% means the suite systematically under-reports refusal failures, and that is recorded alongside the metric rather than corrected away — a known bias with a size is usable; an unknown one is not.

// Published with the metric, every time. The suite's report literally
// prints this line under the refusal score.
// "appropriate-scope: 0.86 (judge under-reports failures by ~11%; human
//  ceiling kappa 0.66, measured 2026-09-14 on 100 cases)"

Re-calibration, and when it is required

const RECALIBRATE_WHEN = [
  'the judge model or version changes',          // chapter 105's pinning
  'the rubric wording changes',                  // any edit, however small
  'the case set gains a new stratum',
  'quarterly, on a fresh 50-case sample',        // drift in the corpus itself
] as const

The quarterly run on a fresh sample is the one that catches slow drift. Atlas's cited kappa fell from 0.89 to 0.74 over two quarters with no change to the judge, because the corpus had grown and citations now pointed at a wider range of document types, some of which the judge handled poorly. Nothing in the suite would have surfaced that.

Each re-calibration is cheap: 50 cases, two raters, about three hours. Against a suite that drives weekly shipping decisions, that is the right cadence.

What to do with a judge that cannot be calibrated

Sometimes the answer is that the criterion is not measurable this way.

// Atlas tried and abandoned two criteria.
//
// 'tone-appropriate' -- human-to-human kappa 0.41. Two careful people
//   could not agree, so there was nothing for the judge to agree with.
//   Removed from the rubric rather than measured badly.
//
// 'would-satisfy-the-user' -- chapter 105 already named this unanswerable
//   from the response. Human kappa 0.52. It moved to chapter 113, where
//   real users answer it.

Removing a criterion feels like losing coverage and is the opposite. A rubric item with a 0.41 human ceiling contributes noise to every aggregate it enters, and a suite's credibility rests on every number in it meaning something.

Key points

An uncalibrated judge produces readings of unknown meaning. Calibration is a day of work.
Measure human-to-human agreement first. It is the ceiling, and without it a judge's score has no scale.
Atlas's humans agreed 0.79 on whether a refusal was appropriate. That low number is itself a finding about an underspecified policy.
Use kappa, not raw agreement. On a rare property, raw 0.95 hid a kappa of 0.48 — a judge barely better than always saying yes.
Fix a weak criterion by turning judgement into enumeration. Listing each figure and checking it against sources took kappa from 0.48 to 0.76.
Record the error directions. Atlas's refusal judge under-reports failures by 11%, and that is published with the metric.
Re-calibrate on a model change, a rubric edit, a new stratum, and quarterly on a fresh sample. A quarterly run caught a 15-point fall with no judge change.
Remove a criterion humans cannot agree on. A 0.41 ceiling contributes noise to every aggregate it touches.

Next: evaluating retrieval, generation and actions separately — so a falling score says which part broke.