Chapter 105 · Evaluation

LLM-as-judge, and its specific traps

A model scoring another model's work is the only practical way to measure the properties that matter. Six biases with measured sizes, and the prompt structure that survives them.

8 min read

Chapter 104 handled 71% of assertions mechanically. The remaining 29% are the properties that actually distinguish a good answer from a plausible one — whether it is grounded, whether it answered the question asked, whether it should have refused — and there is no mechanical scorer for any of them.

A model is the only practical option. It is also a measurement instrument with known, measurable biases, and the gap between teams who use judges well and badly is entirely in whether they measured them.

Six biases, with sizes

All measured on Atlas's suite, against a 200-case set with human ratings.

Bias	What it is	Measured size
Position	Prefers whichever response is shown first	+12 points to position A
Verbosity	Prefers the longer answer	+9 points per 2× length
Self-preference	Prefers output from its own family	+7 points
Sycophancy	Agrees with a stated expectation in the prompt	+21 points
Leniency	Scores 4/5 when unsure, rarely 1 or 2	68% of scores in one band
Format	Prefers structured output regardless of content	+5 points for bullets

The sycophancy number is the one that invalidates most judge implementations, and it comes from the most natural possible prompt: "The expected answer is X. Does the response match?" A judge told what the answer should be finds it in 21 points more responses than one asked to decide independently.

// The prompt that produces that bias, and it is the obvious one to write.
const BAD = `
Expected answer: {{expected}}
Response: {{response}}
Does the response match the expected answer? Score 1-5.
`

// The fix is to make the judge extract before it compares, so the comparison
// happens against something it committed to without seeing the target.
const BETTER = `
Read the RESPONSE and answer these in order.

1. What specific claim does the response make about the question asked?
   Quote it.
2. Does the response cite a source for that claim? Which?
3. Only now: does the claim in (1) state that {{property}}?

Output JSON: {"claim": "...", "citation": "...", "satisfies": true|false}
`

The reordering cut the measured sycophancy from 21 points to 4.

Position and verbosity, and the cheap fixes

// Position bias in a pairwise comparison is fixed by running both orders
// and discarding disagreements. It doubles the cost of the comparison and
// it is not optional.
export async function compare(a: string, b: string, question: string) {
  const [ab, ba] = await Promise.all([
    judgePair(question, a, b),
    judgePair(question, b, a),
  ])
  if (ab.winner === 'first' && ba.winner === 'second') return 'a'
  if (ab.winner === 'second' && ba.winner === 'first') return 'b'
  return 'tie'            // the judge contradicted itself: no signal
}

Atlas's judge contradicts itself on 17% of pairs. Those are reported as ties rather than resolved, because a coin flip dressed as a measurement is worse than an admitted absence of one.

// Verbosity bias has no clean fix, so it is controlled rather than removed:
// score against a rubric with explicit criteria instead of asking which is
// better. A rubric item is satisfied or not, and length cannot satisfy one.

That reframing — from preference to rubric — removes verbosity, format and most of self-preference at once, because none of them can make a specific criterion true. It is the single highest-value change available to a judge implementation.

The rubric

// Atlas's generation rubric. Five binary criteria, each independently
// checkable, each with a stated failure example in the prompt.
const RUBRIC = [
  { id: 'answers-the-question', text: 'The response addresses what was asked, rather than an adjacent question.' },
  { id: 'grounded',             text: 'Every factual claim appears in the provided sources.' },
  { id: 'cited',                text: 'Each factual claim carries a citation to a source that states it.' },
  { id: 'no-invention',         text: 'The response adds no specific figure, date or name absent from the sources.' },
  { id: 'appropriate-scope',    text: 'The response neither refuses something answerable nor answers something it should decline.' },
] as const

Binary is deliberate. A 1-to-5 scale invites the leniency bias — 68% of Atlas's scores landed on 4 before the switch — and the resulting distribution has almost no resolving power. Five binary criteria give a 0-to-5 score with the same range and none of the bunching.

// And the judge sees one criterion at a time. Asking for five judgements in
// one call let the first answer anchor the rest: agreement with human
// raters was 0.71 batched against 0.88 one at a time.
const results = await Promise.all(RUBRIC.map((c) => judgeCriterion(c, response, sources)))

Five calls instead of one is five times the cost of the judged 29%, which is about $0.60 of a $2.10 run. Seventeen points of agreement for sixty cents.

Which model judges

// Three rules, each of which cost something to learn.
const JUDGE_CONFIG = {
  // 1. A different family from the one being judged. Self-preference is
  //    +7 points, and it is invisible because it moves every score.
  model: 'a different provider from the generator',

  // 2. A fresh context. Chapter 83 measured a model reviewing its own work
  //    in its own context agreeing with itself: 0.62 against human raters,
  //    rising to 0.91 with a clean context and a different family.
  context: 'clean',

  // 3. Pinned, and treated as infrastructure. A judge that upgrades under
  //    you re-baselines every historical number silently.
  version: 'pinned, changed only with a re-calibration run',
}

The third is the one teams discover painfully. Atlas's provider rolled a judge model forward and the suite's aggregate moved 4 points overnight with no change to the system. Everything measured before and after that date is on different scales, and the only recovery is re-running the old configuration — which is why chapter 112 stores the judge version with every result.

A judge is part of the measurement apparatus, not part of the product. It gets pinned, versioned and re-calibrated exactly like a reference weight.

What a judge should never be asked

// Three questions that produce confident numbers with no information.
const NEVER_ASK = [
  'Is this answer good?',             // no criterion; measures tone
  'How helpful is this, 1-10?',       // leniency + no resolving power
  'Would a user be satisfied?',       // the judge has never met a user
]

The third is the one that keeps appearing because it sounds like the thing that matters. It is unanswerable from the response alone, and the judge answers it anyway, with a number, every time. Chapter 113 is where that question actually gets answered.

Key points

The 29% of assertions that need judgement are the ones that distinguish a good answer from a plausible one. A model is the only practical scorer.
Six measured biases: position +12, verbosity +9, self-preference +7, sycophancy +21, leniency, format +5.
Showing the judge the expected answer costs 21 points of sycophancy. Make it extract the claim first, then compare: down to 4.
Run pairwise comparisons in both orders and report contradictions as ties. Atlas's judge contradicts itself on 17% of pairs.
Replace preference with a rubric. A criterion cannot be satisfied by being longer, prettier or from the same model family.
Use binary criteria. A 1-5 scale put 68% of scores in one band.
Judge one criterion per call. Batched agreement was 0.71; one at a time, 0.88, for about sixty cents a run.
Pin the judge and record its version. A silent judge upgrade moved Atlas's aggregate four points with no system change.
Never ask whether an answer is good, helpful, or satisfying. Those produce confident numbers with no information.

Next: calibrating a judge against human ratings — the step that makes any of these numbers mean something.