Chapter 64 · The model layer

Comparing two models on your own task

A leaderboard measures someone else's work. The procedure for comparing models on yours in an afternoon, the five metrics that decide it, and the statistics that stop you fooling yourself.

11 min read

The last chapter of section 5, and it assembles the measurement apparatus the section has been building into one procedure you will run every time a model changes.

Chapter 51 chose the first model. This is the comparison you run on every successor, and the discipline is the same as chapter 39's embedding bake-off: your corpus, your subsets, reported together.

Why the leaderboard cannot answer this

A public benchmark averages across tasks none of which is yours, on inputs that are not your inputs, scored by a method that is not your scoring.

Atlas's task is narrow and specific: read six retrieved passages, write two sentences, cite them correctly, and decline when they do not contain the answer. Nothing on a leaderboard measures that combination, and the capability that matters most here — abstention — is barely measured anywhere.

A benchmark tells you which models to test. Only your test set tells you which to ship.

The five metrics

// src/evals/compare.ts
export interface ModelScore {
  correct: number          // the answer is right
  grounded: number         // every claim is supported by a cited source
  citationsValid: number   // every cited number exists and supports the claim
  abstention: number       // declines when it should, answers when it should
  falseAbstention: number  // ⚠️ declines when it should not. Chapter 63.
  p50Ms: number
  p95Ms: number
  costPer1k: number
}

Reported as a table, never as a single composite. A weighted score hides the trade that the comparison exists to reveal.

The procedure

// scripts/compare-models.ts
export async function compareModels(models: string[], cases: EvalCase[]) {
  // ⚠️ Everything except the model is fixed: retrieval config, prompt
  // (core only, per chapter 54), temperature, context budget. Varying two
  // things measures neither.
  const fixed = {
    retrieval: RETRIEVAL_CONFIG,
    prompt: 'answer.core.md',
    temperature: 0,
    budget: CONTEXT_BUDGET,
  }

  // Retrieve ONCE and reuse. The same chunks for every model, so the
  // comparison is of generation alone — and it removes retrieval's own
  // variance from the result.
  const retrieved = new Map(await Promise.all(cases.map(async (c) =>
    [c.id, await retrieveAndRerank(c.question, { ...fixed.retrieval, scope: scopeFor(c) })] as const,
  )))

  const results: Record<string, ModelScore> = {}
  for (const model of models) {
    // n=3 and take the majority, per chapter 58. A single run on a
    // non-deterministic system is a sample of size one.
    results[model] = await scoreModel(model, cases, retrieved, { ...fixed, n: 3 })
  }
  return results
}

Retrieving once and reusing is the detail that makes the comparison trustworthy. Running retrieval per model introduces variance that has nothing to do with the models, and on a 120-case set that variance is comparable to the difference being measured.

Reading the result

Atlas, 120 cases, three candidates:

	Incumbent	Successor A	Successor B
Correct	0.91	0.94	0.92
Grounded	0.98	0.99	0.99
Citations valid	0.99	1.00	0.99
Abstention	0.97	0.96	0.98
False abstention	0.06	0.04	0.11
p50	1,500 ms	1,320 ms	980 ms
p95	3,100 ms	2,700 ms	1,900 ms
Cost	1.0×	1.4×	0.6×

B is faster and cheaper and looks competitive on the headline. Its false-abstention rate of 0.11 is disqualifying: it declines on one answerable question in nine, which is chapter 63's failure — a system that is reliable and not useful.

A is the better model and costs 40% more. Whether that is worth it is a business decision, and the table is what makes it a decision rather than a guess.

Subsets, again

const SUBSETS = ['exact_token', 'negation', 'multi_chunk', 'multi_hop',
                 'conflict', 'abstention_expected', 'follow_up'] as const

Subset	n	Incumbent	A	B
Negation	8	0.75	0.88	0.75
Conflict	6	0.83	1.00	0.83
Multi-hop	12	0.83	0.92	0.83
Abstention expected	22	0.95	0.91	1.00
Everything else	72	0.94	0.95	0.94

The aggregate difference between the incumbent and A — three points — is entirely in three small subsets. On the 72 ordinary cases they are indistinguishable.

That changes the decision. If negation, conflicts and multi-hop are 22% of real traffic, A is clearly worth 40%. If they are 3%, it is not, and the right answer is routing: the incumbent by default, A for the classified-hard cases.

// Which is a measurable choice rather than a compromise.
export function modelFor(question: string, route: Route): string {
  if (route.multiHop) return MODELS.a
  if (detectNegation(question) || route.conflictLikely) return MODELS.a
  return MODELS.incumbent
}
// Measured: 0.94 correctness at 1.09x cost, against 0.94 at 1.4x for A
// everywhere. The routing captures the gain for a quarter of the premium.

Not fooling yourself

With 120 cases, a three-point difference is 3.6 questions. Two checks before believing it.

// 1. Paired bootstrap. The cases are the same for both models, so pair them
//    — a paired test is much more sensitive than comparing two averages.
export function pairedBootstrap(a: Row[], b: Row[], runs = 10_000) {
  const diffs = a.map((row, i) => (+b[i].correct) - (+row.correct))
  const observed = mean(diffs)

  let atLeastAsExtreme = 0
  for (let i = 0; i < runs; i++) {
    const resampled = diffs.map(() => diffs[Math.floor(Math.random() * diffs.length)])
    if (Math.abs(mean(resampled)) >= Math.abs(observed)) atLeastAsExtreme++
  }
  return { observed, p: atLeastAsExtreme / runs }
}
// Atlas, incumbent vs A: observed +0.030, p = 0.041. Real, and narrowly.
// Incumbent vs B: observed +0.008, p = 0.63. Noise.

// 2. Look at the cases that CHANGED, not the average. Chapter 48's rule.
const { improved, regressed } = compare(incumbentRows, aRows)
// improved (9): neg-002 neg-005 cfl-001 cfl-004 mh-003 mh-007 ...
// regressed (3): abs-011 abs-014 abs-019   <- all abstention cases

The regression list being entirely abstention cases is the finding that the aggregate could not produce, and it is consistent with A's slightly lower abstention score. Three regressions in one subset is a pattern, not noise, and it is what to check before shipping.

Judged metrics need calibrating

grounded and citationsValid at the entailment level are judged by a model, and chapter 106 covers this properly. The short version for a comparison:

// ⚠️ Never judge a model's output with itself. A model rates its own
// phrasing favourably, and the bias is large enough to reverse a close
// comparison.
const JUDGE = 'a model from a different family than any candidate'

// And calibrate the judge once against human ratings before trusting it.
// 40 answers, rated by two people, compared to the judge.
// Atlas: judge-human agreement 0.89 on grounded, 0.94 on citations valid.
// ⚠️ Below ~0.85, the judge is adding noise rather than signal.

When to run it

Trigger	Scope
A new model is available	Full comparison
The incumbent is deprecated	Full, with a deadline
Quarterly review	Incumbent against best successor
Cost pressure	Cheaper candidates only, against the same bar
A quality complaint	Not this — diagnose first

The last row matters. A quality complaint is a retrieval problem four times in five, and running a model comparison in response is how a week gets spent on the component that was working.

Section 5 is done

	Section start	End
Correct	—	0.91
Grounded	—	0.98
Citations valid	—	0.99
Correct abstention	—	0.97
False abstention	—	0.06
p50 total	1,774 ms	1,810 ms
p50 time to first token	—	640 ms

Section 6 is tools: Atlas stops answering and starts doing. Everything in this section still applies, and the stakes change — a wrong answer is embarrassing, and a wrong action is expensive.

The failure mode: the comparison that was run once

The comparison is run, a model is chosen, and the script is never run again. Eighteen months later the incumbent is three generations old, as chapter 54 described — but there is a second, worse version of this.

The comparison is re-run and the test set has not changed. The 120 cases were harvested when the system launched, and the question distribution has moved: people have learned what Atlas does well and now ask different things. The comparison is scored against questions nobody asks any more, and it picks a model optimised for a distribution that no longer exists.

// The guard: assert the test set still reflects production before trusting
// a comparison run against it.
export async function assertTestSetCurrent() {
  const composition = await productionQuestionComposition({ days: 30 })
  const testComposition = testSetComposition()

  for (const subset of SUBSETS) {
    const drift = Math.abs(composition[subset] - testComposition[subset])
    if (drift > 0.10) {
      throw new Error(
        `test set drift on '${subset}': production ${pct(composition[subset])}, ` +
        `test set ${pct(testComposition[subset])}. Re-harvest before comparing models.`,
      )
    }
  }
}

Running that check on Atlas at six months found follow_up at 31% in production against 9% in the test set — chapter 49's harvesting finding, reappearing as a reason not to trust a model comparison.

A model comparison is only as current as the test set it runs against. Check the distribution before you check the models.

Key points

A benchmark tells you which models to test; your test set tells you which to ship.
Five metrics in a table, never a composite. A weighted score hides the trade.
Fix everything but the model, and retrieve once and reuse. Per-model retrieval adds variance comparable to the effect.
Run n=3 and take the majority. A single run is a sample of size one.
A disqualifying number can sit behind a competitive headline: 0.11 false abstention is reliable and useless.
Read subsets. Three points of aggregate difference was entirely in three small subsets, which turns the decision into a routing question.
Routing captured the gain for a quarter of the premium: 0.94 at 1.09× instead of 1.4×.
Use a paired bootstrap, and look at which cases changed. Three regressions in one subset is a pattern.
Never judge with a candidate model. Calibrate the judge against human ratings; below ~0.85 agreement it is noise.
Do not run a model comparison in response to a quality complaint. Diagnose first.
Check the test set still matches production before trusting any comparison.

Next, and the start of section 6: from answering to doing — and the failure cost changing from embarrassing to expensive.