Chapter 51 · The model layer

Choosing a model, and the criteria that matter

Six chunks and 2,520 tokens arrive at the model layer. The four criteria that decide which model reads them, and why the leaderboard is the least useful input.

10 min read

Section 4 ended with six chunks, 2,520 tokens, and held-out recall of 0.94. Section 5 is about what a model does with them.

The job is narrower than general model selection. This model is not writing code or solving puzzles. It reads six passages and writes a grounded, cited answer or declines — and the capabilities that distinguish frontier models on hard reasoning benchmarks are largely irrelevant to it.

The four criteria

1. Instruction adherence under long context

The capability that actually matters here, and the one benchmarks cover worst.

The prompt from chapter 9 contains several non-negotiable instructions: cite every claim, never answer from general knowledge, use the exact abstention phrase. A model that follows them 97% of the time and a model that follows them 99.5% of the time differ by a factor of six in how often Atlas invents something.

// Measure adherence directly. This is the single most informative test in
// the bake-off and it needs no judge.
export async function adherenceRate(model: string, cases: RetrievalCase[]) {
  const results = await Promise.all(cases.map(async (c) => {
    const chunks = await retrieveAndRerank(c.question, { scope: scopeFor(c), limit: 6 })
    const answer = await generate(c.question, chunks.results, { model })
    return {
      cited: /\[\d+\]/.test(answer.text),
      // ⚠️ Every cited number must exist. A citation of [7] against six
      // sources is the most common adherence failure and it looks correct.
      citationsValid: [...answer.text.matchAll(/\[(\d+)\]/g)]
        .every((m) => Number(m[1]) <= chunks.results.length),
      exactAbstention: !c.expectAbstention || isExactAbstention(answer.text),
    }
  }))
  return {
    cited: mean(results.map((r) => +r.cited)),
    citationsValid: mean(results.map((r) => +r.citationsValid)),
    abstention: mean(results.map((r) => +r.exactAbstention)),
  }
}

2. Groundedness

Does it answer from the sources, or from what it knows about returns policies in general?

This is the capability with the largest spread between models and the one most worth paying for. It is also measurable without a judge for a specific and useful subset:

// Counterfactual grounding: give the model sources stating something the
// world does not believe, and see whether it reports the source or the world.
const COUNTERFACTUAL = {
  question: 'How long do customers have to return damaged goods?',
  sources: ['[1] Returns Policy > Damaged goods\nDamaged goods may be ' +
            'returned within 90 days of delivery.'],
  expect: '90',          // the source
  wrongButCommon: '30',  // the prior
}

A model answering "30 days" against a source saying 90 is failing in exactly the way that matters, and the test takes one call.

3. Cost at your volume

const questionsPerDay = 800
const inputTokens = 2_520 + 600      // context + prompt
const outputTokens = 220

// ⚠️ Input dominates by roughly 14:1. A model priced attractively on output
// and poorly on input is expensive for this workload, which is the opposite
// of how models are usually compared.
const daily = questionsPerDay * (inputTokens + outputTokens)

The input-heavy shape is what makes reranking a cost lever as much as a quality one, and it is why chapter 60 treats context assembly as a budget problem.

4. Latency

// Time to first token matters more than total for a streaming interface,
// because it is what the person experiences as "is it working".
interface LatencyProfile {
  ttftMs: number        // time to first token
  tokensPerSecond: number
  totalMs: number       // ttft + output / rate
}

Against the targets — p50 ≤ 4s — a 220-token answer at 60 tokens/second is 3.7 seconds of generation alone. At 120 tokens/second it is 1.8. That difference is larger than anything else in the budget.

What does not matter

Worth saying explicitly, because these absorb the most attention.

Capability	Why it is irrelevant here
Hard reasoning benchmarks	This task is reading comprehension over six passages
Code generation	Atlas writes prose
Very long context	The context is 2,520 tokens. See chapter 52
Multilingual	Unless your corpus is
Creative writing	Actively undesirable

The model is doing reading comprehension with citation discipline. Choose for that, not for the thing the model is famous for.

Model tiers, and routing between them

The useful insight is that not every question needs the same model.

// src/config.ts — a tier per route, not one model for everything.
export const models = {
  // The router: a classification into three buckets. Tiny, fast, cheap.
  routing: 'claude-haiku-5-5',

  // Most answers: six passages, a cited paragraph. This is the workhorse
  // and where almost all the spend goes.
  generation: 'claude-sonnet-5-5',

  // Multi-hop planning and anything with an irreversible action. The
  // failure cost justifies the price. Chapters 47 and 71.
  planning: 'claude-opus-5-5',
} as const

// Routing by what the request actually needs, which is chapter 2's argument
// with model names attached.
export function modelFor(route: Route): string {
  if (route.kind === 'planning') return models.planning
  if (route.kind === 'question' && route.multiHop) return models.planning
  return models.generation
}

Measured on Atlas: routing 94% of questions to the mid tier and 6% to the top tier cost 22% of what sending everything to the top tier would, with no measurable difference in answer quality on the single-hop cases.

⚠️ Measure that claim on your own task before adopting it. The whole point of this section is that the cheaper tier is sufficient for reading comprehension over six passages, and a task with more reasoning in it will not behave the same way.

The bake-off

Same shape as the embedding bake-off, and the same discipline: your test set, your subsets, reported together.

// scripts/model-bakeoff.ts
// Retrieval is FIXED across candidates. Varying two things at once measures
// neither.
const FIXED_RETRIEVAL = RETRIEVAL_CONFIG   // from chapter 50

export async function bakeOff(candidates: string[], cases: EvalCase[]) {
  for (const model of candidates) {
    const rows = await Promise.all(cases.map(async (c) => {
      const chunks = await retrieveAndRerank(c.question, { ...FIXED_RETRIEVAL, scope: scopeFor(c) })
      const t0 = performance.now()
      const answer = await generate(c.question, chunks.results, { model })
      return { ...scoreCase(c, answer), latencyMs: performance.now() - t0,
               tokensIn: answer.usage.input, tokensOut: answer.usage.output }
    }))

    console.table([{
      model,
      correct: round(mean(rows.map((r) => +r.pass))),
      grounded: round(mean(rows.map((r) => +r.grounded))),
      citedAll: round(mean(rows.map((r) => +r.citationsValid))),
      abstainedCorrectly: round(abstentionRate(rows)),
      p50Ms: percentile(rows.map((r) => r.latencyMs), 0.5),
      costPer1k: costOf(rows, model),
    }])
  }
}

Representative results

Atlas, 120 cases, retrieval fixed:

Model	Correct	Grounded	Citations valid	Abstains	p50	Relative cost
Small	0.79	0.91	0.88	0.52	900 ms	1.0×
Mid	0.91	0.98	0.99	0.89	1,500 ms	3.2×
Large	0.93	0.99	1.00	0.94	2,600 ms	16×

Three readings.

The small model's citation validity of 0.88 disqualifies it, and that number matters more than its correctness of 0.79. Twelve percent of its answers cite a source number that does not exist — which is an answer that looks verified and is not, and it is worse than being obviously wrong.

Abstention separates the tiers more than correctness does. 0.52 against 0.89 is the difference between a system that guesses when it does not know and one that says so. That is the capability worth paying for.

Mid to large buys two points of correctness for five times the cost. Not worth it as a default; worth it for the 6% of requests where a wrong answer is expensive.

The failure mode: choosing on a sample of twelve answers

The natural approach is to run a dozen questions through two models and read the answers. The larger model's prose is noticeably better — more fluent, better organised, more confident.

That impression is real and it is measuring the wrong thing. Fluency is not correctness, and on this task the three numbers that matter are groundedness, citation validity and abstention. None of them is visible by reading good answers, because a fluent ungrounded answer reads better than a correct hedged one.

// The two cases that distinguish models, and neither is in a normal sample.

// 1. The counterfactual: sources contradicting the model's prior.
//    A weak model reports its prior and sounds confident doing it.

// 2. The unanswerable: a question the sources do not address.
//    A weak model answers anyway, fluently. This is where 0.52 comes from.
const ABSTENTION_CASES = cases.filter((c) => c.expect.abstains)

⚠️ Weight the abstention subset heavily and sample it deliberately. If your eval set is 10% abstention cases and real traffic is 25% unanswerable — which is normal early on, per chapter 49 — the aggregate understates the gap between models by more than the gap itself.

Read answers to judge tone. Measure groundedness, citation validity and abstention to judge a model. The better-sounding model is sometimes the one that invents more.

Key points

This task is reading comprehension over six passages with citation discipline. Choose for that, not for reasoning benchmarks.
Measure instruction adherence directly, including whether cited source numbers exist. An invalid citation looks verified and is not.
Test groundedness with counterfactual sources: does it report the source or its prior?
Input tokens dominate by roughly 14:1, so input pricing decides cost for this workload.
Time to first token matters more than total on a streaming interface.
Route by tier. 94% to the mid tier cost 22% of sending everything to the top tier, with no quality difference on single-hop cases.
Fix retrieval across candidates. Varying two things measures neither.
Abstention separates model tiers more than correctness does, and it is the capability worth paying for.
Do not choose by reading a dozen answers. A fluent ungrounded answer reads better than a correct hedged one.

Next: context windows — and why a larger one is not the answer to a retrieval problem.