Chapter 107 · Evaluation

Evaluating retrieval, generation and actions separately

An end-to-end score tells you something broke and not what. Three harnesses that localise a regression in minutes, and the attribution rule that makes the end-to-end number worth keeping.

8 min read

A single end-to-end number has one job and does it badly. When it falls four points, the cause is in retrieval, in the prompt, in the model, in the tool surface or in the corpus, and the only way to find out is to change one thing at a time and re-run.

Chapter 48 made this argument for retrieval. This chapter generalises it: every layer that can fail independently gets a harness that can fail independently, and the end-to-end score becomes a check on the composition rather than the primary instrument.

Three harnesses, three units of work

Harness	Input	Output	Needs a model?	Run time
Retrieval	Question	Ranked chunk ids	No	9 s
Generation	Question + fixed context	Response	Yes	3 min
Action	Question + fixed context	Tool calls	Yes	90 s
End-to-end	Question	Response	Yes	6 min

The retrieval harness running in nine seconds with no model call is the property that changes behaviour. It is cheap enough to run on every commit that touches the retriever, which means retrieval regressions are caught by the person who wrote them rather than by a weekly suite.

The fixed context is what makes generation measurable

// The generation harness pins the context so the only variable is the
// model and the prompt. Without pinning, a generation score moves whenever
// retrieval moves, and the two are permanently confounded.
interface GenerationCase {
  id: string
  question: string
  context: Chunk[]          // frozen, committed, never regenerated
  expectations: Expectation[]
}

// Three context sets per case, because the interesting failures are not
// in the easy condition.
const CONTEXT_CONDITIONS = {
  ideal:      'exactly the chunks that answer it',
  realistic:  'what retrieval actually returned on the day the case was made',
  adversarial:'the answer plus the chapter 62 near-miss distractor',
} as const

Atlas's generation scores by condition:

Condition	Groundedness	Correct answer
ideal	0.99	97%
realistic	0.96	91%
adversarial	0.89	74%

The adversarial column is where model and prompt changes actually show up. A prompt edit moved ideal by half a point and adversarial by six — and a suite running only the ideal condition would have called that change neutral.

Freezing the context has a cost worth naming: the frozen chunks go stale as the corpus moves, and a generation case eventually tests the model against a document that no longer exists. Atlas refreshes the frozen contexts quarterly and treats a refresh as a re-baseline, with the before and after both recorded.

The action harness

// Tool selection and argument construction, scored without executing
// anything. Chapter 78 tested the tools; this tests the decision to call
// them, which is a different failure surface.
interface ActionCase {
  id: string
  question: string
  context: Chunk[]
  expect: {
    tools: string[]                    // in any order
    neverTools?: string[]
    args?: Record<string, unknown>     // exact, per tool
    confirmationRequired?: boolean
  }
}

// Nothing executes. The harness intercepts the call and scores it, which
// makes the whole thing safe to run against production tool definitions.
const calls = await planOnly(question, context, TOOL_SURFACE)

Three separable metrics, and they move independently:

Metric	What it catches	Atlas
Tool selection accuracy	Wrong tool chosen	0.94
Argument validity	Right tool, malformed args	0.88
Hazard rate	A write tool called without confirmation	0.004

Argument validity being the weakest is the common shape, and it is why chapter 69 exists. Separating it from selection is what makes it visible — an end-to-end score folds both into one failure.

The hazard rate gets its own treatment: it is reported as a count, never as a rate averaged into anything, and a single occurrence blocks a release. Four in a thousand is not a number to feel good about; it is four cases where a write would have fired unconfirmed.

Attribution: making the end-to-end number diagnostic

// When end-to-end fails, attribute it mechanically rather than by hand.
// The three harnesses already hold everything needed.
type Attribution = 'retrieval' | 'generation' | 'action' | 'composition'

function attribute(e2e: CaseResult, r: RetrievalResult, g: CaseResult): Attribution {
  if (!r.foundGoldChunk) return 'retrieval'       // it never had the answer
  if (!g.passed) return 'generation'              // it failed with ideal context too
  if (e2e.toolMismatch) return 'action'
  return 'composition'                            // each layer fine, the whole wrong
}

Atlas's failures over a quarter, attributed:

Attribution	Share
retrieval	51%
generation	22%
action	9%
composition	18%

Half of all end-to-end failures are retrieval, which is the finding that redirected a quarter of engineering effort. Before attribution existed, the team's intuition put most failures on the model, because a wrong answer reads like a model problem — the model is the thing that said the wrong sentence.

The 18% composition bucket is the one worth keeping an eye on. Every layer passed its own harness and the whole still failed, which usually means the realistic context differs from what the generation harness pinned, or a tool result arrived in a shape generation handled badly. It is the residual that justifies keeping an end-to-end run at all.

Three harnesses tell you what broke. The end-to-end run tells you whether the pieces still fit together, which is a question none of the three can answer.

Running them at different cadences

const CADENCE = {
  retrieval:  'every commit touching retrieval, the retriever config, or the corpus',
  generation: 'every prompt or model change',
  action:     'every tool-surface change',
  e2e:        'before every release, and nightly',
} as const

Four suites at four cadences is more infrastructure than one suite, and it is the arrangement people actually run — because the nine-second one is free and the six-minute one happens while you do something else. Chapter 114 is about defending those numbers as the suite grows.

Key points

An end-to-end score says something broke, not what. Every independently-failing layer needs an independently-failing harness.
The retrieval harness needs no model and runs in nine seconds, so regressions are caught by the author rather than by a weekly run.
Pin the context for generation cases. Without it, generation scores move whenever retrieval moves and the two are confounded.
Run three context conditions. A prompt change moved the ideal condition half a point and the adversarial one six.
Refresh frozen contexts quarterly and treat it as a re-baseline, recorded both sides.
Score actions without executing: tool selection, argument validity and hazard rate move independently. Argument validity is usually weakest.
Report the hazard count, never a rate. One occurrence blocks a release.
Attribute failures mechanically. Atlas's were 51% retrieval, against a team intuition that blamed the model.
Keep the end-to-end run for the composition residual — 18% of failures, invisible to every layer harness.

Next: regression testing a prompt change — the comparison that gets run most.