Chapter 103 · Evaluation

Writing a good eval case

The expectations field decides whether a case is worth having. Four kinds of assertion, why a reference answer is the weakest of them, and the review that keeps a set honest.

8 min read

Chapter 102 sourced the questions. A question alone is not a case: a case is a question plus a statement of what a good answer looks like, and that second half is where sets go wrong.

The default is to write a reference answer and compare against it. It is the weakest option available, and understanding why points at the better ones.

Why a reference answer is weak

// The case everyone writes first. { question: 'what are Northline’s payment terms?', reference: 'Northline is on net 45 terms, per the 2026 supply agreement.', }

Three problems, in increasing severity.

It over-specifies. "Northline pays within 45 days of invoice" is equally correct and scores badly on anything short of a semantic comparison, which chapter 104 shows has its own failure modes.

It under-specifies. The reference says nothing about whether a citation was required, whether the answer should have mentioned the early-payment discount, or whether inventing a late fee would be acceptable. A response matching the reference and adding a fabricated sentence scores well.

It rots. When the contract changes, the reference is wrong and the suite reports a regression that is actually a corpus update. Chapter 112 is about this failure at scale, and a set of 240 reference answers is 240 facts to maintain.

A reference answer encodes one correct response. What a case needs to encode is the property that makes any response correct.

Four kinds of expectation

// Each is independently checkable, and a case usually carries two or three. type Expectation = | { kind: 'must-contain'; value: string; why: string } // a fact | { kind: 'must-not-contain'; value: string; why: string } // a known trap | { kind: 'must-cite'; documentId: string } // provenance | { kind: 'behaviour'; value: BehaviourAssertion } // shape
// The same question, written properly. { id: 'northline-terms', question: 'what are Northline’s payment terms?', tags: ['lookup', 'single-source', 'direct'], expectations: [ { kind: 'must-contain', value: '45', why: 'the term itself; any phrasing' }, { kind: 'must-cite', documentId: 'supply-agreement-northline-2026' }, { kind: 'must-not-contain', value: '30', why: 'the superseded term, still present in the 2024 agreement in the corpus' }, { kind: 'behaviour', value: { abstains: false, asksClarification: false } }, ], }

The must-not-contain is the one that earns its keep. It encodes a known near-miss — the superseded 2024 agreement sitting in the corpus, which is exactly the near-miss distractor that produces a fluent wrong answer. A case without it passes whether or not the system picks the right document.

Atlas's cases average 3.4 expectations. The ones that have caught real regressions skew heavily toward must-not-contain and must-cite.

The why field is not documentation

// Every expectation carries a reason, and it is load-bearing. { kind: 'must-contain', value: '45', why: 'the term itself; any phrasing' }

Eighteen months later someone will see this case failing and have to decide whether the case is wrong or the system is. Without the why, the cheapest resolution is to edit the case, which is the one thing chapter 102 forbade.

Atlas made why mandatory after a review found four cases that had been quietly relaxed over a quarter. None of the edits were malicious and all of them were the same reasoning: this assertion seems too strict. With a reason attached, the same person has to argue against a stated intent rather than against a bare string.

Behaviour assertions

The properties that are about the response's shape rather than its content.

interface BehaviourAssertion { abstains?: boolean // chapter 63: should it decline? asksClarification?: boolean // chapter 92: is the question ambiguous? callsTool?: string[] // which tools should fire neverCallsTool?: string[] // and which must not requiresConfirmation?: boolean // chapter 72, for write actions maxSteps?: number // chapter 80: should not take twelve }
// An out-of-scope case. There is no correct answer to compare against, // which is exactly why reference answers cannot express this class. { id: 'comp-freight-authority', question: 'can i comp the freight on this order', tags: ['out-of-scope', 'action'], expectations: [ { kind: 'behaviour', value: { abstains: true, neverCallsTool: ['apply_credit'] } }, { kind: 'must-contain', value: 'approval', why: 'it should name who can authorise, not just refuse' }, ], }

Eighteen percent of Atlas's traffic is this shape, and none of it is expressible as a reference answer. That gap is the single strongest argument for the expectation model.

What makes a case worth keeping

// The review checklist, applied when a case is proposed. A case failing any // of these is noise that will be ignored or edited away within a quarter. const CASE_REVIEW = [ 'Would a competent person agree every expectation is required?', 'Does it fail on at least one plausible wrong implementation?', 'Does it pass on at least two different correct phrasings?', 'Is it independent of a fact likely to change this quarter?', 'Does its tag set place it in exactly one stratum?', ] as const

The second is the one most proposed cases fail. A case that passes no matter what the system does measures nothing and costs a model call on every run. The cheap test is to run the case against a deliberately broken configuration — retrieval disabled, or the top chunk replaced with a random one — and confirm it fails.

// Run once when a case is added. Atlas rejected 23 of 280 proposed cases // this way, all of which passed with retrieval turned off entirely. async function discriminates(c: EvalCase): Promise<boolean> { const sabotaged = await answer(c.question, { retrieval: 'disabled' }) return !(await score(c, sabotaged)).passed }

Twenty-three cases that passed with retrieval off is the kind of finding that makes a suite trustworthy, because every one of them would otherwise have inflated the aggregate score forever.

Cases for things that should never happen

A small set of cases assert properties rather than answers, and they run against every configuration.

// These five are in every run and have never been allowed to fail. They are // the closest thing the suite has to a conventional test. const INVARIANTS = [ { id: 'no-cross-user-recall', ...}, // chapter 94 { id: 'no-unlisted-egress', ...}, // chapter 99 { id: 'citation-ids-exist', ...}, // chapter 57 { id: 'no-write-without-confirm', ...}, // chapter 72 { id: 'respects-permission-filter', ...}, // chapter 45 ]

They are kept in the eval suite rather than in CI for one reason: they need a real model call to be meaningful, and a mocked model cannot violate them. Their failure is a different signal from a score moving, and the runner reports them separately so a red invariant is never averaged into a green aggregate.

Key points

  • A case is a question plus a statement of what makes any answer correct.
  • Reference answers over-specify, under-specify and rot. 240 references is 240 facts to maintain.
  • Four expectation kinds: must-contain, must-not-contain, must-cite, and behaviour.
  • must-not-contain encodes the known near-miss. Without it a case passes whether or not the right document was found.
  • Every expectation carries a why. Without one, a failing case gets relaxed instead of investigated.
  • Behaviour assertions cover abstention, clarification, tool calls and confirmation. 18% of Atlas's traffic is only expressible this way.
  • Prove each case discriminates by running it against a sabotaged configuration. Atlas rejected 23 of 280 that passed with retrieval disabled.
  • Keep invariants in the suite, reported separately. A red invariant must never be averaged into a green aggregate.

Next: exact-match, fuzzy and semantic scoring — turning expectations into numbers without a model.