Chapter 102 · Evaluation
Building the first eval set from real questions
Where the questions come from, why invented ones mislead, and how to stratify a set of forty so it represents the traffic instead of the author's imagination.
7 min read
Chapter 49 built a set for retrieval alone: questions paired with the passages that should come back. This chapter builds the end-to-end set, and most of the sourcing argument carries over. The part that does not is what makes it worth its own chapter — an end-to-end case has to specify what a good answer is, and that is a much softer target than a passage id.
Invented questions are a different distribution
The fastest way to get forty cases is to write forty questions. It produces a set that measures something other than the system.
Atlas's team wrote 40 questions before launch. After three months of real traffic, the two distributions compared:
| Property | Written by the team | Real traffic |
|---|---|---|
| Median length | 11 words | 6 words |
| Contains a typo or abbreviation | 0% | 31% |
| Fully specified | 95% | 42% |
| Answerable from one source | 80% | 64% |
| Should be refused | 5% | 18% |
Every row is the same bias. People writing test questions write the question they would ask if they were being careful, about a thing they know the system can do. Real users type "northline po 4471 status" and "can i comp the freight on this" — the first underspecified, the second something the assistant must decline.
The last row is the expensive one. A set with 5% refusal cases cannot detect a change that makes the system worse at refusing, and chapter 111 is about why that is the regression most likely to go unnoticed.
Sourcing from traffic you already have
-- Stratified sample rather than the top N. The top N is dominated by one
-- or two question shapes, and a set of forty near-duplicates measures one
-- thing forty times.
with scored as (
select id, question, intent_tag, source_tag,
row_number() over (partition by intent_tag order by random()) as rn
from question_log
where asked_at > now() - interval '90 days'
and not is_test
)
select * from scored where rn <= 6;Before there is traffic, the sources in order of value:
| Source | Yields | Note |
|---|---|---|
| The channel the assistant replaces | The real distribution | A support inbox, a shared mailbox, a chat channel |
| Search logs on the intranet | Underspecified queries | Closest thing to real phrasing |
| Interviews with five users | Intent, badly phrased | Ask what they asked last week, not what they would ask |
| Written by the team | Coverage of known capability | Useful as a supplement, misleading as a base |
Atlas's first 40 came from a shared mailbox: 18 months of questions reps had emailed to the operations team, which is exactly the traffic the assistant was built to absorb.
Stratifying forty cases
A set has to be a sample of something. Without an explicit frame, it is a sample of what the author happened to think of.
// The frame. Each axis is a thing a change could plausibly break on its own,
// and each cell needs enough cases that a cell-level number means anything.
interface Strata {
intent: 'lookup' | 'policy' | 'comparison' | 'action' | 'out-of-scope'
sources: 1 | 2 | 'many'
difficulty: 'direct' | 'requires-inference' | 'ambiguous'
}Atlas's 40, by intent:
| Intent | Cases | Share | Traffic share |
|---|---|---|---|
| lookup | 14 | 35% | 38% |
| policy | 10 | 25% | 22% |
| comparison | 5 | 12% | 9% |
| action | 4 | 10% | 13% |
| out-of-scope | 7 | 18% | 18% |
Matching the traffic share is the default and not the rule. Two deliberate departures are worth naming:
// 1. Over-sample the dangerous. Out-of-scope is 18% of traffic and could
// justifiably be 30% of the set, because a failure there is a wrong
// commitment to a customer rather than a mildly worse answer.
// 2. Over-sample the fragile. Multi-hop questions are 4% of traffic and
// break on almost every retrieval change, so they earn more cases than
// their share. Chapter 47 explains why they are the fragile class.Both are defensible and both must be written down, because a set weighted differently from the traffic produces an aggregate score that is not the production quality — and somebody will quote it as if it were.
Deduplicating by what the case tests
// Two questions that differ in wording and exercise the same path are one
// case. The cheap check is on the retrieved set rather than on the text:
// if two questions retrieve the same top-5 and expect the same answer
// shape, the second adds nothing.
async function isDuplicate(a: EvalCase, b: EvalCase): Promise<boolean> {
const [ra, rb] = await Promise.all([retrieve(a.question), retrieve(b.question)])
const overlap = jaccard(ra.slice(0, 5).map((c) => c.id), rb.slice(0, 5).map((c) => c.id))
return overlap > 0.8 && a.expectations[0].kind === b.expectations[0].kind
}Atlas's first pass collected 64 candidates and deduplicated to 40. The 24 removed were almost all lookup — the most common intent, and the one where wording varies most while the path does not.
Freezing the set, and the rule about using it
// The set is version-controlled, reviewed like code, and append-only in
// practice. Editing a case because the system now fails it is the single
// most destructive thing anyone can do to a suite, and it happens by
// accident when the set lives in a spreadsheet.One rule enforces the rest: a case may be changed only when it was wrong when written. A case the system started failing is a finding, and the PR that changes it has to say which of those two it is.
Atlas keeps a held-out tenth that is never looked at during tuning. Over a quarter, tuned-set scores improved 11 points and held-out scores improved 6 — the gap is the overfitting the hold-out exists to measure, and it is why chapter 50 insisted on one for retrieval too.
An eval set is a frozen description of what good looks like. The moment it can be edited to match what the system does, it describes the system instead.
Growing it from production
// Every case after the first forty comes from something that went wrong.
// Three triggers, all of which already exist for other reasons.
const GROWTH_TRIGGERS = {
thumbsDown: 'chapter 98 already captures these',
abstention: 'a refusal the user then rephrased into an answerable question',
judgeDisagreement: 'the judge and a reviewer disagreed — chapter 106',
}Atlas went 40 to 240 over two quarters entirely this way. Nobody sat down to write cases after the first batch, which is the property that makes the practice survive: the set grows as a side effect of handling complaints, and a complaint that becomes a case cannot silently return.
Key points
- Invented questions are a different distribution: longer, fully specified, typo-free, and short on the refusal cases that matter most.
- Source from the channel the assistant replaces. Atlas's first 40 came from 18 months of a shared mailbox.
- Stratify explicitly on intent, source count and difficulty. Without a frame, the set samples the author's imagination.
- Match traffic share by default, and depart from it deliberately for the dangerous and the fragile. Write the departures down.
- Deduplicate on the path exercised, not the wording. 64 candidates became 40, and the removals were almost all the commonest intent.
- A case may be changed only if it was wrong when written. Anything else turns the set into a description of the system.
- Hold out a tenth. Atlas's tuned scores rose 11 points and held-out 6; the gap is the overfitting.
- Grow from complaints, abstentions and judge disagreements. The set reaches 240 without anyone writing cases.
Next: writing a good eval case — what goes in the expectations field.