Chapter 49 · Retrieval

Building a retrieval test set from real questions

The thirty-four written in advance measure anticipated language. Here is how to harvest real questions, label them without it becoming a job, and keep the set representative as it grows.

11 min read

The previous chapter ended on a gap: 0.99 on the deliberate test set, 0.94 on real questions, and widening. This chapter closes it.

The thirty-four cases from chapter 7 were the right way to start — they existed before the system and they shaped the slice. They are not a sample of what people ask, because they were written by people imagining what people would ask.

What real questions look like

Atlas's first month, 412 questions from 19 of 24 reps. The differences from the written set are systematic.

Property	Written set	Real questions
Median length	11 words	6 words
Contain a pronoun	0%	23%
Follow-up to a previous turn	0%	31%
Contain internal jargon	18%	64%
Contain a typo	0%	9%
Are a yes/no confirmation	6%	27%

The last row is the one with teeth. "Is Henderson still Net 30?" is the shape chapter 46 showed steering retrieval toward the assumption in the question, and it is more than a quarter of real traffic while being almost absent from the written set.

Harvesting, without labelling everything

412 questions is too many to label by hand and far more than you need. Sample deliberately rather than taking the first fifty.

-- Four strata, each catching a different failure. ~40 cases total.
-- 1. Retrieved nothing. The clearest signal.
select 'no_results' as stratum, id, question, asked_at
  from question_log
 where result_count = 0 and asked_at > now() - interval '30 days'
 order by random() limit 10

union all
-- 2. Marked unhelpful by the person who asked.
select 'unhelpful', id, question, asked_at
  from question_log
 where feedback = 'unhelpful' and asked_at > now() - interval '30 days'
 order by random() limit 10

union all
-- 3. Asked repeatedly by different people. High value, probably working.
select 'frequent', id, question, asked_at
  from (
    select id, question, asked_at,
           count(*) over (partition by normalise(question)) as asks
      from question_log where asked_at > now() - interval '30 days'
  ) q where asks >= 4 order by random() limit 10

union all
-- 4. A plain random sample. ⚠️ Without this the set is all failures, and a
--    set of only failures cannot detect a regression on what works.
select 'random', id, question, asked_at
  from question_log
 where asked_at > now() - interval '30 days'
 order by random() limit 10;

The fourth stratum is the one people skip. A test set harvested only from failures measures your worst cases and goes blind to regressions on the 90% that work.

Labelling, which is the actual cost

A case needs the supporting chunks identified, and that is a human judgement. The trick is to make the human confirm rather than search.

// scripts/label-case.ts — propose, then confirm. Labelling a case takes
// about ninety seconds this way and about ten minutes without it.
export async function proposeLabels(question: string) {
  // Deliberately deep and unfiltered: as an administrator, 30 candidates,
  // so the answer is almost certainly in the list even when production
  // retrieval failed to surface it.
  const candidates = await retrieveAndRerank(question, { scope: ADMIN, limit: 30 })

  return candidates.map((c, i) => ({
    index: i + 1,
    document: c.title,
    heading: c.headingPath,
    excerpt: c.content.slice(0, 200),
    productionRank: c.productionRank ?? null,   // where the real pipeline put it
  }))
}

label: "whats the rma window on damaged stuff"            [stratum: no_results]

  1. Returns and Credits Policy > Returns > Damaged goods        (prod: —)
     Damaged goods may be returned within 30 days of delivery…
  2. Returns and Credits Policy > Returns > Late returns         (prod: 1)
     Items returned after 30 days are assessed individually…
  3. RMA Procedure > Raising an RMA                              (prod: —)
     An RMA number must be obtained before any return is shipped…

  supporting? 1,3    distractors? 2    tags? exact_token,jargon

Two things that make this work:

Show where production ranked each candidate. A chunk the administrator-scoped deep search found at rank 1 and production never returned is the finding, visible without any analysis.

Capture distractors while labelling. Candidate 2 is the near-miss, and recording it costs one keystroke now and is the thing that makes the numbers honest later.

# The case, written verbatim. ⚠️ Keep the typo and the jargon. Correcting
# "whats" to "what is" removes exactly the property being tested.
- question_id: real-0041
  question: "whats the rma window on damaged stuff"
  asker: inside_sales
  stratum: no_results
  tags: [jargon, typo, single_source]
  supporting:
    - { document: "Returns and Credits Policy", heading: ["Returns", "Damaged goods"] }
    - { document: "RMA Procedure", heading: ["Raising an RMA"] }
  distractors:
    - { document: "Returns and Credits Policy", heading: ["Returns", "Late returns"] }

Follow-ups need their history

31% of real questions are follow-ups, and a follow-up without its context is a different question.

- question_id: real-0088
  question: "what about electronics"
  asker: inside_sales
  history:
    - question: "What's the returns window for damaged goods?"
  tags: [follow_up, short]
  # The honest expectation: the corpus does not distinguish electronics, so
  # correct behaviour is to retrieve the general policy, and generation
  # abstains on the category. Chapter 10's abstention case.
  supporting:
    - { document: "Returns and Credits Policy", heading: ["Returns", "Damaged goods"] }
  expect_abstention_on_category: true

// Which means the harness must replay history, not just the question.
const prepared = await prepareQuery(testCase.question, testCase.history ?? [])
const results = await retrieveAndRerank(prepared.search, { scope, ...config })

Without that, every follow-up case tests a question nobody asked and the context stitching that was built for them is never exercised.

Keeping the set representative as it grows

A set that grows only by adding failures drifts toward being a set of failures.

// Target a composition, and report drift from it rather than discovering it.
const TARGET_COMPOSITION = {
  no_results: 0.15,
  unhelpful: 0.15,
  frequent: 0.20,
  random: 0.35,
  written_original: 0.15,   // the first 34, kept as a stable baseline
} as const

export function compositionDrift(cases: RetrievalCase[]) {
  const actual = countBy(cases, (c) => c.stratum)
  return Object.entries(TARGET_COMPOSITION).map(([stratum, target]) => ({
    stratum,
    target,
    actual: (actual[stratum] ?? 0) / cases.length,
    drift: (actual[stratum] ?? 0) / cases.length - target,
  }))
}

Keeping the original 34 as a labelled stratum is worth it. They are the only cases whose history is continuous from before the system existed, so they are the baseline that makes a long-run comparison meaningful.

// And retire cases deliberately, rather than accumulating forever.
export function retirementCandidates(cases: RetrievalCase[], history: EvalHistory) {
  return cases.filter((c) => {
    const runs = history.forCase(c.question_id)
    // Passed every time for six months: it is no longer measuring anything.
    return runs.length > 50 && runs.every((r) => r.recall === 1 && r.mrr === 1)
  })
}

⚠️ Retire by moving to a separate stable.yaml that runs weekly rather than per-commit, not by deleting. A case that has passed 50 times is still the one that catches a catastrophic regression; it just does not need to run on every pull request.

How big

Set size	Detects	Labelling cost
30	A 15-point change	1 hour
100	A 7-point change	3 hours
250	A 4-point change	7 hours
1,000	A 2-point change	A project

The useful number for most internal systems is 100 to 250. Below 100 the noise band is wider than most real improvements, which is chapter 31's overfitting problem. Above 250 the labelling cost grows faster than the discrimination.

Atlas settled at 120: the original 34 plus 86 harvested over two months.

What the real cases revealed

Scored against the production pipeline at the time:

Subset	n	recall@6	Finding
Written original	34	0.99	The set it was tuned on
Harvested: frequent	24	0.96	Common questions work
Harvested: random	32	0.91	The honest number
Harvested: unhelpful	16	0.69	—
Harvested: no results	14	0.43	—

The bottom two rows were invisible before this exercise, and they decompose usefully:

no_results (14), diagnosed:
  not_ingested      6   scanned contracts — the known gap
  not_retrieved     5   all jargon: "rma", "bol", "cogs"
  ranked_out        3

Five of fourteen were a missing glossary, which is a data file and twenty minutes of work. That finding came from harvesting, not from tuning, and no amount of parameter search would have produced it.

The failure mode: labelling with the system's own answer

The quickest way to label is to run the question, look at what came back, and mark the plausible chunk as supporting.

This produces a test set that defines correct as whatever the system currently returns. Scores are high immediately, every change looks like a regression, and the set actively resists improvement.

// Three guards against it.

// 1. Label from a DEEP administrator search, not from production output.
//    30 candidates, unfiltered, so the labeller sees what production missed.
const candidates = await retrieveAndRerank(question, { scope: ADMIN, limit: 30 })

// 2. Hide the production ranking until after the choice is made.
//    Shown first, it anchors the labeller on rank 1.
const display = candidates.map((c) => omit(c, ['productionRank', 'score']))

// 3. For any case where nothing in 30 candidates is right, label it
//    `supporting: unknown` and escalate — the answer may not be in the
//    corpus at all, which is chapter 18's no-source finding, not a
//    retrieval failure.

The third is the one that found the six scanned contracts in the table above. A labeller who must pick something from the list would have picked the nearest contract chunk, and six cases would have been permanently mislabelled as retrieval failures against a document that does not exist.

A labeller who can only choose from what was retrieved cannot record that the right answer was never there.

Key points

The written set measures anticipated language. Real questions are shorter, full of pronouns and jargon, and a quarter are confirmations.
Harvest in four strata: no results, unhelpful, frequent, and random. The random stratum is what keeps the set from being all failures.
Label by confirming a deep administrator-scoped search, not by searching. Ninety seconds a case instead of ten minutes.
Capture distractors while labelling. One keystroke now, honest numbers later.
Keep the question verbatim, typos and jargon included. Correcting them removes the property under test.
Follow-ups need their history recorded, and the harness must replay it.
Target a composition and report drift. Keep the original cases as a stable baseline stratum.
Retire always-passing cases to a weekly set rather than deleting them.
100 to 250 cases is the useful range. Below 100, noise exceeds most real improvements.
Never label from production output. A labeller who can only choose from what was retrieved cannot record that the answer was never there.

Next, and the last chapter of this section: tuning against the test set, and knowing when to stop.