Chapter 48 · Retrieval

Evaluating retrieval on its own

The harness, formalised: what to measure, what to report, how to make a regression fail a build, and the four metrics that are worth tracking against the nine that are not.

12 min read

Chapter 37 argued that retrieval is measured without a model. Chapters 41 to 47 have been using that harness informally. This chapter builds it properly, because everything after this point — tuning, section 5's prompt work, section 9's full eval suite — depends on it being trustworthy.

The four metrics worth tracking

There are a dozen retrieval metrics in the literature. Four of them tell you something you would act on.

// src/evals/metrics.ts
export interface RetrievalScore {
  /** Did any supporting chunk reach the window? The ceiling. */
  recall: number
  /** 1/rank of the first supporting chunk. Rank quality. */
  mrr: number
  /** Share of the window that was supporting. The cost of recall. */
  precision: number
  /** Supporting chunks found, over supporting chunks that exist. Coverage
   *  for questions whose answer spans several chunks. */
  completeness: number
}

completeness is the one usually omitted, and it matters for any corpus where answers span clauses. A question needing both the rule and its exception scores recall = 1 when only the rule is found, and the answer is wrong.

export function score(results: Retrieved[], supporting: Set<number>, k: number): RetrievalScore {
  const window = results.slice(0, k)
  const found = window.filter((r) => supporting.has(r.id))
  const firstIndex = window.findIndex((r) => supporting.has(r.id))

  return {
    recall: found.length > 0 ? 1 : 0,
    mrr: firstIndex === -1 ? 0 : 1 / (firstIndex + 1),
    precision: window.length ? found.length / window.length : 0,
    completeness: supporting.size ? found.length / supporting.size : 0,
  }
}

The nine not worth tracking, briefly: NDCG and MAP require graded relevance you do not have; hit rate is recall; F1 blends two numbers that should be read separately because they belong to different stages; and the rest are variants.

The harness

// src/evals/retrieval-harness.ts
export async function runRetrievalEval(
  cases: RetrievalCase[],
  config: RetrievalConfig,
): Promise<EvalReport> {
  const rows = await mapWithConcurrency(cases, 8, async (testCase) => {
    const supporting = new Set(await resolveSupporting(testCase.supporting))
    const distractors = new Set(await resolveSupporting(testCase.distractors ?? []))
    const scope = await resolveScope(users[testCase.asker])

    const t0 = performance.now()
    const result = await retrieveAndRerank(testCase.question, { scope, ...config })
    const latencyMs = performance.now() - t0

    const s = score(result.results, supporting, config.limit)
    return {
      id: testCase.id,
      ...s,
      latencyMs,
      // ⚠️ Did a distractor outrank the answer? This is the number that
      // moves when something subtle breaks, and it moves before recall does.
      distractorAboveAnswer: rankOf(result.results, distractors) <
                             rankOf(result.results, supporting),
      diagnosis: diagnose(testCase, supporting, result.results),
      tags: testCase.tags ?? [],
    }
  })

  return { rows, summary: summarise(rows), bySubset: subsets(rows) }
}

distractorAboveAnswer is the early-warning metric. A change that degrades ranking shows up there one or two cases before it shows up in recall, because recall is binary at the window boundary and this is not.

Report subsets, always

An aggregate over 34 cases hides everything. Chapter 39 showed four embedding models tying on the aggregate and differing on codes; the same applies to every change.

# Tag every case, and the tags are the report.
- question_id: spc-003
  question: "What is the lead time for SKU-4411?"
  tags: [exact_token, single_source, product_specs]
  supporting:
    - document: "SKU-4411 Specification"
      heading: ["Availability"]

// The subsets that have earned their place on this corpus.
const SUBSETS = [
  'exact_token',      // chapter 42: where keyword wins
  'paraphrase',       // chapter 42: where vectors win
  'negation',         // chapter 38: where both are weak
  'multi_chunk',      // completeness matters
  'multi_hop',        // chapter 47
  'permission_narrow',// chapter 45: asked as a scoped user
  'follow_up',        // chapter 46: needs context
] as const

retrieval eval — 34 cases, config: hybrid+rerank, k=6

subset              n   recall   mrr   prec   compl   distractor↑
all                34    0.97   0.71   0.38    0.89         2
exact_token        11    1.00   0.82   0.41    0.95         0
paraphrase          7    1.00   0.74   0.36    0.86         0
negation            4    0.75   0.44   0.29    0.63         2  ⚠️
multi_chunk         6    1.00   0.68   0.44    0.72         0
multi_hop           7    1.00   0.61   0.33    0.81         0
permission_narrow   5    1.00   0.77   0.39    0.90         0
follow_up           3    1.00   0.70   0.35    0.88         0

The negation row is the only interesting one in that table, and it would be invisible in the aggregate. Recall 0.75, completeness 0.63, and both distractor inversions in the corpus — which is chapter 38's warning showing up as a number.

Making a regression fail the build

// scripts/retrieval-gate.ts
// Thresholds per subset, committed, and only ever raised deliberately.
const THRESHOLDS: Record<string, { recall: number; mrr: number }> = {
  all: { recall: 0.95, mrr: 0.65 },
  exact_token: { recall: 0.95, mrr: 0.75 },
  paraphrase: { recall: 0.95, mrr: 0.65 },
  negation: { recall: 0.70, mrr: 0.40 },     // honest about a weak area
  permission_narrow: { recall: 1.00, mrr: 0.70 },  // ⚠️ no tolerance here
}

export function gate(report: EvalReport): string[] {
  const failures: string[] = []
  for (const [subset, threshold] of Object.entries(THRESHOLDS)) {
    const s = report.bySubset[subset]
    if (!s) { failures.push(`subset '${subset}' has no cases`); continue }
    if (s.recall < threshold.recall) {
      failures.push(`${subset}: recall ${s.recall.toFixed(2)} < ${threshold.recall}`)
    }
    if (s.mrr < threshold.mrr) {
      failures.push(`${subset}: mrr ${s.mrr.toFixed(2)} < ${threshold.mrr}`)
    }
  }
  return failures
}

Two deliberate choices in those thresholds.

The negation threshold is set at the current weak number, not at an aspiration. A gate set above current performance fails on every commit and is disabled within a week. Set it just below where you are, and raise it when you improve.

permission_narrow has no tolerance. It is the leakage gate adjacent: a narrowly-scoped user failing to retrieve their own permitted content is a correctness bug, not a quality number.

# .github/workflows/retrieval.yml — it runs in seconds, so it runs on every PR.
- run: npm run evals:retrieval -- --gate

Comparing two configurations honestly

// Paired comparison. With 34 cases the aggregate difference is noise; what
// is informative is WHICH cases changed.
export function compare(before: EvalReport, after: EvalReport) {
  const byId = new Map(before.rows.map((r) => [r.id, r]))
  const improved: string[] = []
  const regressed: string[] = []

  for (const row of after.rows) {
    const prev = byId.get(row.id)
    if (!prev) continue
    if (row.recall > prev.recall || (row.recall === prev.recall && row.mrr > prev.mrr + 0.1)) {
      improved.push(row.id)
    }
    if (row.recall < prev.recall || (row.recall === prev.recall && row.mrr < prev.mrr - 0.1)) {
      regressed.push(row.id)
    }
  }
  return { improved, regressed, net: improved.length - regressed.length }
}

comparison: hybrid -> hybrid + rerank

improved   (7): spc-003 spc-007 pol-011 ctr-004 pri-002 ret-009 inv-001
regressed  (1): pol-004
net: +6

pol-004  "What is the escalation path for a damaged shipment?"
         recall 1 -> 1, mrr 1.00 -> 0.33
         The reranker demoted the correct chunk from 1 to 3.

A named regression is actionable; a headline that moved from 0.91 to 0.97 is not. pol-004 is worth looking at individually, and that is only possible because the comparison is paired.

Report which cases changed, not how the average moved. With a small test set, the average is the least informative view of it.

Keeping the harness honest

Four properties, each with a failure it prevents.

// 1. Resolution must throw on a missing reference. An empty supporting set
//    makes every case pass. (Chapter 18.)
if (resolved.length === 0) throw new Error(`unresolved: ${ref.document} / ${ref.heading}`)

// 2. The harness must use the same code path as production. A harness with
//    its own retrieval function measures the harness.
import { retrieveAndRerank } from '../retrieve/index.js'   // not a copy

// 3. Scope must be resolved the same way. A harness running as an
//    administrator cannot see permission regressions. (Chapter 45.)
const scope = await resolveScope(users[testCase.asker])

// 4. The config under test must be explicit, not ambient. A run that picks
//    up config from the environment is not reproducible.
export async function runRetrievalEval(cases: RetrievalCase[], config: RetrievalConfig)

The second is the one that quietly happens. A harness is written early, production retrieval gains the per-document cap and the floor and stratification, and the harness keeps calling a simplified version — so the numbers describe a system nobody runs.

// The test for it, which is crude and effective.
test('the harness calls production retrieval', async () => {
  const source = await read('src/evals/retrieval-harness.ts')
  expect(source).toMatch(/from '\.\.\/retrieve\//)
  expect(source).not.toMatch(/select .* from chunks/i)   // no SQL of its own
})

The failure mode: the test set that became the specification

Thirty-four cases, measured on every commit, gated in CI. Over three months every change is evaluated against them and the numbers climb: 0.89, 0.93, 0.97, 0.99.

Real questions, measured in chapter 46, sit at 0.94 — and the gap is widening, because each tuning round fits the system a little more closely to these thirty-four phrasings.

The test set stopped being a sample of the question distribution and became the definition of it.

// Three defences, and the third is the one that actually works.

// 1. A held-out set, scored rarely. (Chapter 31.)
const { train, holdout } = splitCases(cases, { trainRatio: 0.7, seed: 42 })

// 2. Track the gap between test-set and production retrieval explicitly.
logger.info('eval.gap', {
  testSetRecall: report.summary.recall,
  productionRecall: await productionRecallEstimate(),   // from chapter 49
  gap: report.summary.recall - (await productionRecallEstimate()),
})

// 3. Grow the test set from production questions, continuously.
//    A test set that is not growing is being overfitted by definition.

-- The queue for defence 3: real questions that retrieved badly, which are
-- the candidates for new cases. Chapter 49 turns these into cases.
select q.question, q.asked_at, q.top_score, q.result_count
  from question_log q
 where q.asked_at > now() - interval '7 days'
   and (q.result_count = 0 or q.top_score < 0.45 or q.feedback = 'unhelpful')
 order by q.asked_at desc
 limit 50;

⚠️ A test set that has not grown in two months is not stable, it is stale. The questions people ask change as they learn what the system can do, and a fixed set measures an increasingly historical distribution.

Key points

Four metrics: recall, MRR, precision, completeness. Completeness is the one usually omitted and it matters wherever answers span chunks.
Track whether a distractor outranked the answer. It moves before recall does.
Tag every case and report subsets. Aggregates hide the only interesting row.
Gate in CI with per-subset thresholds set just below current performance. A gate above where you are gets disabled.
Give permission-scoped retrieval no tolerance. It is a correctness check wearing a quality metric.
Compare configurations pairwise and name the cases that changed. The average is the least informative view of a small set.
The harness must import production retrieval, resolve scope the same way, throw on unresolved references, and take explicit config.
A test set that is not growing is being overfitted. Grow it from production questions that retrieved badly.

Next: building a retrieval test set from real questions — where the cases come from once people are using it.