Chapter 37 · Retrieval

Retrieval as its own system

Retrieval has its own quality bar, its own tests and its own failure modes, and it is measured before a model sees anything. The ceiling argument, and what section 4 is going to move.

10 min read

Section 4 opens on a specific number. The first full ingest produced recall@12 of 0.89, precision@12 of 0.11, and two named failing questions. Everything in this section moves those.

The organising idea is one sentence: retrieval is a system with its own quality bar, and it is measured without a model in the loop.

The ceiling argument

Generation quality is bounded above by retrieval quality. If the supporting passage is not in the context window, the best available model produces either an abstention or an invention, and the abstention is the good outcome.

This is why the failure classification from chapter 10 splits retrieval:absent from generation, and why that split is worth the effort: the two buckets send you to different sections of this series.

A wrong answer is a retrieval problem about four times in five. Investigating the prompt first is the most common waste of time in this whole discipline.

Measuring without a model

The practical consequence, and the thing that makes this section tractable: retrieval evaluation is fast, deterministic and free.

	Retrieval eval	End-to-end eval
Runs in	Seconds	Minutes
Cost	Zero	A model call per case
Deterministic	Yes	No
Tells you	Whether the passage was found	Whether the answer was right

// evals/retrieval.ts — the whole harness. No model, no judge, no flakiness.
export async function scoreRetrieval(cases: RetrievalCase[], config: RetrievalConfig) {
  const rows = await Promise.all(cases.map(async (testCase) => {
    const supporting = new Set(await resolveSupporting(testCase.supporting))
    const scope = await resolveScope(users[testCase.asker])
    const results = await retrieve(testCase.question, { scope, ...config })

    return {
      id: testCase.id,
      recall12: recallAtK(results, supporting, 12),
      recall5: recallAtK(results, supporting, 5),
      rr: reciprocalRank(results, supporting),
      precision12: precisionAtK(results, supporting, 12),
      rank: results.findIndex((r) => supporting.has(r.id)) + 1 || null,
    }
  }))

  return { rows, ...aggregate(rows) }
}

Being able to run this in seconds changes how you work. A chunking change, a weight change, a filter change — each is a two-second experiment rather than a twenty-minute one, and that difference is what makes chapter 50's tuning possible at all.

What retrieval is actually doing

Five stages, and the section is organised around them.

The important structural point is that filtering happens during candidate generation, not after it. The diagram shows filter as stage 3 for clarity; in the SQL it is in the same where clause as the vector search, for the reason chapter 13 gave. A forbidden chunk must never be a candidate.

Recall and precision do different jobs

The two numbers are not two views of the same thing. They constrain different stages.

// Recall is the ceiling. Precision is the budget.
//
// Stage 2 (candidate generation) is judged on RECALL at a large k. Its job is
// to not lose the answer; it is allowed to return a lot of rubbish.
const candidates = await retrieve(question, { scope, limit: 50 })   // recall@50

// Stage 4 (reranking) is judged on PRECISION at a small k. Its job is to
// throw away the rubbish without losing the answer.
const window = await rerank(question, candidates, { keep: 12 })     // precision@12

Atlas's numbers read differently in that light:

Metric	Value	Reading
recall@50	0.94	Candidate generation is nearly good enough
recall@12	0.89	Five percentage points lost to ranking, not to search
precision@12	0.11	One useful chunk in nine reaching the model
MRR	0.59	The answer is around rank 2 on average when found

The gap between recall@50 and recall@12 is the reranking opportunity — the passage is already being found and is being ranked below the window. That is a cheaper problem than a passage that is not found at all, and it is chapter 44.

Where this section's gains come from

Ordered by what they typically buy, which is not the order people attempt them.

Change	Typical gain	Chapter
Hybrid keyword + vector	+0.04 to +0.10 recall	43
Reranking	+0.15 to +0.30 precision	44
Query rewriting	+0.02 to +0.06 recall	46
A better embedding model	+0.01 to +0.05 recall	39
A different vector store	~0	40

The last row is deliberate and it is the one that most often absorbs a week. Vector stores differ on operations, cost and scaling. They do not differ meaningfully on retrieval quality for a corpus of this size, because they are all computing the same distance over the same vectors. Chapter 40 is about choosing one on the things that do differ.

The second row is where the money is, and it is why precision@12 of 0.11 is reported as headroom rather than as a failure.

The three questions to ask of any retrieval failure

Before changing anything, classify. This is the retrieval-level version of chapter 10's three buckets.

export function diagnoseRetrieval(
  testCase: RetrievalCase, supporting: Set<number>, results: Retrieved[],
): Diagnosis {
  // 1. Is the passage in the corpus at all?
  if (supporting.size === 0) return 'not_ingested'

  // 2. Was it a candidate? If not, candidate generation lost it — a matching
  //    problem, addressed by hybrid search or query rewriting.
  const deepResults = results.slice(0, 50)
  if (!deepResults.some((r) => supporting.has(r.id))) return 'not_retrieved'

  // 3. Was it a candidate but ranked out of the window? A ranking problem,
  //    addressed by reranking.
  if (!results.slice(0, 12).some((r) => supporting.has(r.id))) return 'ranked_out'

  return 'ok'
}

Three diagnoses, three different chapters, and no overlap between them. A not_retrieved case fixed by buying a reranker is a week spent on the wrong stage.

# Run it over the test set and read the distribution before touching anything.
npm run evals:retrieval -- --diagnose
# ok             30
# ranked_out      2    spc-007, pol-011
# not_retrieved   0
# not_ingested    2    ctr-003, ctr-008   (known: scanned contracts excluded)

Two ranked_out and zero not_retrieved is a very specific instruction: skip to reranking, and do not spend time on hybrid search yet. The section still covers hybrid search, because a corpus with different questions will show the opposite distribution — the point is to read your own.

What retrieval is not responsible for

Worth being explicit, because scope creep here is common.

Not responsible for the answer being right. A perfectly retrieved passage that the model misreads is a generation failure.

Not responsible for deciding there is no answer. Returning zero results above the score floor is retrieval doing its job correctly. Abstaining gracefully is chapter 63.

Not responsible for freshness. A retrieved chunk from a stale index is chapter 19's problem, and the stamp travels with the chunk.

Not responsible for permissions being correct — only for enforcing them. The predicate came from chapter 14; retrieval's duty is to put it in the where clause and never fetch a forbidden chunk.

The failure mode: tuning retrieval by reading answers

The way this section goes wrong is to evaluate retrieval by looking at generated answers.

It is natural — the answer is what you care about, and reading twenty answers feels like evidence. It is a bad instrument for three reasons:

It is non-deterministic. The same retrieval produces different answers, so a change that looks like an improvement may be sampling.
It conflates two systems. A better answer from the same chunks is a generation change.
It is slow and expensive, so you run it rarely, so you tune on a handful of cases.

// ❌ The instrument that feels right.
for (const q of questions) console.log(await ask(q))   // read 20 answers, form a view

// ✅ The instrument that measures the thing being changed.
const before = await scoreRetrieval(cases, currentConfig)
const after  = await scoreRetrieval(cases, candidateConfig)
console.table({ before: summary(before), after: summary(after) })

Read answers to find new failure modes — that is genuinely what it is good for, and it is how must_not_contain cases get written. Then encode what you found as a retrieval case and measure it mechanically from then on.

Read answers to discover problems. Measure retrieval to decide whether you fixed one.

Key points

Generation quality is bounded above by retrieval quality. A wrong answer is a retrieval problem about four times in five.
Retrieval evaluation is fast, deterministic and free, which is what makes systematic tuning possible.
Filtering belongs inside candidate generation, not after it. A forbidden chunk must never be a candidate.
Candidate generation is judged on recall at a large k; reranking is judged on precision at a small k.
The gap between recall@50 and recall@12 is the reranking opportunity, and it is a cheaper problem than a passage never found.
Reranking and hybrid search buy the most. Changing vector store buys approximately nothing in quality.
Diagnose every failure as not-ingested, not-retrieved or ranked-out. The three send you to different chapters.
Read answers to discover problems; measure retrieval to decide whether you fixed one.

Next: embeddings, for someone shipping — what a vector actually represents, and the properties that matter in production.