Chapter 44 · Retrieval
Reranking, and when it earns its latency
Precision@12 of 0.12 means one useful chunk in eight reaching the model. A cross-encoder fixes that, costs 200ms, and the decision about whether to pay is a budget question with a clear answer.
11 min read
After hybrid retrieval, Atlas's recall@12 is 0.97 and precision@12 is 0.12. The passage is almost always found. Seven of every eight chunks sent to the model are not useful.
That is what reranking fixes, and it is the largest remaining quality gain in this section.
Why precision matters when recall is solved
Three consequences of a window that is seven-eighths noise.
Cost. Twelve chunks at ~420 tokens is ~5,000 input tokens per question, of which ~4,400 are irrelevant. At 800 questions a day that is most of the bill.
Latency. Generation time scales with input length. Trimming the window is the most direct latency lever available.
Quality. The one that justifies the chapter. A model given one relevant passage among eleven distractors is measurably more likely to answer from a distractor — particularly when the distractor is topically adjacent, like the late-returns clause beside the damaged-goods clause.
Recall decides whether a right answer is possible. Precision decides how likely it is.
Bi-encoders and cross-encoders
The structural difference, and the reason reranking can do what retrieval cannot.
A bi-encoder must compress a chunk into a vector before it knows what will be asked. A cross-encoder reads the question and the chunk together, so it can tell that the chunk mentions 30 days in the context of late returns rather than damaged goods — a distinction no single vector can carry.
The cost is that it cannot be precomputed. Every (question, chunk) pair is a forward pass, which is why it reranks 50 candidates rather than scoring 44,387.
Wiring it in
// src/retrieve/rerank.ts
export async function rerank(
question: string,
candidates: Retrieved[],
options: { keep: number; minScore?: number },
): Promise<Retrieved[]> {
if (candidates.length <= options.keep) return candidates // nothing to do
const scores = await crossEncoder.score(
candidates.map((c) => ({
query: question,
// ⚠️ Rerank the text the model will SEE, including the heading path.
// Scoring different text from what is sent means the ranking is for a
// document that does not exist.
document: `${c.headingPath.join(' > ')}\n\n${c.content}`,
})),
)
return candidates
.map((c, i) => ({ ...c, rerankScore: scores[i] }))
.filter((c) => c.rerankScore >= (options.minScore ?? -Infinity))
.sort((a, b) => b.rerankScore - a.rerankScore)
.slice(0, options.keep)
}// And the pipeline, which now over-fetches deliberately.
const candidates = await hybridRetrieve(question, { scope, limit: 50 })
const window = await rerank(question, candidates.results, { keep: 6, minScore: 0.1 })Retrieving 50 and keeping 6 is the shape that makes this work: candidate generation optimises recall at k=50, reranking optimises precision at k=6, and each stage is measured on the metric it controls — exactly the split chapter 37 set up.
What it buys, measured
| Configuration | recall@k | precision@k | Tokens | p50 latency |
|---|---|---|---|---|
| Hybrid, top 12 | 0.97 | 0.12 | 5,040 | 14 ms |
| Hybrid 50 → rerank → top 12 | 0.97 | 0.21 | 5,040 | 218 ms |
| Hybrid 50 → rerank → top 6 | 0.97 | 0.38 | 2,520 | 214 ms |
| Hybrid 50 → rerank → top 3 | 0.94 | 0.61 | 1,260 | 211 ms |
| Hybrid, top 6, no rerank | 0.91 | 0.19 | 2,520 | 11 ms |
The third row is the result worth dwelling on. Against the baseline it holds recall at 0.97, triples precision, halves the tokens sent to the model, and costs 200ms.
The last row is the control that makes the case: simply truncating to 6 without reranking loses six points of recall. The reordering is doing the work, not the truncation.
The fourth row shows where it breaks. At top 3, recall falls — some answers genuinely need two or three chunks, and three is not enough margin.
Is 200ms worth it
A budget question with a clear answer, against the targets from chapter 10: p50 ≤ 4s, p95 ≤ 10s.
| Stage | Before | After |
|---|---|---|
| Embed the question | 60 ms | 60 ms |
| Hybrid retrieval | 14 ms | 14 ms |
| Rerank | — | 200 ms |
| Generation | 2,400 ms | 1,500 ms |
| Total | 2,474 ms | 1,774 ms |
Reranking made the system faster, because halving the input tokens saved more generation time than the reranker cost. That is the common outcome and it is counter-intuitive enough that teams often reject reranking on latency grounds without measuring it.
⚠️ This depends on generation dominating the budget, which it does whenever the model is writing a paragraph. For a system returning only a list of links, the arithmetic reverses.
Hosted against self-hosted
// Hosted: an API call. Simple, one more dependency and one more egress path.
const scores = await fetch('https://api.provider.com/rerank', {
method: 'POST',
body: JSON.stringify({ query: question, documents, model: 'rerank-v3' }),
}).then((r) => r.json())| Hosted | Self-hosted | |
|---|---|---|
| Latency | 150–300 ms | 40–120 ms on GPU, 300–900 ms on CPU |
| Cost | Per 1,000 documents scored | Infrastructure |
| Data egress | ⚠️ Every candidate chunk leaves | None |
| Ops | None | A service with uptime |
The egress row is the one that interacts with earlier decisions. A hosted reranker receives 50 chunks of retrieved content per question — more content than generation sees, since generation only gets 6. For a corpus under the restrictions in chapter 17, that is a second processor to assess, and it is easy to miss because it feels like infrastructure rather than a model call.
A small cross-encoder is genuinely small — tens of millions of parameters, not billions — so self-hosting is more tractable here than for generation.
Cheaper alternatives, and what they are worth
Reranking is not the only precision lever. Three others, all nearly free, and worth measuring before paying 200ms.
// 1. Maximal marginal relevance: penalise a candidate for similarity to
// already-selected ones. Diversity without a model.
export function mmr(candidates: Retrieved[], keep: number, lambda = 0.7): Retrieved[] {
const selected: Retrieved[] = []
const pool = [...candidates]
while (selected.length < keep && pool.length) {
let bestIndex = 0, bestScore = -Infinity
for (const [i, c] of pool.entries()) {
const redundancy = selected.length
? Math.max(...selected.map((s) => cosineSimilarity(c.embedding, s.embedding)))
: 0
const score = lambda * c.score - (1 - lambda) * redundancy
if (score > bestScore) { bestScore = score; bestIndex = i }
}
selected.push(pool.splice(bestIndex, 1)[0])
}
return selected
}// 2. Score-gap truncation: cut where the score drops off, rather than at a
// fixed k. A question with one clear answer sends one chunk.
export function truncateAtGap(results: Retrieved[], maxKeep: number): Retrieved[] {
for (let i = 1; i < Math.min(results.length, maxKeep); i++) {
if (results[i].score < results[i - 1].score * 0.6) return results.slice(0, i)
}
return results.slice(0, maxKeep)
}// 3. The per-document cap from chapter 41, which is already in place.Measured on Atlas:
| Technique | precision@6 | recall@6 | Added latency |
|---|---|---|---|
| Baseline, hybrid top 6 | 0.19 | 0.91 | 0 |
| + per-document cap | 0.22 | 0.93 | 0 |
| + MMR | 0.25 | 0.93 | 3 ms |
| + score-gap truncation | 0.29 | 0.92 | 0 |
| Cross-encoder rerank | 0.38 | 0.97 | 200 ms |
The free techniques get from 0.19 to 0.29 — about half the available gain for none of the cost, and worth taking regardless. The cross-encoder is the only one that also improves recall, because it is the only one that can promote a chunk that hybrid ranked low.
The failure mode: reranking a window that was already truncated
The subtle one, and it undoes the entire technique while appearing to work.
// ❌ Retrieve 12, rerank 12, keep 6. The reranker can only reorder what
// hybrid already chose — so recall is capped at hybrid's recall@12, and
// the chunk ranked 23rd that the cross-encoder would have promoted to
// first was never a candidate.
const candidates = await hybridRetrieve(question, { scope, limit: 12 })
const window = await rerank(question, candidates.results, { keep: 6 })// ✅ Retrieve 50, rerank 50, keep 6. The reranker gets a deep candidate set.
const candidates = await hybridRetrieve(question, { scope, limit: 50 })
const window = await rerank(question, candidates.results, { keep: 6 })| Candidates reranked | recall@6 | precision@6 |
|---|---|---|
| 12 | 0.91 | 0.33 |
| 25 | 0.95 | 0.36 |
| 50 | 0.97 | 0.38 |
| 100 | 0.97 | 0.38 |
The wrong version still improves precision from 0.19 to 0.33, so it looks like it is working. Recall stays at 0.91 — six points below what it should be — and nothing in the precision number reveals it.
// The assertion that prevents it, because the relationship is easy to break
// in a refactor and impossible to see in the output.
export async function retrieveAndRerank(question: string, options: RetrieveOptions) {
const candidateLimit = options.limit * config.retrieval.overFetchFactor // 8x
if (candidateLimit < 25) {
throw new Error(
`reranking ${candidateLimit} candidates to keep ${options.limit}: ` +
`too shallow. The reranker cannot promote what retrieval did not return.`,
)
}
// ...
}And the diminishing return above 50 is worth knowing: 100 candidates costs twice the reranker time for no gain. 50 is the knee on this corpus, and like every other knee in this section it should be measured rather than inherited.
Key points
- Recall decides whether a right answer is possible; precision decides how likely it is.
- A cross-encoder reads the question and chunk together, so it can make distinctions no single vector carries. That is why it cannot be precomputed.
- Rerank the exact text the model will see, heading path included.
- Retrieve deep, keep shallow: 50 candidates, keep 6. Recall held at 0.97, precision tripled, tokens halved.
- Reranking often makes the system faster, because halving input tokens saves more generation time than the reranker costs.
- A hosted reranker sees more content than generation does — 50 chunks against 6. That is a processor decision, not an infrastructure one.
- The free techniques — per-document cap, MMR, score-gap truncation — get about half the precision gain for no latency. Take them anyway.
- Only the cross-encoder improves recall, because only it can promote a low-ranked chunk.
- Reranking a shallow candidate set looks like it works: precision improves and recall is silently capped. Assert on the over-fetch factor.
Next: metadata filters and permission-aware retrieval — the filter in the hot path, and what it does to an approximate index.