Chapter 41 · Retrieval

Your first vector search, and why it disappoints

Four specific disappointments, each with a specific cause and a specific fix. Nothing here is a reason to abandon vector search, and all four are visible in one afternoon.

11 min read

The index is built, the corpus is in, and the first serious session of asking it questions is disappointing. This is universal and it is not a sign that something is broken.

Four disappointments, in the order they appear. Each has a cause, a fix, and a measurement.

Disappointment 1: it always returns something

Ask about electronics when the corpus has nothing about electronics. You get twelve results, the top one scoring 0.41, and an answer about restocking fees.

The cause is chapter 38's second property: distance is relative. order by ... limit 12 returns the twelve nearest chunks in the corpus whether or not any of them is relevant. There is no notion of "far enough away to be irrelevant" unless you supply one.

// The floor, derived rather than guessed. Run both distributions once.
export async function calibrateFloor(): Promise<number> {
  // Questions you know the corpus answers.
  const known = await Promise.all(ANSWERABLE.map(async (q) =>
    (await retrieve(q, { scope: ADMIN, limit: 1 }))[0]?.score ?? 0))

  // Questions about things that are definitely absent.
  const absent = await Promise.all(UNANSWERABLE.map(async (q) =>
    (await retrieve(q, { scope: ADMIN, limit: 1 }))[0]?.score ?? 0))

  // ⚠️ The floor goes between the distributions, nearer the absent side —
  // a false negative costs an abstention, a false positive costs a wrong answer.
  const floor = percentile(absent, 0.95)
  logger.info('floor.calibrated', {
    floor, knownP05: percentile(known, 0.05), absentP95: percentile(absent, 0.95),
  })
  return floor
}

Atlas's numbers: answerable questions' top score p05 was 0.44; unanswerable questions' top score p95 was 0.31. A floor of 0.33 separates them with room on both sides.

// Applied, with the distinction that matters for generation.
const results = rows.filter((r) => r.score >= config.retrieval.floor)
if (results.length === 0) {
  // Not an error, and not an empty list either. A named outcome that
  // generation handles explicitly, per chapter 9.
  return { kind: 'no_relevant_results', bestScore: rows[0]?.score ?? 0 }
}

⚠️ If the two distributions overlap heavily, no floor separates them and the problem is upstream — usually chunking or the heading path. The calibration is also a diagnostic.

Disappointment 2: it misses exact terms

"What is the lead time for SKU-4411?" returns chunks about lead times in general and never the specification sheet for that SKU.

The cause is chapter 38's fourth property: codes embed almost identically. SKU-4411 and SKU-4412 are 0.98 similar, so the vector contributes almost nothing to distinguishing them, and ranking is decided by the rest of the sentence — which is about lead times generally.

This is not fixable with vectors. It is chapter 43's subject, and the interim measurement is to size the problem:

// How many of your questions contain an exact token? For Atlas: 11 of 34.
const CODE = /\b(?:[A-Z]{2,}-\d{2,}|\d{4,}|v\d+\.\d+|§\s?\d+(?:\.\d+)*)\b/
const withCodes = cases.filter((c) => CODE.test(c.question))
console.log(`${withCodes.length}/${cases.length} questions contain an exact token`)
console.log('recall on those:', round(scoreSubset(withCodes).recall12))   // 0.61
console.log('recall on the rest:', round(scoreSubset(others).recall12))   // 1.00

A perfect 1.00 on questions without codes and 0.61 on questions with them is an unambiguous instruction about where the next work goes.

Disappointment 3: it returns five chunks of the same document

A question about returns retrieves twelve chunks, nine of them from the Returns and Credits Policy, five of those adjacent. The context window is full and covers one document's worth of ground.

The cause is that adjacent chunks of a relevant document are all relevant, especially with overlap. Ranking by similarity alone has no concept of redundancy.

-- Diversity by document, in SQL. Rank within each document, then take the
-- best few per document before the global ordering.
with scored as (
  select c.id, c.content, c.document_id, c.heading_path,
         1 - (c.embedding <=> $1) as score,
         row_number() over (
           partition by c.document_id order by c.embedding <=> $1
         ) as rank_in_doc
    from chunks c
    join documents d on d.id = c.document_id
   where d.deleted_at is null
     and d.classification = any($2)
     and (d.account_id is null or d.account_id = any($3))
   order by c.embedding <=> $1
   limit 100                      -- over-fetch before diversifying
)
select * from scored
 where rank_in_doc <= 3           -- at most three chunks per document
 order by score desc
 limit 12;

⚠️ The over-fetch must come before the per-document cap, or you are diversifying a list that was already truncated. Fetching 100 and keeping 12 costs nothing — the index work is dominated by the search, not by the row count.

Three per document is a starting point. Measure it:

for (const cap of [1, 2, 3, 5, 12]) {
  const s = await scoreRetrieval(cases, { limit: 12, perDocumentCap: cap })
  console.log(cap, round(s.recall12), round(s.precision12), s.distinctDocs)
}
// 1   0.85  0.09  12.0
// 2   0.90  0.11   8.4
// 3   0.91  0.11   6.9    <- knee
// 5   0.90  0.11   5.2
// 12  0.89  0.11   4.1

A cap of 1 is too aggressive — a two-chunk answer gets truncated. A cap of 3 is the knee, and it is also the point at which a multi-part answer still fits.

Disappointment 4: short questions work worse than long ones

"returns?" retrieves badly. "What is the policy for returning goods that arrived damaged?" retrieves well.

The cause is that a short question is a short text, and a short text's embedding is dominated by its few tokens. There is less signal to match against a 500-token chunk.

Two fixes, and the first is cheap:

// 1. Expand the question with conversation context, when there is any.
//    "returns?" following "tell me about the Henderson account" becomes a
//    much better query. Chapter 46 does this properly.
const expanded = history.length
  ? `${history.slice(-2).map((t) => t.question).join(' ')} ${question}`
  : question

// 2. Detect and handle the degenerate case rather than retrieving badly.
if (countTokens(question) < 3 && !history.length) {
  return { kind: 'question_too_short' }
}

The second is the honest answer for a genuinely contextless two-word query. Asking the person what they want is better than retrieving randomly and answering confidently.

The query that does all four

// src/retrieve/search.ts — the version at the end of this chapter.
export async function retrieve(
  question: string,
  options: { scope: Scope; limit?: number },
): Promise<RetrievalResult> {
  const limit = options.limit ?? 12
  if (countTokens(question) < 3) return { kind: 'question_too_short', results: [] }

  const vector = await embedQuery(question)

  const { rows } = await db.query(
    `with scored as (
       select c.id, c.content, c.heading_path, c.kind, c.document_id,
              d.title, d.source, d.source_updated_at, d.parse_warnings,
              1 - (c.embedding <=> $1) as score,
              row_number() over (partition by c.document_id
                                 order by c.embedding <=> $1) as rank_in_doc
         from chunks c
         join documents d on d.id = c.document_id
        where d.deleted_at is null
          and d.classification = any($2)
          and (d.account_id is null or d.account_id = any($3))
          and c.retrievable
        order by c.embedding <=> $1
        limit $4
     )
     select * from scored
      where rank_in_doc <= $5 and score >= $6
      order by score desc
      limit $7`,
    [toVector(vector), options.scope.classifications, options.scope.accountIds,
     limit * 8, config.retrieval.perDocumentCap, config.retrieval.floor, limit],
  )

  if (rows.length === 0) return { kind: 'no_relevant_results', results: [] }
  return { kind: 'ok', results: rows.map(toRetrieved) }
}

Every clause in it is one of this chapter's four fixes or an earlier chapter's requirement: the permission filter from chapter 13, retrievable from boilerplate suppression, the per-document cap, the floor, and the payload columns from chapter 33.

Where this leaves the numbers

	Before	After
recall@12	0.89	0.91
precision@12	0.11	0.11
Distinct documents in the window	4.1	6.9
Unanswerable questions returning results	100%	6%

The headline recall barely moved. The fourth row is the real result: questions the corpus cannot answer now return nothing 94% of the time instead of always returning something plausible — which removes a whole class of confident wrong answer that no generation change could have fixed.

Precision is unchanged, because none of these fixes is a ranking improvement. That is reranking, two chapters away.

The failure mode: the floor that was tuned on a full corpus

The floor is calibrated at 0.33 and works. Six months later the corpus has tripled, and questions start returning nothing.

The cause is that similarity scores drift with corpus density. With three times the chunks, the nearest neighbour of a random query is closer — there are simply more candidates near any point — so the "absent" distribution shifts up, and a floor calibrated against the old one is now below it.

// Recalibrate on a schedule, and alert on movement rather than waiting for
// complaints. This is cheap: two sets of questions, one query each.
export async function checkFloorCalibration() {
  const absentP95 = percentile(
    await Promise.all(UNANSWERABLE.map((q) => topScore(q))), 0.95)

  if (absentP95 > config.retrieval.floor) {
    logger.error('floor.below_noise', {
      floor: config.retrieval.floor, absentP95,
      note: 'Unanswerable questions now pass the floor. Recalibrate.',
    })
  }
  // The other direction matters too: a floor far above the noise is
  // suppressing real answers.
  if (absentP95 < config.retrieval.floor - 0.1) {
    logger.warn('floor.too_high', { floor: config.retrieval.floor, absentP95 })
  }
}

Keep the calibration questions. Both sets — answerable and unanswerable — are a permanent asset, and the unanswerable set is the one nobody thinks to write down.

Key points

Vector search always returns something. A score floor is the only thing that makes "nothing relevant" expressible.
Calibrate the floor from two distributions — answerable and unanswerable questions — and place it nearer the absent side.
Exact tokens are not a vector problem. Size it: split recall by whether the question contains a code.
Cap chunks per document, and over-fetch before capping. Three is usually the knee.
Short questions retrieve badly. Expand from conversation history, and refuse a genuinely contextless two-word query.
These four fixes move recall barely and move spurious answers enormously: unanswerable questions returning results fell from 100% to 6%.
Precision is unchanged by all of this. Precision is reranking.
Similarity distributions drift as the corpus grows. Recalibrate on a schedule and keep both question sets.

Next: keyword search — and why it beats vectors more often than anyone expects.