Chapter 42 · Retrieval

Keyword search, and why it beats vectors more often than expected

BM25 is thirty years old, costs nothing to run, and wins on the questions vectors fail hardest. What it is good at, what it is bad at, and how to set it up properly in Postgres.

11 min read

The previous chapter left one problem unfixed: recall of 0.61 on the eleven questions containing a SKU, a clause number or a version string, against 1.00 on everything else.

Keyword search fixes that, and it is worth being clear about the framing. This is not a fallback for when the clever approach fails. On a meaningful share of real questions, BM25 is simply the better retrieval method, and a system without it is leaving the easiest available gains on the table.

What BM25 scores

Term frequency, weighted by how rare the term is across the corpus, normalised by document length.

score(D, Q) = Σ IDF(qᵢ) · f(qᵢ, D) · (k₁ + 1)
                          ─────────────────────────────────
                          f(qᵢ, D) + k₁ · (1 - b + b · |D|/avgdl)

Three behaviours fall out of it, and all three are exactly what vectors lack.

Rare terms dominate. SKU-4411 appears in two chunks of 44,387, so its IDF is enormous. Any chunk containing it outranks everything else, which is correct and is the opposite of the vector behaviour where it is nearly invisible.

Saturation. The k₁ term means the fifth occurrence of a word adds much less than the second, so a chunk that repeats a term does not dominate on repetition alone.

Length normalisation. The b term stops long chunks winning simply by containing more words.

Where each method wins

Question	Vector	Keyword
"How long to return damaged goods?"	✅	⚠️ if the chunk says "merchandise"
"Lead time for SKU-4411"	❌	✅
"What does clause 7.3 say?"	❌	✅
"Can we send things back?"	✅	❌ no shared terms
"Error E-4471 on the packing line"	❌	✅
"What's our policy on late deliveries?"	✅	✅
"Restocking fee"	✅	✅

The two failure columns are complementary, which is the entire argument for hybrid retrieval. Vectors fail on exact tokens; keywords fail on paraphrase. Neither failure mode is a subset of the other.

Setting it up in Postgres

-- A generated column, so it cannot drift from the content.
alter table chunks add column search_vector tsvector
  generated always as (
    setweight(to_tsvector('english', coalesce(array_to_string(heading_path, ' '), '')), 'A')
    || setweight(to_tsvector('english', content), 'B')
  ) stored;

create index chunks_search on chunks using gin (search_vector);

Weighting the heading path above the content is the keyword-side equivalent of chapter 32's embedding prefix, and it buys the same thing: a chunk under Returns → Damaged goods ranks higher for "damaged goods returns" than one that mentions the phrase in passing.

-- The query. ts_rank_cd is the cover-density variant: it rewards query terms
-- appearing close together, which matters for multi-word questions.
select c.id, c.content, c.heading_path,
       ts_rank_cd(c.search_vector, query, 32) as score
  from chunks c
  join documents d on d.id = c.document_id,
       websearch_to_tsquery('english', $1) as query
 where c.search_vector @@ query
   and d.deleted_at is null
   and d.classification = any($2)
   and (d.account_id is null or d.account_id = any($3))
 order by score desc
 limit $4;

websearch_to_tsquery rather than plainto_tsquery is deliberate: it handles quoted phrases and -exclusions the way people expect from a search box, and it never throws on malformed input — plainto_tsquery is fine but to_tsquery will error on a stray operator, which is a crash on a user's question mark.

⚠️ Postgres full-text search is not true BM25. ts_rank_cd lacks proper IDF weighting, which is precisely the property that makes rare tokens dominate. For most corpora it is close enough; where it is not, the options are a BM25 extension or a dedicated search engine, and you should measure before assuming you need either.

-- Approximate IDF yourself if ts_rank_cd under-weights rare terms. Crude,
-- effective, and worth measuring against the test set before adopting.
select c.id,
       ts_rank_cd(c.search_vector, query, 32)
         * (1 + ln(1 + (select count(*) from chunks)::float
                      / greatest(df.document_frequency, 1))) as score
  from ...

Tokenisation is where it goes wrong

The single most common keyword-search defect, and it is invisible until you look.

-- Look at what the tokeniser actually did. Do this before anything else.
select to_tsvector('english', 'Lead time for SKU-4411 is 6 weeks per clause 7.3');
-- 'clause':8 '6':6 '7.3':9 'lead':1 'sku':4 'sku-4411':3 '4411':5 'time':2 'week':7

Good news: sku-4411, sku and 4411 are all indexed, so a query for SKU-4411 matches. But:

select to_tsvector('english', 'Part number 4411-A revision 2');
-- '2':5 '4411':3 '4411-a':2 'a':4 'number':1 'revis':4

-- And the English stemmer does things you may not want on technical text.
select to_tsvector('english', 'The packing line was operating under specification');
-- 'line':3 'oper':5 'pack':2 'specif':7   -- 'specification' -> 'specif'

Stemming conflates specification with specific, which is usually fine and occasionally bad. The fix where it matters is a second column with the simple configuration — no stemming, no stop words — searched alongside.

alter table chunks add column search_exact tsvector
  generated always as (to_tsvector('simple', content)) stored;
create index chunks_search_exact on chunks using gin (search_exact);

// Route by the question. An exact-token question goes to the exact column.
const hasCode = /\b(?:[A-Z]{2,}-[\w.]+|\d{4,}|§\s?\d+(?:\.\d+)*)\b/.test(question)
const column = hasCode ? 'search_exact' : 'search_vector'
const dictionary = hasCode ? 'simple' : 'english'

-- Domain terms the stemmer gets wrong go in a synonym dictionary, which is
-- a file rather than code and is owned by whoever knows the vocabulary.
-- $SHAREDIR/tsearch_data/domain.syn:
--   merchandise goods
--   rma returns
--   sku part
create text search dictionary domain_syn (template = synonym, synonyms = domain);

The synonym dictionary is the fix for the one keyword failure that matters on Atlas's corpus: policies say "merchandise", people say "goods". Two lines in a file, and the keyword side stops missing a whole category.

What it costs

Essentially nothing, which is the other half of the argument.

	Vector	Keyword
Index build, 44,387 chunks	2 min	8 s
Index size	273 MB	31 MB
Query latency p50	9 ms	4 ms
Cost per query	One embedding call	Zero
Cost per ingest	One embedding call per chunk	Zero

No API call, no model, no network. A keyword query is a local index lookup, which also makes it the sensible degradation path when the embedding provider is unavailable.

// Graceful degradation, per chapter 135. Keyword-only is much better than
// nothing, and the answer can say so.
try {
  return await hybridRetrieve(question, options)
} catch (error) {
  if (!(error instanceof EmbeddingUnavailable)) throw error
  logger.warn('retrieval.degraded', { reason: 'embedding provider unavailable' })
  return { ...(await keywordRetrieve(question, options)), degraded: true }
}

Measuring it alone

Before combining, score keyword search by itself on the same test set. The comparison is the input to chapter 43's weighting.

npm run evals:retrieval -- --method keyword

Subset	Vector	Keyword
All 34 cases	0.91	0.76
Contains an exact token (11)	0.61	1.00
Paraphrase, no shared terms (7)	1.00	0.29
Everything else (16)	0.94	0.88

Three readings:

Keyword is worse overall and perfect where vectors fail. 0.61 to 1.00 on the exact-token subset is the entire gap identified in the previous chapter, closed.
Keyword collapses on paraphrase, as expected. 0.29 on questions sharing no vocabulary with the passage.
They are close on the ordinary middle, 0.94 against 0.88 — which means on most questions either works, and the combination is cheap insurance rather than a compromise.

⚠️ Do not read "0.91 against 0.76" as "vectors win". The aggregate is the least informative number in that table. The subsets are the decision.

The failure mode: the stop word that was the question

A rep asks: "Can we ship to a PO box?"

The English configuration removes can, we, to, a as stop words. po is two characters and is also removed by some configurations. What remains is ship and box, and the query returns chunks about shipping boxes — packaging specifications.

The passage that answers it says "Deliveries cannot be made to PO Boxes", and it is not returned, because its distinguishing term was discarded at index time as well.

-- Diagnose before guessing. This shows exactly what survived.
select websearch_to_tsquery('english', 'Can we ship to a PO box');
-- 'ship' & 'box'

select websearch_to_tsquery('simple', 'Can we ship to a PO box');
-- 'can' & 'we' & 'ship' & 'to' & 'a' & 'po' & 'box'

Three responses, and the right one is all three:

// 1. Always search the exact column too, not only for detected codes.
//    The `simple` configuration keeps 'po'.

// 2. Hybrid. The vector side handles 'PO box' as a concept and does not
//    care about stop words at all. This is chapter 43 earning its place
//    on a question that looks like plain English.

// 3. A domain stop-word exception list, for terms that are stop words in
//    English and meaningful in your corpus.

-- A custom configuration that keeps the short domain tokens.
create text search configuration domain_english (copy = english);
alter text search configuration domain_english
  alter mapping for asciiword with domain_syn, english_stem;

The general lesson, and it is the reason this chapter sits before hybrid rather than after:

Every retrieval method discards information. Keyword search discards stop words and paraphrase; vector search discards exact tokens and negation. Neither is a safety net for the other by accident — you have to run both deliberately.

Key points

BM25-style ranking wins on rare exact tokens, which is precisely where vectors fail hardest.
The two failure modes are complementary, not nested. That is the argument for hybrid.
Weight the heading path above content in the tsvector, as the embedding side does.
Use websearch_to_tsquery — it handles phrases and never throws on a user's punctuation.
Postgres full-text is not true BM25; it lacks proper IDF. Measure before deciding that matters.
Inspect the tokeniser output before anything else. Stemming and stop words are where this silently breaks.
Keep a second simple column with no stemming for exact-token questions, and a synonym dictionary for domain vocabulary.
Keyword search costs nothing: no API call, 8-second builds, 31 MB, 4ms queries. It is also the degradation path when embeddings are unavailable.
Score it alone and read the subsets, not the aggregate. 0.61 to 1.00 on exact tokens is the finding; 0.91 to 0.76 overall is noise.

Next: hybrid retrieval — fusing two ranked lists whose scores are not comparable.