Chapter 43 · Retrieval

Hybrid retrieval, and fusing two ranked lists

Two methods, two incomparable score scales. Reciprocal rank fusion, why it beats weighted score blending, and the one query that runs both in Postgres.

11 min read

Chapter 41 showed vectors failing on exact tokens. Chapter 42 showed keywords failing on paraphrase. Both methods are already built. This chapter combines them, and it is the largest single recall gain in this section.

Why you cannot just add the scores

Cosine similarity runs 0 to 1 with most of the mass between 0.1 and 0.6. ts_rank_cd is unbounded, typically 0.01 to 0.9, and its distribution depends on query length and term rarity.

// ❌ The obvious approach, and it is wrong.
const combined = 0.5 * vectorScore + 0.5 * keywordScore

Three problems, and the third is the one that is hard to fix:

Different ranges. A keyword score of 0.4 and a vector score of 0.4 mean entirely different things.
Different distributions. Normalising to [0,1] per query helps and does not make them comparable, because the shapes differ.
Query-dependent scaling. A rare term produces high keyword scores for that query. The same weights behave differently question to question, so tuning them on 34 cases tunes to those questions' vocabulary.

Reciprocal rank fusion

Discard the scores. Use only the positions.

RRF(d) = Σ  1 / (k + rank(d, list))
        lists

// The whole algorithm. k=60 is the conventional constant and is not critical.
export function reciprocalRankFusion(
  lists: Array<{ id: number; rank: number }[]>,
  k = 60,
): Map<number, number> {
  const scores = new Map<number, number>()
  for (const list of lists) {
    for (const { id, rank } of list) {
      scores.set(id, (scores.get(id) ?? 0) + 1 / (k + rank))
    }
  }
  return scores
}

Four properties make this the right default.

Scale-free. Only positions are used, so no normalisation and no comparable units are needed.

Robust to one list being wrong. A document ranked first in one list and absent from the other still scores 1/61, which is respectable. A document ranked well in both beats it. That is exactly the behaviour wanted: agreement is evidence, and a single strong signal still counts.

k controls how much the top of each list dominates. At k=60, rank 1 scores 0.0164 and rank 10 scores 0.0143 — a gentle slope that lets a second list's opinion matter. A small k makes rank 1 dominate; a large one flattens everything.

Nothing to tune per corpus. Weighted blending has two parameters fitted to your test set and prone to overfitting on 34 cases. RRF has one constant that is insensitive.

One query, both methods

-- Both searches, fused, with the permission filter applied identically to
-- each. ⚠️ The filter must be inside BOTH branches — a filter applied after
-- the union is the post-filtering pattern chapter 13 rejected.
with
  vector_hits as (
    select c.id, row_number() over (order by c.embedding <=> $1) as rank
      from chunks c join documents d on d.id = c.document_id
     where d.deleted_at is null
       and d.classification = any($3)
       and (d.account_id is null or d.account_id = any($4))
       and c.retrievable
     order by c.embedding <=> $1
     limit $5
  ),
  keyword_hits as (
    select c.id, row_number() over (order by ts_rank_cd(c.search_vector, q, 32) desc) as rank
      from chunks c join documents d on d.id = c.document_id,
           websearch_to_tsquery('english', $2) as q
     where c.search_vector @@ q
       and d.deleted_at is null
       and d.classification = any($3)
       and (d.account_id is null or d.account_id = any($4))
       and c.retrievable
     order by ts_rank_cd(c.search_vector, q, 32) desc
     limit $5
  ),
  fused as (
    select id, sum(1.0 / (60 + rank)) as rrf_score
      from (select id, rank from vector_hits
            union all
            select id, rank from keyword_hits) both
     group by id
  )
select c.id, c.content, c.heading_path, c.kind,
       d.title, d.source, d.source_updated_at, d.parse_warnings,
       f.rrf_score,
       row_number() over (partition by c.document_id order by f.rrf_score desc) as rank_in_doc
  from fused f
  join chunks c on c.id = f.id
  join documents d on d.id = c.document_id
 order by f.rrf_score desc
 limit $6;

One round trip, both indexes, permissions enforced in both branches. The candidate limit $5 is typically 50 per method; the final limit is 12.

// The per-document cap from chapter 41 applies after fusion, not before —
// capping each list separately loses documents that are strong in both.
const results = rows
  .filter((r) => r.rank_in_doc <= config.retrieval.perDocumentCap)
  .slice(0, limit)

The score floor problem

RRF scores have no absolute meaning, so chapter 41's floor cannot be applied to them. A chunk ranked first by both methods always scores 0.0328, whether it is a perfect match or the least-bad chunk in a corpus with no answer.

// Keep the floor on the underlying vector score, carried through fusion.
// The question "is anything here relevant at all?" is answered by the vector
// side; the question "what order?" is answered by RRF.
const relevant = rows.filter((r) =>
  r.vector_score >= config.retrieval.floor || r.keyword_rank <= 3,
)

The keyword_rank <= 3 disjunction matters. A question like "what does clause 7.3 say?" has a low vector score — clause numbers barely embed — and a keyword rank of 1. Applying the vector floor alone would discard exactly the results keyword search exists to find.

-- Which means fusion has to carry both scores through, not just the rank.
select id, sum(1.0 / (60 + rank)) as rrf_score,
       max(vector_score) as vector_score,
       min(keyword_rank) as keyword_rank
  from (...) both
 group by id

What it buys

Subset	Vector	Keyword	Hybrid (RRF)
All 34 cases	0.91	0.76	0.97
Exact token (11)	0.61	1.00	1.00
Paraphrase (7)	1.00	0.29	1.00
Other (16)	0.94	0.88	0.94
precision@12	0.11	0.09	0.12
p50 latency	9 ms	4 ms	14 ms

Recall@12 from 0.91 to 0.97, for five milliseconds and no API call. That is the largest gain available in this section and the cheapest.

Three things worth reading carefully:

Hybrid matches the better method on every subset. It does not average them. RRF's robustness property means a strong signal from one list survives the other list's indifference.

Precision barely moves, 0.11 to 0.12. Hybrid is a recall technique. Precision is reranking.

The two remaining failures are the known scanned-contract exclusions, which are not retrieval failures at all. Against the indexable corpus, recall is 1.00 — which is a signal to stop working on candidate generation entirely.

When weighted blending is worth it

RRF treats both lists equally. Occasionally that is wrong — a corpus of error codes and part numbers where keyword should dominate, or free prose where it should not.

// Weighted RRF: still rank-based, so still scale-free, with one knob.
export function weightedRrf(
  lists: Array<{ list: { id: number; rank: number }[]; weight: number }>,
  k = 60,
): Map<number, number> {
  const scores = new Map<number, number>()
  for (const { list, weight } of lists) {
    for (const { id, rank } of list) {
      scores.set(id, (scores.get(id) ?? 0) + weight / (k + rank))
    }
  }
  return scores
}

⚠️ Measure before adopting it, and expect it not to help. On Atlas's test set, sweeping the vector weight from 0.3 to 0.7 moved recall@12 by less than 0.01 at every point — the fusion is already doing the work, and a weight fitted to 34 cases is fitted to those cases' vocabulary mix rather than to the corpus.

// The sweep, which is five minutes and usually ends the discussion.
for (const w of [0.3, 0.4, 0.5, 0.6, 0.7]) {
  const s = await scoreRetrieval(cases, { fusion: { vectorWeight: w } })
  console.log(w, round(s.recall12), round(s.recall5))
}

Route by query type instead, if anything. A question matching the code pattern from chapter 41 genuinely is a keyword question, and that is a classification rather than a fitted weight.

The failure mode: one list silently empty

Hybrid degrades gracefully in exactly one direction and not the other.

If the keyword side returns nothing — no term survived tokenisation, as in chapter 42's PO box case — fusion is vector-only. Recall drops to 0.91 on those questions and nothing indicates it.

If the vector side returns nothing — the embedding call timed out — fusion is keyword-only, recall drops to 0.76, and again nothing indicates it.

Both look like a working hybrid search that is quietly half a system.

// Log the composition of every result set. Two counters, and they turn an
// invisible degradation into a dashboard line.
const composition = {
  fromVectorOnly: results.filter((r) => !r.keyword_rank).length,
  fromKeywordOnly: results.filter((r) => !r.vector_score).length,
  fromBoth: results.filter((r) => r.keyword_rank && r.vector_score).length,
}
logger.info('retrieval.composition', composition)

// And an assertion, because a method contributing nothing across many
// queries is broken rather than unlucky.
if (composition.fromBoth === 0 && results.length > 3) {
  logger.warn('retrieval.single_method', {
    question, note: 'No chunk was found by both methods. One side may be failing.',
  })
}

-- The aggregate view, worth checking weekly.
select date_trunc('day', asked_at) as day,
       round(avg(from_vector_only)) as vector_only,
       round(avg(from_keyword_only)) as keyword_only,
       round(avg(from_both)) as both
  from question_log
 where asked_at > now() - interval '14 days'
 group by 1 order by 1;

A day where keyword_only and both go to zero is an index that was dropped by a migration, or a tokeniser configuration change. It is visible in one row and invisible in every quality metric, because recall of 0.91 still looks fine.

A hybrid system has two ways to become a single-method system, and neither raises an error. Count the contributions.

Key points

Vector and keyword scores are not comparable: different ranges, different distributions, query-dependent scaling.
Reciprocal rank fusion uses only positions. Scale-free, robust to one list being wrong, nothing to tune per corpus.
Run both branches in one query with the permission filter inside each. A filter after the union is post-filtering.
Apply the per-document cap after fusion, not to each list.
RRF scores have no absolute meaning, so keep the relevance floor on the underlying vector score — with an exception for high keyword ranks, or you discard exactly what keyword search is for.
Hybrid matched the better method on every subset: recall 0.91 to 0.97 for five milliseconds and no API call.
Precision is unchanged. Hybrid is a recall technique.
Weighted fusion usually does not help and overfits a small test set. Sweep it once; prefer routing by query type.
Count how many results came from each method. A hybrid system can become single-method in two directions, silently.

Next: reranking — the precision technique, and when its latency is worth paying.