Chapter 46 · Retrieval

Query rewriting and expansion

The question people type is not the question to search with. Four rewriting techniques, what each costs in latency, and the one that is free and does most of the work.

12 min read

Everything so far has improved how the index is searched. This chapter improves what it is searched for.

Real questions are short, contextual, full of pronouns and internal jargon, and occasionally misspelled. Atlas's test set was written deliberately and is unrepresentative in exactly that way — a point chapter 36 flagged and this chapter addresses.

Four techniques, by cost

Technique	Latency	Gain on Atlas	When
Context stitching	0 ms	+0.04 recall	Always, when there is history
Acronym and synonym expansion	0 ms	+0.03 recall	Always, if you have a glossary
LLM rewriting	300–700 ms	+0.05 recall	Only on detected ambiguity
Hypothetical answer embedding	400–900 ms	+0.02 recall	Rarely worth it

The first two are free and do most of the work. The ordering matters because the instinct is to reach for the model first.

Context stitching

The most common real failure, and it costs nothing.

Rep: "What's the returns window for damaged goods?"
Atlas: "30 days from delivery, if reported within 48 hours. [1]"
Rep: "What about electronics?"

Searching for "What about electronics?" retrieves nothing useful. The question is "what is the returns window for damaged electronics?" and all of that context is in the previous turn.

// src/retrieve/rewrite.ts
// Resolve the question against recent turns without a model call.
const FOLLOW_UP = /^(what about|how about|and |what if|same for|for )\b/i
const PRONOUN = /\b(it|that|those|them|this|they)\b/i

export function stitchContext(question: string, history: Turn[]): string {
  if (history.length === 0) return question

  const isFollowUp = FOLLOW_UP.test(question.trim()) || question.split(/\s+/).length <= 4
  const hasPronoun = PRONOUN.test(question)
  if (!isFollowUp && !hasPronoun) return question

  // ⚠️ Prepend the previous QUESTION, not the previous answer. An answer is
  // long, is mostly retrieved text, and drags the query toward whatever was
  // retrieved last time rather than toward what is being asked now.
  const previous = history[history.length - 1].question
  return `${previous} ${question}`
}

Prepending the previous answer is the mistake to avoid, and it is tempting because the answer contains more words. It makes follow-ups retrieve the same chunks as the original question — which looks like coherence and is actually a loop.

// Two turns of history, capped, for a chain of follow-ups.
const context = history.slice(-2).map((t) => t.question).join(' ')

Acronym and synonym expansion

Internal vocabulary is the second free win, and it overlaps with — but does not replace — the synonym dictionary on the keyword side. That dictionary fixes keyword matching; this fixes the vector query too.

// A glossary, owned by the business and kept as data. Chapter 11's inventory
// interviews are where it comes from.
export const GLOSSARY: Record<string, string> = {
  rma: 'return merchandise authorisation RMA',
  cogs: 'cost of goods sold COGS',
  eta: 'estimated time of arrival ETA',
  'net 30': 'Net 30 payment terms thirty days',
  po: 'purchase order PO',
  bol: 'bill of lading BOL',
}

export function expandGlossary(question: string): string {
  let out = question
  for (const [term, expansion] of Object.entries(GLOSSARY)) {
    const pattern = new RegExp(`\\b${escapeRegex(term)}\\b`, 'gi')
    // Append rather than replace: keep the original token so the keyword
    // side still matches a document that uses the acronym.
    if (pattern.test(out)) out = `${out} ${expansion}`
  }
  return out
}

Appending rather than replacing is the detail. A document that says "RMA" and a question expanded to "return merchandise authorisation" would otherwise lose the exact match that keyword search was there to provide.

⚠️ Keep the glossary short and real. Fifty terms taken from how people actually speak is useful; three hundred scraped from a dictionary adds noise to every query.

LLM rewriting, gated

A model can resolve ambiguity that patterns cannot. It also costs 300–700ms on every question, which is a quarter of the latency budget, so it has to be gated.

// Detect ambiguity cheaply, and only then pay for the model.
export function needsRewrite(question: string, history: Turn[]): boolean {
  const words = question.trim().split(/\s+/).length
  if (words <= 3 && history.length > 0) return true          // "what about those?"
  if (PRONOUN.test(question) && history.length > 0) return true
  if (/\?.*\?/.test(question)) return true                   // two questions in one
  if (words > 40) return true                                // a pasted paragraph
  return false
}

<!-- src/generate/prompts/rewrite.md -->
Rewrite the user's question as a standalone search query.

Rules:
- Resolve pronouns and references using the conversation.
- Keep every specific term: product codes, clause numbers, account names,
  dates. Never paraphrase an identifier.
- Do not answer the question or add information not present.
- If the question is already standalone, return it unchanged.
- Output the query only, with no preamble.

Conversation:
{{history}}

Question: {{question}}

// And a guard on the output, because a rewriter that invents terms sends
// retrieval somewhere worse than the original question did.
export function validateRewrite(original: string, rewritten: string): string {
  // Identifiers in the original must survive.
  const identifiers = original.match(/\b(?:[A-Z]{2,}-[\w.]+|\d{4,}|§\s?\d+(?:\.\d+)*)\b/g) ?? []
  for (const id of identifiers) {
    if (!rewritten.includes(id)) {
      logger.warn('rewrite.dropped_identifier', { original, rewritten, id })
      return original
    }
  }
  // A rewrite much longer than the original has added material.
  if (rewritten.length > original.length * 4 + 60) {
    logger.warn('rewrite.too_long', { original, rewritten })
    return original
  }
  return rewritten
}

Falling back to the original on a suspicious rewrite is the right default. The rewriter is an optimisation, and an optimisation that can make things worse must be able to decline.

Multi-query expansion

Instead of one better query, several different ones — fused by the same RRF used across methods.

// Three phrasings, three retrievals, one fused list. The model call is the
// same cost as a rewrite; the retrievals are parallel and nearly free.
export async function multiQueryRetrieve(question: string, options: RetrieveOptions) {
  const variants = await generateVariants(question, 3)   // includes the original
  const lists = await Promise.all(
    variants.map((v) => hybridRetrieve(v, options)),
  )
  return reciprocalRankFusion(lists.map((l) => l.results))
}

Measured on Atlas: +0.02 recall over single-query hybrid, for one model call and three times the retrieval work. Marginal at recall 0.97, and genuinely useful earlier — when recall was 0.89 the same change bought +0.06.

Query expansion is worth most when recall is weakest. Measure it at your current recall, not at the recall you had when you read about it.

Hypothetical answer embedding

The technique from chapter 38's asymmetry discussion: have a model write a plausible answer, embed that, and search with it — comparing answer-shaped text to answer-shaped text.

// Elegant, and on Atlas it bought +0.02 recall for 400-900ms.
const hypothetical = await callModel(prompt('hypothetical', { question }))
const vector = await embed([hypothetical])

It is worth knowing about and it is rarely the right choice when the heading-path prefix from chapter 32 is already in place — both address the same asymmetry, and the free one addresses most of it.

The pipeline

export async function prepareQuery(question: string, history: Turn[]): Promise<PreparedQuery> {
  // Free, always.
  let text = stitchContext(question, history)
  text = expandGlossary(text)

  // Paid, only when needed.
  if (needsRewrite(question, history)) {
    const rewritten = await rewriteWithModel(text, history)
    text = validateRewrite(text, rewritten)
  }

  return {
    original: question,
    search: text,
    // ⚠️ Keep both. Generation must see what the person asked, not the
    // rewritten form — answering a rewritten question is how a subtle
    // misinterpretation becomes a confident wrong answer.
    rewritten: text !== question,
  }
}

That last comment is the most important line in the chapter. The rewrite exists to find chunks. The model answering must be given the user's own words, or a rewrite that drifted slightly produces an answer to a question nobody asked — and it will read as perfectly coherent.

What it buys

Configuration	recall@6	p50 latency	Model calls
Hybrid + rerank	0.97	214 ms	0
+ context stitching	0.97	214 ms	0
+ glossary	0.98	214 ms	0
+ gated LLM rewrite	0.99	252 ms avg	0.11 per question

On the deliberate test set the gains are small, because those questions are already standalone. The honest measurement is on real questions:

	Test set (34)	Real questions, first month (412)
Hybrid + rerank	0.97	0.81
+ stitching and glossary	0.98	0.91
+ gated rewrite	0.99	0.94

Ten points on real traffic against one point on the test set. That gap is the whole argument for chapter 49: a test set written in advance measures the system against anticipated language, and query rewriting is specifically the technique that handles unanticipated language.

The failure mode: the rewrite that answered the question

A rep asks: "is the Henderson discount still 12%?"

The rewriter, given a conversation in which 12% was mentioned, produces: "Henderson Industrial Supply volume discount 12% confirmation current rate".

Retrieval now searches for text containing 12%, finds the chunk that says 12%, and the model answers "yes, 12%". The actual current rate, after an amendment, is 9% — and the chunk saying so was never retrieved, because the query was steered toward the number in the question.

The rewriter did not lie. It embedded the questioner's assumption into the query, and the query found the assumption.

// Strip assertions from the search query. A question containing a candidate
// answer should search for the FIELD, not for the value.
const ASSERTION = /\b(?:still|confirm|is it|isn't it|right\?|correct\?)\b/i

export function neutraliseAssumptions(question: string): string {
  if (!ASSERTION.test(question)) return question
  // Remove the asserted value so retrieval finds whatever the value is.
  return question
    .replace(/\b\d+(?:\.\d+)?%/g, '')
    .replace(/\b(?:net\s*\d+|\d+\s*days?)\b/gi, '')
    .replace(ASSERTION, '')
    .replace(/\s+/g, ' ')
    .trim()
}
// "is the Henderson discount still 12%?" -> "the Henderson discount"

⚠️ Apply this to the search query only. The model still receives the original question, so it can say "the current rate is 9%, not 12%" — which is the answer the person actually needs and is only possible because retrieval was not steered.

This is a general hazard of query rewriting worth naming: a rewriter that uses the question's content to improve the query can encode the question's errors into it. Confirmation questions are the common case, and they are exactly the questions where being wrong matters.

Key points

Context stitching and glossary expansion are free and do most of the work. Reach for them before the model.
Prepend the previous question, never the previous answer. The answer drags the query toward what was already retrieved.
Append glossary expansions, do not replace terms, or you lose the exact keyword match.
Gate LLM rewriting on detected ambiguity. It costs a quarter of the latency budget on every question otherwise.
Validate the rewrite: identifiers must survive, length must be sane, and fall back to the original when they are not.
Multi-query expansion is worth most when recall is weakest. Measure at your current recall.
Keep the original question for generation. Answering a rewritten question is a confident answer to something nobody asked.
Gains are small on a deliberate test set and large on real traffic — ten points against one.
A rewriter can encode the asker's wrong assumption into the query. Neutralise asserted values in confirmation questions.

Next: multi-hop questions — recognising a question that cannot be answered by one retrieval.