Chapter 47 · Retrieval

Multi-hop questions, and recognising one

Seven of Atlas's thirty-four questions cannot be answered by one retrieval. How to detect them before trying, the two shapes they come in, and why only one of them needs the agent loop.

11 min read

Chapter 18 marked seven of thirty-four questions as multi-source and deferred them. This chapter is where they come due.

A multi-hop question is one where the second thing you need to look up depends on what the first lookup returned. That dependency is the whole difficulty, and it is what no amount of retrieval tuning addresses.

Two shapes, and they are not equally hard

Parallel questions decompose into independent lookups and are easy: fan out, fuse, answer. No loop needed, no model between the steps.

Sequential questions cannot be parallelised, because the second query's parameters come from the first result. This is the only case that genuinely needs the agent loop, and it is section 7.

Of Atlas's seven: five are parallel, two are sequential. That ratio is typical and it matters — most "we need an agent" questions are parallel, and a fan-out answers them at a fraction of the cost.

Detecting one before attempting it

Cheap signals first, a model only when they are ambiguous.

// src/retrieve/multi-hop.ts
const CONJUNCTION = /\b(and|also|as well as|along with|plus)\b/i
const COMPARISON = /\b(compare|versus|vs\.?|difference between|which of|better than)\b/i
const DEPENDENCY = /\b(being applied|consistent with|match(es|ing)?|in line with|according to .+ (is|are)|whether .+ (matches|meets))\b/i
const AGGREGATION = /\b(all|every|each|across|total|average|how many .+ (have|are))\b/i

export function classifyHops(question: string): HopKind {
  if (DEPENDENCY.test(question)) return 'sequential'
  if (AGGREGATION.test(question)) return 'aggregate'
  if (COMPARISON.test(question) || CONJUNCTION.test(question)) return 'parallel'
  return 'single'
}

⚠️ Patterns misclassify. "What are the returns and refunds policies?" contains and and is a single question about one document. The classifier is a routing hint, and the routing has to be safe when it is wrong.

// Safe when wrong: attempt single-hop first, and escalate on evidence.
export async function answerWithHops(question: string, scope: Scope) {
  const kind = classifyHops(question)

  if (kind === 'single' || kind === 'parallel') {
    // Single retrieval first. It is 200ms and it frequently succeeds even
    // on questions the classifier called parallel.
    const direct = await retrieveAndRerank(question, { scope, limit: 6 })
    if (await coversQuestion(question, direct)) return direct
  }

  return kind === 'sequential'
    ? sequentialHops(question, scope)
    : parallelHops(question, scope)
}

Trying the cheap path first is the pattern from chapter 2, applied here: most requests should not pay for the expensive route, and the cheap route failing is cheap.

Parallel: decompose and fuse

// Decomposition is one model call, and it is worth it because the sub-questions
// are reusable — they are logged, cached, and become eval cases.
export async function parallelHops(question: string, scope: Scope) {
  const subQuestions = await decompose(question)   // 2-4, capped

  const lists = await Promise.all(
    subQuestions.map((sub) => retrieveAndRerank(sub, { scope, limit: 4 })),
  )

  // ⚠️ Not RRF here. Each sub-question's results must stay labelled with the
  // sub-question, so generation can answer each part rather than producing
  // one blended answer from a merged list.
  return {
    kind: 'parallel' as const,
    parts: subQuestions.map((sub, i) => ({ question: sub, results: lists[i].results })),
  }
}

<!-- src/generate/prompts/decompose.md -->
Break this question into the independent questions it contains.

Rules:
- Only split where the parts can be answered separately. If answering one
  part requires the answer to another, do not split: return the original.
- At most four parts.
- Keep every identifier exactly as written.
- If the question is already single, return it unchanged.

Question: {{question}}

The instruction not to split dependent parts is what keeps the classifier honest: a decomposer that splits a sequential question produces two lookups neither of which can be done, and the resulting answer is two halves that do not meet.

<!-- And generation receives the structure, not a flat list. -->
The question has several parts. Answer each, with its own citations.

Part 1: {{part1_question}}
Sources:
{{part1_sources}}

Part 2: {{part2_question}}
Sources:
{{part2_sources}}

Sequential: the case that needs a loop

// Bounded, explicit, and it stops rather than looping. Section 7 generalises
// this; here it is deliberately a fixed two-step with a hard limit.
export async function sequentialHops(question: string, scope: Scope, maxHops = 3) {
  const trace: Hop[] = []
  let context: Retrieved[] = []

  for (let hop = 0; hop < maxHops; hop++) {
    // What is the next thing to look up, given what we have?
    const next = await planNextHop(question, context, trace)
    if (next.kind === 'sufficient') break
    if (next.kind === 'stuck') {
      // ⚠️ An honest stop. A loop that cannot proceed must say so rather
      // than retrieving something adjacent and answering from it.
      return { kind: 'incomplete', trace, reason: next.reason }
    }

    const results = next.tool
      ? await callTool(next.tool, next.arguments, scope)   // section 6
      : await retrieveAndRerank(next.query, { scope, limit: 4 })

    trace.push({ hop, action: next, resultCount: results.length })
    context = [...context, ...results]
  }

  return { kind: 'sequential', context, trace }
}

Two properties are doing the work.

The trace is a first-class return value, not a log line. A sequential answer that cannot be explained step by step is not checkable, and chapter 89 is about making it legible to a human.

stuck is a distinct outcome. The common failure is a loop that, unable to find the second thing, retrieves something topically near it and answers confidently from a partial chain.

The aggregate case, which is neither

"How many of our accounts have payment terms longer than Net 30?"

This is not multi-hop. It is a question retrieval cannot answer at all, for the reason chapter 15 gave: retrieval returns a sample, and a count over a sample is a confident wrong number.

// Detect and refuse, or route to a tool that can actually count.
if (classifyHops(question) === 'aggregate') {
  const tool = findAggregateTool(question)      // section 6
  if (tool) return callTool(tool, extractArguments(question), scope)

  return {
    kind: 'cannot_answer',
    message:
      'This needs a count across all accounts, which I cannot do from ' +
      'document search. The contracts team can run this report.',
  }
}

Routing an aggregate question to a human with the reason is the correct answer, and it is the known-gap pattern applied at retrieval time.

What it costs and buys

Atlas's seven multi-hop questions:

Approach	Answered correctly	p50 latency	Model calls
Single retrieval	2 of 7	214 ms	1
Parallel decomposition	5 of 7	580 ms	2
Sequential loop (max 3 hops)	7 of 7	2,100 ms	3–5
Sequential on all 34 questions	34 of 34	2,100 ms	3–5

The last row is the trap. Running every question through the loop answers everything and costs ten times as much on the 27 questions that did not need it — which is chapter 2's routing argument with a number attached.

// So the router is the economic decision, and its errors are asymmetric.
// Misrouting a multi-hop question to single retrieval costs one cheap
// failed attempt. Misrouting a single question to the loop costs 10x,
// every time, forever.

The failure mode: the chain that drifted

Atlas is asked: "Is the Henderson discount being applied correctly to their current orders?"

Hop 1 retrieves the Henderson contract: 12% above $25,000. Correct. Hop 2 queries Henderson's current orders: three orders, discounts of 12%, 12%, 0%. Hop 3 — the model, seeing a 0%, retrieves the policy on discount exceptions. The answer: "Mostly yes. One order has no discount, which is permitted under the exceptions policy for expedited freight."

Plausible, well-cited, and wrong. The 0% order is below $25,000, so no discount applies and nothing is exceptional. The exceptions policy was retrieved because the model went looking for an explanation and retrieval obligingly provided the nearest one.

Three defences, and they compound.

// 1. A hop must be justified by the question, not by a surprising result.
//    Pass the ORIGINAL question into the planner at every hop and require
//    the plan to reference it.
const next = await planNextHop(question, context, trace)
if (!next.justification?.includes('to answer the original question')) {
  logger.warn('hop.unjustified', { hop, plan: next })
}

// 2. Check the arithmetic before explaining it. A deterministic check beats
//    a retrieved explanation, and it is three lines.
for (const order of orders) {
  const expected = order.total >= contract.threshold ? contract.discount : 0
  if (order.discount !== expected) discrepancies.push({ order, expected })
}
// ⚠️ Only retrieve an explanation for a discrepancy that actually exists.
if (discrepancies.length === 0) return { kind: 'no_discrepancy', orders }

// 3. Cap the hops low. maxHops = 3 is not a performance limit; it is the
//    number of chances the chain has to wander. Most real sequential
//    questions are two hops.

The second defence is the general lesson, and it applies well beyond this question:

Where a step can be computed, compute it. A model asked to reason over retrieved facts will find a story that fits; a comparison in code either matches or does not.

Key points

A multi-hop question is one where the second lookup depends on the first result. That dependency is the difficulty.
Parallel and sequential are different problems. Most "we need an agent" questions are parallel and need a fan-out, not a loop.
Detect with cheap patterns, route safely, and try single retrieval first — it often succeeds on questions the classifier called parallel.
Keep sub-question results labelled rather than fusing them, so generation answers each part.
Instruct the decomposer never to split dependent parts. Two unanswerable halves are worse than one hard question.
Make the trace a return value, and make stuck a distinct outcome from an answer.
Aggregate questions are not multi-hop. They are unanswerable by retrieval; route to a tool or to a human with the reason.
Running every question through the loop works and costs ten times as much on the ones that did not need it. The router's errors are asymmetric.
A chain drifts by explaining a result that needed no explanation. Where a step can be computed, compute it.

Next: evaluating retrieval on its own — the harness, formalised, before the model is allowed back in.