Chapter 18 · The data layer

Mapping sources to questions

Which data answers which asks, and the uncomfortable discovery that several questions have no source at all. The matrix that tells you what to build and what to stop promising.

10 min read

You have an inventory, a permission model, and decisions about what stays out. You also have 34 questions from chapter 7.

This chapter crosses the two. It takes two hours and it produces three things: a build order, a list of questions that cannot be answered, and the retrieval test set that section 4 tunes against, building on the hello-world path.

The matrix

Questions down the side, sources across the top. One row per question, and in each cell a marker for the relationship.

Question	Policy	Specs	Contracts	Pricing wb	Orders db	Tickets
Returns window for damaged goods	●
Lead time for SKU-4411		●
Henderson's agreed discount			●	○
Minimum order for north tier 3				●
Stock on hand, Dallas					●
Open tickets for an account						●
Is Henderson's discount being applied?			●	○	●
Why is Henderson past due?			○		●	●
What did we agree about expedited freight?			○

● means the source is sufficient on its own. ○ means it contributes but is not sufficient.

Three patterns are visible immediately, and each one tells you something different.

Pattern 1: single-source questions

One ● and nothing else. These are the majority — for Atlas, 24 of 34 — and they are the whole of the first slice.

They matter disproportionately because they are the only questions whose failures are unambiguous. If a single-source question is answered wrongly, the fault is in ingestion, chunking, retrieval or generation for one source, and the failure classification from chapter 10 will say which.

Build order falls out of counting them per source:

// Count ● per source. The ordering this produces is the build order, and it
// is usually different from the order people expect.
export function buildOrder(matrix: Cell[]) {
  const sufficient = new Map<string, number>()
  for (const cell of matrix) {
    if (cell.relation !== 'sufficient') continue
    sufficient.set(cell.source, (sufficient.get(cell.source) ?? 0) + 1)
  }
  return [...sufficient.entries()]
    .sort((a, b) => b[1] - a[1])
    .map(([source, count]) => ({ source, answersAlone: count }))
}

For Atlas: policy documents 12, product specifications 9, the pricing workbook 4, contracts 5, orders 3, tickets 1. The pricing workbook — one file, nobody mentioned it — answers more questions alone than the orders database.

Pattern 2: multi-source questions

Two or more marks. "Is Henderson's discount being applied to their current orders?" needs a contract term and a live query, and it is genuinely harder for a reason worth being precise about.

The dotted edge is the difficulty. The second step's input comes from the first step's output, which means it cannot be parallelised and cannot be a single retrieval. This is a multi-hop question, it is chapter 47, and it is the first thing that genuinely requires the agent loop rather than a workflow — because the number of hops is not known in advance.

Mark these in the matrix and then defer them. For Atlas, 7 of 34 are multi-source, and attempting them in the first slice means building the loop before retrieval is measured.

Pattern 3: the questions with no source

The uncomfortable column. "What did we agree about expedited freight for Henderson?" has one ○ and no ●, because the agreement was made in an email thread and sales email is excluded for the reasons in chapter 12.

For Atlas, 3 of 34 questions are unanswerable from in-scope sources. That number is the most valuable output of this exercise, and there are exactly four honest things to do with each one.

Response	When it applies
Create the source	The knowledge exists and should be written down. Best outcome.
Connect the excluded source	Only if the exclusion reason has actually been resolved.
Scope the question out	Tell users it is out of scope. Honest and unpopular.
Route to a human	The system says who to ask. Underrated.

The fourth is underrated because it converts a failure into a useful answer:

// A known gap, answered honestly and usefully. Better than an abstention and
// much better than an invented answer.
export const KNOWN_GAPS = [
  {
    pattern: /expedit\w+ (freight|shipping)|freight (terms|agreement)/i,
    response:
      'Freight arrangements are agreed per account and are not in the documents ' +
      'available to me. Sales Operations holds these — ask in the #sales-ops channel.',
  },
] as const

⚠️ Keep this list short and keep it in review. A growing gap list is a system quietly becoming a switchboard, and at that point the honest answer is to create the missing sources instead.

Marking the supporting chunks

The matrix tells you which source answers a question. The retrieval test set needs one level finer: which chunk.

# evals/retrieval.yaml — built after the first ingest, and the thing section 4
# tunes against. Document-and-heading rather than chunk ids, because chunk ids
# change on every re-chunk and this file should survive that.
- question_id: ret-001
  question: "How long do customers have to return damaged goods?"
  supporting:
    - document: "Returns and Credits Policy"
      heading: ["Returns", "Damaged goods"]
  # Chunks that look right and are not. A retrieval set without distractors
  # scores an unrealistically high number.
  distractors:
    - document: "Returns and Credits Policy"
      heading: ["Returns", "Late returns"]

Identifying by document and heading rather than by chunk ID is a deliberate choice. Chunk IDs are an artifact of a chunking decision, and chapter 31 changes that decision deliberately and measures the effect. A test set keyed to chunk IDs is invalidated by the experiment it exists to support.

The distractors field is what makes the numbers honest. Retrieval that ranks the right chunk first out of 45,000 is a weaker result than it sounds if the near-miss chunk was never a contender.

// Resolve document+heading to chunk ids at run time, so the file survives
// re-chunking and a missing resolution is reported rather than silently passing.
export async function resolveSupporting(ref: SupportingRef): Promise<number[]> {
  const { rows } = await db.query(
    `select c.id
       from chunks c join documents d on d.id = c.document_id
      where d.title = $1 and c.heading_path = $2 and d.deleted_at is null`,
    [ref.document, ref.heading],
  )
  if (rows.length === 0) {
    // Loudly. A silently empty supporting set makes every case pass.
    throw new Error(`unresolved supporting ref: ${ref.document} / ${ref.heading.join(' > ')}`)
  }
  return rows.map((r) => r.id)
}

That throw is worth the awkwardness it will occasionally cause. An unresolved reference that returns an empty array makes !supporting.some(...) false for every case, and a test suite that passes everything looks exactly like a test suite that is working.

Coverage, before anything is built

The matrix gives you a predicted coverage number, and comparing it to the target from chapter 10 is a useful reality check.

	Count	Share
Single-source, in-scope source	24	71%
Multi-source, deferred	7	21%
No source	3	9%

71% against a 70% target, with zero code written. That is not a comfortable margin — it says every one of the 24 has to work, and the first slice has no headroom.

Two honest responses: lower the target for version one, or move one deferred question into scope. What you should not do is assume the multi-source questions will be easy. They are chapter 47 for a reason.

The failure mode: the matrix nobody updates

The matrix is built, it informs the build order, and then it goes stale, because the questions change. People ask things nobody anticipated, and that is the system working.

A stale matrix is worse than none, because decisions keep getting made from it.

The fix is to generate it from the question log rather than maintaining it by hand:

-- Questions people actually ask, grouped by what answered them. After a month
-- of real usage this is strictly better than the matrix you built by hand.
select coalesce(d.source, 'no_source_found') as source,
       count(*) as questions,
       round(avg(case when a.abstained then 1 else 0 end) * 100) as abstain_pct
  from question_log q
  left join answer_sources a on a.question_id = q.id
  left join documents d on d.id = a.document_id
 where q.asked_at > now() - interval '30 days'
 group by 1
 order by questions desc;

The no_source_found row is the live version of the no-source column, and it is the most useful number in the system after launch. It tells you what to connect next, measured rather than guessed, and it is how the excluded sources get revisited on evidence instead of on enthusiasm.

Key points

Cross questions with sources. Two hours produces a build order, a gap list and the retrieval test set.
Count sufficient-on-their-own cells per source to get the build order. It is usually not the order people expect.
Multi-source questions have a dependency between steps, which is why they need the agent loop. Mark them and defer them.
Questions with no source are the most valuable output. Create the source, resolve the exclusion, scope it out, or route to a human.
Key the retrieval test set to document and heading, not chunk IDs, so it survives the re-chunking experiments it exists to support.
Include distractors, or the retrieval numbers flatter you.
Throw on an unresolved supporting reference. An empty supporting set makes every case pass.
After launch, generate the matrix from the question log. The no_source_found count is what decides the next source.

Next: freshness, source by source — how current each source has to be, and the rule that decides what may be indexed at all.