Chapter 22 · The data layer

A dry run, by hand

Answer three real questions with no system at all. Two hours of manual work finds the structural problems that would otherwise surface in week nine of ingestion.

11 min read

The last chapter of section 2, and the cheapest hour in the whole series.

You have a registry that validates, a permission model, and 34 questions. Before building any ingestion, answer three of those questions by hand — as a person, reading the actual sources, writing down every step.

The reason this works is that a system can only automate a process that exists. If you cannot answer the question manually from the sources in the registry, no amount of retrieval quality will produce the answer, and you will spend weeks concluding that your chunking is bad.

The method

Pick three questions deliberately:

One you are confident about — a single-source policy lookup.
One that is permission-sensitive — where two people should get different answers.
One you suspect is harder than it looks.

For each, open the sources, find the answer, and record the trace.

# evals/dry-run.md — a trace per question. Write the steps, not just the answer.
question: "How long do customers have to return damaged goods?"
asker: inside_sales

steps:
  - "Opened the document store, searched 'returns'. 4 results."
  - "Two are the Returns and Credits Policy: v3 (2024) and v4 (2026)."
  - "v3 is not marked superseded anywhere. Opened both to compare."
  - "v4, section 4.2: 30 days from delivery, damage reported within 48 hours."
  - "v3, section 4.1: 14 days. Different number, no indication it is obsolete."

answer: "30 days from delivery, if damage is reported within 48 hours."

findings:
  - id: F1
    severity: high
    finding: >
      Superseded documents are present in the store with no supersession marker.
      Indexing both means retrieval will return 14 days and 30 days with similar
      scores, and the model will pick one or report a conflict.
    fix: "Detect and exclude superseded versions at ingest. New work."

That finding is from a two-minute exercise, and it is the kind of thing that would otherwise appear as "retrieval quality is bad" in week nine.

What the three questions found for Atlas

Question 1: the returns window

The trace above. One high-severity finding: two live versions of the same policy, with no supersession marker. The inventory had noted "two of which are superseded and still present" — the dry run turned that note into a blocker with a known fix.

// The fix, specified by the dry run rather than discovered later.
// Version detection at ingest: same title, different version, keep the newest
// and soft-delete the rest.
export function selectCurrentVersions(docs: SourceDocument[]): SourceDocument[] {
  const byTitle = new Map<string, SourceDocument[]>()
  for (const doc of docs) {
    const key = normaliseTitle(doc.title)     // strips 'v3', '(2024)', 'final'
    byTitle.set(key, [...(byTitle.get(key) ?? []), doc])
  }

  return [...byTitle.values()].map((group) => {
    if (group.length === 1) return group[0]
    const sorted = group.sort((a, b) => +b.effectiveDate - +a.effectiveDate)
    // ⚠️ Report, do not silently drop. A group of 5 where you expected 2 is a
    // naming problem in normaliseTitle, and it would otherwise hide 3 documents.
    logger.warn('ingest.superseded', {
      title: group[0].title,
      kept: sorted[0].id,
      superseded: sorted.slice(1).map((d) => d.id),
    })
    return sorted[0]
  })
}

Question 2: the permission-sensitive one

"What discount does the Henderson account get?" — asked as a rep who owns the account, and as a rep who does not.

The trace found something the permission model had missed. The answer is in two places: the contract, which is account-scoped, and the pricing workbook, which is classified financial. A rep who owns Henderson can see the contract and therefore the discount. A rep who does not own it cannot see the contract — but the workbook holds the standard tier discount for the region, and if the asker's own accounts put them in that region, they can see the standard rate.

findings:
  - id: F2
    severity: medium
    finding: >
      The answer is reachable through two sources with different classifications.
      A rep who cannot see the Henderson contract can see the standard regional
      tier, and may reasonably infer Henderson's rate from it.
    decision: >
      Acceptable. The standard tier is not account-specific information, and
      inferring from it is no different from what a rep can do today. Recorded
      so that it is a decision rather than an oversight.
    eval_case: "Added leak-017: standard tier is answerable; the contract's
      negotiated rate is not."

This is the most valuable category of finding in the whole exercise: not a bug, but an inference path that needed a decision. It would never have surfaced from reading the permission model, because the permission model is per-source and the problem is across sources. Chapter 16 called this recombination; the dry run is how you find an instance of it.

Question 3: the one you suspect is harder

"What are the payment terms for the Henderson account?"

steps:
  - "Found the Henderson contract. Section 7: Net 30."
  - "Noticed a second file: 'Henderson - Amendment 2 - 2025-09.pdf'."
  - "The amendment changes payment terms to Net 45."
  - "The amendment has no reference to the parent contract in its metadata."
  - "Searched for other amendments: 340 files matching 'Amendment' across the store."
  - "Checked 10. None carry a parent reference in metadata. The parent is only
     identifiable from the filename and the text."

answer: "Net 45, per Amendment 2 dated September 2025."

findings:
  - id: F3
    severity: critical
    finding: >
      Contract amendments are separate documents with no structural link to the
      contract they amend. Retrieving the contract alone gives Net 30, which is
      wrong, confidently, with a correct citation.
    scale: "340 amendment files across 3,100 contracts."
    fix: >
      Link amendments to parents at ingest, from the filename convention, and
      carry a 'superseded_by' marker on the amended clause. Where a parent
      cannot be resolved, exclude the contract from the index rather than
      indexing a document known to be partly obsolete.

Critical, and it is the single most important thing this section produced. A system built without it answers payment-term questions wrongly for an unknown fraction of 3,100 accounts, in a way that passes every test unless the test set happens to include an amended account.

The fix is not better retrieval. Even perfect retrieval returning both documents leaves the model to decide which is current, from two chunks that each state a number with equal confidence. The fix is at ingestion: the amended clause carries a marker, so the chunk itself says it was superseded.

// Amendments resolved at ingest. The marker goes INTO the chunk text, because
// that is the only place the model is guaranteed to see it.
export function applyAmendments(contract: ParsedDocument, amendments: ParsedDocument[]) {
  for (const amendment of amendments.sort((a, b) => +a.date - +b.date)) {
    for (const clause of amendment.clauses) {
      const original = contract.clauses.find((c) => c.number === clause.number)
      if (!original) continue
      original.content =
        `⚠️ SUPERSEDED by Amendment ${amendment.number} (${fmtDate(amendment.date)}). ` +
        `Current terms: ${clause.content}\n\n` +
        `Original text: ${original.content}`
    }
  }
  return contract
}

Three questions, four findings

#	Finding	Severity	Effect on the plan
F1	Superseded policy versions present, unmarked	High	New ingest step: version detection
F2	Discount inferable through a second classification	Medium	A recorded decision, plus one eval case
F3	340 amendments with no link to their parent	Critical	New ingest step; contracts blocked until it exists
F4	Document titles are inconsistent enough to break version grouping	Medium	`normaliseTitle` needs real test cases

Two hours. Three new ingestion requirements, one architectural decision, one eval case. All of them would otherwise have been found during section 3 or, worse, after launch by somebody quoting Net 30.

Every one of these findings is about the data rather than about the system. That is why the exercise works before any system exists, and why doing it later finds less.

The failure mode: doing it with the wrong person

The one way this exercise fails is being run by the engineer alone.

An engineer answering "what are the payment terms for Henderson" reads the contract, finds Net 30, and stops — because the contract is clearly the source of payment terms and the answer was clearly found. The amendment is only noticed by somebody who knows amendments exist.

Run it with the person who answers these questions today, sitting beside you, and ask them to narrate. They will do things that are not in any documentation: check a second folder, remember that this account is unusual, know that the policy document is out of date. Those habits are the findings.

Three prompts that produce the useful material:

"Where else might this be?"            — finds the shadow copies
"How do you know that's current?"      — finds the supersession problems
"Is there an account where this is different?" — finds the exceptions

The third question is the one that found the amendments.

Section 2 is done

What you have now: an inventory, four source shapes routed correctly, a permission predicate per source verified against the source systems, exclusions with reasons, residency and retention recorded, a question-to-source matrix with a measured coverage prediction, freshness decisions with sync intervals, a registry that validates in the build, and four findings that change the ingestion design before it is written.

Section 3 builds the ingestion pipeline. It is a long section, and almost every chapter in it is shorter than it would have been because of the decisions made here — most of all because the registry means a new source is an entry rather than a refactor.

Key points

Answer three real questions by hand, before building anything. A system can only automate a process that exists.
Pick one confident question, one permission-sensitive question, and one you suspect is harder than it looks.
Record the steps, not just the answer. The findings are in the steps.
Expect findings about the data rather than the system. That is why the exercise is cheaper now than later.
Cross-source inference paths show up here and nowhere else, because permission models are written per source.
A document whose content is superseded needs the marker inside the chunk text. Retrieval returning both versions does not help.
Run it with the person who does the job today, and ask where else it might be, how they know it is current, and which account is different.

Next, and the start of section 3: the ingestion pipeline end to end — built against a registry that already knows what every source is, and against four findings that would otherwise have arrived late.