Chapter 36 · Ingestion

Running the first full ingest

The numbers a run produces, which of them are health and which are quality, and the twelve-line report that turns a successful ingest into a list of things to fix.

12 min read

The last chapter of section 3. Everything is built: adapters, parsers, chunking, metadata, sync and deletion.

Running it is the easy part. Reading what it produced is this chapter, and it matters because a run that completes successfully tells you almost nothing about whether the corpus is any good.

Before pressing go

Four things, in order. Each one has saved a long run from being wasted.

# 1. Dry run: discovery only. Confirms auth, scope and counts without cost.
npm run ingest -- --dry-run
# policy_docs      340 discovered
# product_specs  1,900 discovered   (1,140 native, 760 scanned)
# contracts      3,100 discovered
# pricing_wb         1 discovered
# tickets       41,000 discovered

# 2. One document per source, end to end. Inspect the chunks by hand.
npm run ingest -- --source policy_docs --limit 1 --inspect

# 3. A 50-document sample per source, scored against the test set. If recall
#    is 0.3 on a sample, it will be 0.3 on the whole corpus an hour later.
npm run ingest -- --sample 50 && npm run evals:retrieval

-- 4. Capacity. 45,000 rows with a 1,536-dimension vector is ~300 MB of
--    vectors alone, before text and indexes. Check before, not during.
select pg_size_pretty(pg_database_size(current_database()));

The third is the one people skip and the one that pays. A full run that produces a bad corpus costs an hour and the cost of re-embedding; a sample that produces a bad corpus costs four minutes.

The report

Twelve lines, printed at the end of every run. This is the artefact worth building, because it is what makes the numbers readable rather than available.

ingest complete — 2026-11-03 02:41, 63 min

source          discovered  ingested  skipped  failed   chunks   warnings
policy_docs            340       338        2       0    4,120          0
product_specs        1,900     1,138      760       2   18,340        214
contracts            3,100     3,047       53       0   21,880         31
pricing_workbook         1         1        0       0       47          0
                     -----     -----    -----   -----   ------      -----
                     5,341     4,524      815       2   44,387        245

skips by reason                      warnings by kind
  ocr_confidence_below_threshold 760   ocr_confidence_0.80_to_0.92  214
  no_account_reference            53   no_tables_in_spec_document    24
  unclassified_path                2   no_heading_path                7

orphans: 0 chunks without a document · 11 documents without chunks  ❌

Everything in it comes from counters the pipeline already maintains. The value is entirely in presenting them together, where the relationships are visible.

Health numbers against quality numbers

The distinction that makes the report useful.

Health — did the machinery work	Quality — is the corpus good
discovered, ingested, failed	recall@12 on the test set
orphans	mean chunk size and its spread
wall time, cost	documents with no heading path
skips matching expectation	duplicate chunk rate

A run can be perfect on the left and bad on the right. That is the normal case for a first ingest, and the mistake is to read a green health report as a finished corpus.

Reading the Atlas report

Five observations, in the order a person actually notices them.

760 skips in product_specs is expected — it matches the triage count from chapter 27 exactly. An expected skip count that matches is a healthy signal; the same number unexplained would be the headline problem.

2 failures in product_specs is not expected. Failures are things that were not anticipated, and two is small enough to read individually and large enough to be a pattern.

npm run ingest:skips -- --source product_specs --kind failed
# SPEC-2291.pdf  parse  mean word length 31.4 — likely garbled
# SPEC-3307.pdf  parse  space ratio 0.021 — spacing lost

Both are the parse checks working. Both are subset-font failures, which means a fix to the parser would recover them — and because the hash includes the parser version, that fix re-ingests exactly these two.

53 contracts with no account reference is the classification rule refusing to guess. Correct behaviour and a real coverage gap: 53 contracts are not retrievable by anyone. Worth a query to the CRM owner about why those accounts are unlinked.

11 documents without chunks is a failure, not a statistic. It is in red because chapter 35 made it one.

44,387 chunks against the 45,000 predicted in chapter 7's sizing. A prediction made before any code, within 1.5%. Worth noting because it means the sizing method works and can be trusted for the next source.

The quality checks that run after

-- 1. Chunk size distribution. A long tail of tiny chunks means the chunker
--    is fragmenting; a spike at the maximum means it is truncating.
select width_bucket(tokens, 0, 1200, 12) * 100 as bucket, count(*)
  from chunks group by 1 order by 1;

-- 2. Duplicate content. Boilerplate — headers, disclaimers, standard clauses —
--    repeated across thousands of documents dilutes retrieval badly.
select md5(content) as h, count(*), min(content) as example
  from chunks group by 1 having count(*) > 20 order by 2 desc limit 20;

-- 3. Structure coverage. Documents whose chunks have no heading path parsed
--    badly, whatever else the report said.
select d.source,
       count(*) filter (where cardinality(c.heading_path) = 0) as no_path,
       count(*) as total,
       round(100.0 * count(*) filter (where cardinality(c.heading_path) = 0) / count(*)) as pct
  from chunks c join documents d on d.id = c.document_id
 group by 1 order by 4 desc;

The duplicate query is the most consistently surprising. On Atlas's contracts it found a standard confidentiality clause appearing in 2,847 chunks — identical text, 2,847 vectors, all equally close to any question about confidentiality.

// Boilerplate suppression: keep one canonical copy, link the rest. The
// threshold is corpus-dependent; measure before choosing it.
export async function suppressBoilerplate(minOccurrences = 20) {
  const { rows } = await db.query(
    `select md5(content) as hash, array_agg(id order by id) as ids
       from chunks group by 1 having count(*) >= $1`,
    [minOccurrences],
  )
  for (const { ids } of rows) {
    // Keep the first; mark the rest non-retrievable but present, so a
    // document still reads completely when expanded by chapter 32's
    // section-prefix query.
    await db.query(`update chunks set retrievable = false where id = any($1)`, [ids.slice(1)])
  }
}

⚠️ Suppress rather than delete. The clause is genuinely part of each contract, and a reader expanding a contract's section should still see it. It just should not be 2,847 separate candidates in a ranked list.

Then score it

npm run evals:retrieval
# recall@12     0.89   (target 0.85)  ✅
# mrr           0.59                  ✅
# precision@12  0.11                  — expected, reranking in section 4
#
# failures by classification
#   retrieval:absent       3   ret-014, ctr-003, ctr-008
#   retrieval:ranked_low   1   spc-007

ctr-003 and ctr-008 are the two questions blocked by the scanned contracts exclusion — a known gap, correctly reflected, and not a retrieval defect. ret-014 and spc-007 are real and are section 4's starting work.

The three-bucket classification earning its keep here is the point: four failures that would otherwise be "quality is 0.89" are instead two known gaps and two specific things to investigate.

Make the report a gate

// scripts/ingest-gate.ts — runs after every ingest, in CI and on schedule.
export function gateIngest(report: IngestReport): string[] {
  const failures: string[] = []

  if (report.orphanDocuments > 0) failures.push(`${report.orphanDocuments} documents with no chunks`)
  if (report.failed > report.discovered * 0.01) failures.push(`${report.failed} failures (>1%)`)

  for (const source of report.sources) {
    // An unexplained skip rate. Expected skips are declared in the registry.
    const expected = source.expectedSkipRate ?? 0
    const actual = source.skipped / Math.max(source.discovered, 1)
    if (actual > expected + 0.05) {
      failures.push(`${source.id}: skipped ${(actual * 100).toFixed(0)}%, expected ${(expected * 100).toFixed(0)}%`)
    }
    // The absence check from chapter 23, once more, because this is where
    // somebody will actually read it.
    if (source.discovered === 0) failures.push(`${source.id}: discovered nothing`)
  }

  if (report.recallAt12 !== undefined && report.recallAt12 < report.recallTarget) {
    failures.push(`recall@12 ${report.recallAt12} below target ${report.recallTarget}`)
  }
  return failures
}

Declaring the expected skip rate per source in the registry is what makes this workable. 760 of 1,900 is 40%, and declaring it turns the largest number in the report from an alarm into a confirmation — while an unexplained 5% jump still fails.

Section 3 is done

The offline half exists. 44,387 chunks, permission-scoped, structure-preserving, incrementally synced, with deletions reconciled and a gated report.

Section 4 is retrieval, and it starts from a specific number: recall@12 of 0.89, precision@12 of 0.11, two named failing questions. That precision figure is the headroom — nine of every ten retrieved chunks are not useful, and closing that gap is most of what section 4 does.

The failure mode: the corpus that looks finished

The report is green. Recall is 0.89 against a target of 0.85. The instinct is to move on.

Two things are true that the report does not say.

Recall is measured on 34 questions written before anything existed. Real users ask differently — shorter, with internal jargon, with typos, and about things nobody anticipated. A corpus at 0.89 on the anticipated set can be considerably worse on the real one, and there is no way to know until people use it.

The test set has no questions about the 760 skipped documents, because they were skipped before anybody wrote questions about them. Coverage is measured over what was indexed, which quietly excludes what was not.

// So report coverage against the SOURCE, not against the index. One line,
// and it is the honest version of the headline number.
const indexable = report.discovered - report.expectedExclusions
const indexed = report.ingested
console.log(
  `corpus coverage: ${indexed} of ${indexable} indexable documents ` +
  `(${((indexed / indexable) * 100).toFixed(1)}%), ` +
  `${report.discovered - indexable} deliberately excluded`,
)
// corpus coverage: 4,524 of 5,286 indexable documents (85.6%), 55 deliberately excluded

85.6% is a different and more honest headline than 0.89 recall, and it is the number to put in front of whoever decides whether to ship the first slice.

Key points

Dry run, one document, a 50-document sample scored against the test set, then capacity. In that order.
Build the twelve-line report. The counters already exist; the value is in seeing them together.
Separate health numbers from quality numbers. A perfect health report says nothing about the corpus.
An expected skip count that matches the prediction is a healthy signal. Declare expected skip rates in the registry so unexplained ones still fail.
Failures are things you did not anticipate. Read all of them individually while there are few.
Run the three quality queries: size distribution, duplicate content, structure coverage. Boilerplate repeated thousands of times should be suppressed, not deleted.
Gate on the report, including recall against target.
Report coverage against the source, not against the index. Recall measured over what you indexed hides what you did not.

Next, and the start of section 4: retrieval as its own system, with its own quality bar — starting from recall 0.89, precision 0.11, and two named failing questions.