Chapter 138 · Production

Scaling ingestion as the corpus grows

The pipeline nobody watches until it breaks. What changes between 10,000 and 1,000,000 chunks, the backlog that is invisible until it is a week deep, and the three metrics that predict trouble.

8 min read

Chapter 23 built the pipeline. Chapter 34 made it incremental. This chapter is what happens eighteen months later, when the corpus is thirty times larger and the pipeline is the least-watched component in a system that depends on it completely.

The framing worth holding: ingestion is the only part of the system with no user waiting on it, which is exactly why it fails silently and why its failures surface as quality problems somewhere else.

What changes with scale

	10k chunks	100k	1M
Full re-ingest	20 min	3.5 h	1.5 days
Full re-embed cost	$2	$19	$190
Vector index build	40 s	9 min	2.5 h
A bad parse affects	A few answers	A source	Everything, invisibly
Backlog visible in	Minutes	Hours	Days

The bottom two rows are the ones that change the engineering, not the top three. At a million chunks, a full re-ingest stops being a recovery option — which means every operation has to be incremental and resumable, and a pipeline designed around "just run it again" has quietly lost its recovery path.

The backlog nobody sees

// The failure Atlas actually had. Ingestion is scheduled, succeeds, and
// processes less than arrives. Nothing errors. Nothing alerts. Freshness
// degrades by hours per day until someone notices an answer is a week
// stale.
const SYMPTOM = {
  jobStatus: 'success, every run',
  errorRate: 0,
  documentsProcessed: 'rising, which looks healthy',
  queueDepth: 'rising faster',     // nobody was plotting this
}

-- The query that would have caught it on day two. Arrival rate against
-- processing rate, which is the only pair that matters.
select date_trunc('day', d) as day,
       sum(discovered) as arrived,
       sum(indexed) as processed,
       sum(sum(discovered) - sum(indexed)) over (order by date_trunc('day', d)) as backlog
  from ingestion_daily
 where d > now() - interval '30 days'
 group by 1 order by 1;

// And the metric that makes it legible to someone who is not watching:
// not queue depth, but how OLD the oldest unprocessed item is.
const oldestPending = await queue.oldestEnqueuedAt()
metrics.gauge('ingestion.lag_seconds', Date.now() - oldestPending)
// Alert at 4 hours. Depth is a number nobody has intuition for; lag is
// "the newest documents are four hours old", which anyone can act on.

Atlas's backlog reached nine days before a rep asked why a contract signed last week was not findable. Nine days of a green dashboard.

Making every stage resumable

// At scale, any stage can be interrupted: a deploy, a provider outage,
// an OOM on one enormous PDF. Each stage commits its own progress.
interface IngestionCheckpoint {
  sourceId: string
  cursor: string              // provider-specific: a timestamp, a page token, an etag
  lastCompletedAt: Date
  documentsProcessed: number
}

// The property that matters: a stage that dies mid-source resumes from
// the cursor rather than from the start. Without it, one source that
// consistently fails 80% of the way through never completes, and the
// retry does the same 80% of the work every time.

Atlas had exactly that: a document store whose pagination timed out near the end of a large folder, retried nightly, and re-processed 11,000 chunks a night for three weeks without ever reaching the last 400.

Partitioning the work

// One queue for everything means one enormous PDF blocks the small
// documents behind it. Partition by expected cost.
const LANES = {
  fast:  { maxBytes: 100_000,    concurrency: 20 },   // most things
  slow:  { maxBytes: 20_000_000, concurrency: 4 },    // scanned PDFs, big spreadsheets
  huge:  { maxBytes: Infinity,   concurrency: 1 },    // quarantine; often a mistake
}

// The huge lane is mostly a detector. A 400 MB file in a document store
// is usually a backup, an export or a video someone renamed, and
// processing it is rarely the right answer. Atlas logs and skips above
// a threshold, with a weekly report of what was skipped.

Re-embedding without a window

// Chapter 137's migration, at scale, with no maintenance window. The
// technique is the same as any online index rebuild.
const REEMBED = [
  '1. New index, written to alongside the old by the same pipeline.',
  '2. Backfill in cost-ordered batches, checkpointed, at off-peak.',
  '3. Both indexes receive new documents throughout.',
  '4. Shadow queries compare; chapter 49’s test set gates the flip.',
  '5. Flip by flag. Delete the old after a week.',
]

// Step 2's cost ordering matters at a million chunks: backfill the
// documents that are actually retrieved first, so the new index is
// useful long before it is complete. Atlas's top 8% of documents by
// retrieval frequency covered 71% of queries.
const order = await documentsByRetrievalFrequency({ since: '90 days' })

Deletions, at scale

// Chapter 35's orphan problem gets worse, because at a million chunks
// nobody notices 400 orphans and they answer questions.
const RECONCILE = `
Weekly: list every document id in the source of truth, compare against
distinct document ids in the index, delete the difference.
`

// ⚠️ With the guard that has to exist, because the failure mode is
// catastrophic and the cause is mundane.
const missing = indexed.size - inSource.size
if (missing > 0.02 * indexed.size) {
  throw new Error(`reconciliation would delete ${missing} chunks (>2%) — refusing; check the source listing`)
}

A source API that returns an empty or truncated listing on an auth failure, run against a naive reconciler, deletes the corpus. The 2% guard turns that into an alert. Atlas's guard has fired twice, both times a paginated listing that silently stopped early.

Three metrics that predict trouble

const PREDICTIVE = [
  { m: 'ingestion lag (oldest pending)', alert: '>4 h',
    why: 'the backlog metric. Everything else is downstream of it.' },
  { m: 'parse failure rate by source', alert: '>2% or any step change',
    why: 'a source changed format. Chapter 27 — silent, and it degrades one source only.' },
  { m: 'chunks per document, by source', alert: 'median shifts >30%',
    why: 'the best single indicator that extraction broke. A PDF parser falling back to a worse mode halves it.' },
]

The third is the most useful and the least obvious. A change in chunks-per-document is visible within one ingestion cycle and precedes any quality signal by days — it caught a library upgrade that silently stopped extracting table content, where the first downstream symptom would have been answers quietly missing figures.

The weekly ingestion report

// One page, to a human, because this subsystem has no user.
const REPORT = [
  'documents added, updated, deleted, by source',
  'parse failures, with three examples each',
  'chunks per document, this week against the trailing median',
  'ingestion lag, max over the week',
  'oversized files skipped',
  'reconciliation deletions, with the guard margin',
]

Atlas's report is read in about two minutes and has caught four problems before they reached an answer. The argument for it is simply that a subsystem with no user needs a reader, and a dashboard nobody opens is not one.

Key points

Ingestion is the only component with no user waiting on it, which is why it fails silently and surfaces as a quality problem elsewhere.
At a million chunks a full re-ingest is 1.5 days and stops being a recovery option. Every operation must be incremental and resumable.
Atlas's backlog reached nine days with a green dashboard, because nobody plotted arrival rate against processing rate.
Alert on ingestion lag — how old the oldest unprocessed item is — not queue depth. Lag is actionable; depth is a number nobody has intuition for.
Checkpoint a cursor per source. Atlas re-processed 11,000 chunks nightly for three weeks without ever finishing a folder.
Partition by expected cost so one enormous file does not block everything. Treat a huge-lane arrival as a detector, not a job.
Re-embed online, backfilling in retrieval-frequency order. Atlas's top 8% of documents covered 71% of queries.
Guard reconciliation with a maximum-deletion threshold. A truncated source listing run against a naive reconciler deletes the corpus; Atlas's 2% guard has fired twice.
Watch chunks per document by source. It is the earliest indicator that extraction broke and precedes any quality signal by days.
Send a weekly report to a human. A subsystem with no user needs a reader.

Next: on-call for an agent: what to alert on — turning all of this into pages somebody will answer.