Chapter 35 · Ingestion

Deletions, and the chunk that outlives its document

The half of sync nobody tests. Five ways a document leaves the corpus, only one of which produces an event, and why a deleted document is a worse failure than a stale one.

12 min read

The previous chapter kept the index current for documents that exist. This one handles the ones that stop existing, and it is consistently the weakest part of systems like this because additions are visible and removals are not.

The asymmetry is worth stating plainly: a stale document gives an out-of-date answer; a deleted document gives an answer the company has decided should not exist. The second is a compliance problem, sometimes a legal one, and it is silent.

Five ways a document leaves

Only the first produces an event, and the other four are the ones that bite.

#	How	Detected by
1	Deleted in the source	A delete event, if the source emits one
2	Moved out of the ingested scope	Reconciliation
3	Permission revoked for the integration account	Looks identical to #2
4	Stopped qualifying — reopened, field cleared	A qualification check
5	Superseded by a new version	Supersession detection

Three and two being indistinguishable is the hard part of this chapter, and it is why the default on an absence cannot be deletion.

Soft delete, always

-- From the schema in chapter 8, and now the reasoning.
update documents set deleted_at = now() where id = $1;

Never a hard delete, for four reasons:

Reversibility. A permission blip removes 400 contracts from discovery. With a soft delete that is an update to undo; with a hard delete it is a re-ingest and re-embed.
Audit. "Was this document retrievable on the 14th?" is answerable.
Debuggability. A cited answer referencing a document that no longer exists is traceable.
Cost. Resurrection costs one update; re-embedding 40 chunks costs a model call.

-- The partial index means soft-deleted rows cost nothing at query time.
create index documents_live on documents (classification, account_id)
  where deleted_at is null;

⚠️ Soft delete is not an exception to erasure. A right-to-erasure request requires the content gone, not flagged, and that is a hard delete with a tombstone:

-- Erasure: remove content, keep the fact that something was removed.
update documents
   set deleted_at = now(), title = '[erased]', erased_at = now()
 where id = $1;
delete from chunks where document_id = $1;     -- content and vectors, gone

Two different operations with two different names. Conflating them is how a soft-delete-everything policy quietly fails an erasure obligation.

Reconciliation, and the absence problem

// src/ingest/reconcile.ts
export async function reconcile(source: SourceDefinition, adapter: SourceAdapter) {
  const inSource = new Set<string>()
  for await (const doc of adapter.discover()) inSource.add(doc.sourceRef)

  // ⚠️ Before trusting an absence, sanity-check the discovery itself. An
  // empty or badly truncated listing must never be read as "everything was
  // deleted", which is the single most destructive bug available here.
  const { rows: live } = await db.query(
    `select source_ref from documents where source = $1 and deleted_at is null`,
    [source.id],
  )

  const missing = live.map((r) => r.source_ref).filter((ref) => !inSource.has(ref))
  const missingRatio = missing.length / Math.max(live.length, 1)

  if (missingRatio > (source.expectedChurn ?? 0.05)) {
    throw new IngestAnomaly(
      `${source.id}: ${missing.length} of ${live.length} documents absent ` +
      `(${(missingRatio * 100).toFixed(0)}%). Refusing to delete. ` +
      `Check integration permissions and scope before re-running.`,
    )
  }

  for (const ref of missing) await softDelete(source.id, ref)
  return { checked: live.length, deleted: missing.length }
}

The refusal threshold is the whole safety design. An integration that loses access to a folder sees documents vanish, and a reconciliation that trusts absence will soft-delete 400 contracts in one run — correctly, according to its inputs, and catastrophically.

Refusing and alerting leaves 400 slightly stale contracts, which is the better failure by a wide margin.

Grace periods for intermittent sources

An API that returns a partial page under load produces a false absence. Requiring an absence to persist converts a flake into a delay.

// Two consecutive reconciliations must agree before anything is deleted.
export async function markAbsent(sourceId: string, ref: string) {
  const { rows } = await db.query(
    `insert into absences (source, source_ref, first_seen_absent, runs)
     values ($1, $2, now(), 1)
     on conflict (source, source_ref)
       do update set runs = absences.runs + 1
     returning runs, first_seen_absent`,
    [sourceId, ref],
  )

  const { runs, first_seen_absent } = rows[0]
  const ageHours = (Date.now() - +first_seen_absent) / 3_600_000

  // Both conditions: twice, and over an hour apart. A single bad run cannot
  // satisfy the second even if retried immediately.
  if (runs >= 2 && ageHours >= 1) {
    await softDelete(sourceId, ref)
    await db.query(`delete from absences where source = $1 and source_ref = $2`, [sourceId, ref])
  }
}

/** A document that reappears clears its absence record. */
export async function markPresent(sourceId: string, ref: string) {
  await db.query(`delete from absences where source = $1 and source_ref = $2`, [sourceId, ref])
}

For a source where deletion matters more than flake tolerance — anything with a retention obligation — set the grace period to zero and accept the re-ingests. That is a per-source decision belonging in the registry.

Supersession: the delete with no delete

The fifth path, and the one the dry run found twice: a policy with two live versions, and 340 contract amendments. Nothing is deleted; the old document is simply no longer true.

// Supersession is a relationship, not a removal. Both documents exist; one
// is no longer retrievable.
export async function markSuperseded(oldId: string, newId: string) {
  await db.query(
    `update documents set superseded_by = $2, deleted_at = now() where id = $1`,
    [oldId, newId],
  )
}

-- Which gives a useful answer to "why can I not find the old one?"
select d.id, d.title, s.title as superseded_by_title, d.deleted_at
  from documents d
  left join documents s on s.id = d.superseded_by
 where d.superseded_by is not null
 order by d.deleted_at desc;

Keeping the link rather than just the flag is what makes a support question answerable in one query. "That version was replaced by the 2026 policy on the 3rd" is a better answer than "it is not in the index".

Orphans, in both directions

-- 1. Chunks whose document is gone. The foreign key cascade should prevent
--    this; check anyway, because a direct delete during an incident will not
--    have gone through the cascade.
select c.document_id, count(*) as orphaned_chunks
  from chunks c
  left join documents d on d.id = c.document_id
 where d.id is null
 group by 1;

-- 2. Documents with no chunks. An interrupted ingest, or a parse that
--    produced nothing. Invisible to retrieval and counted as indexed.
select d.id, d.title, d.indexed_at
  from documents d
  left join chunks c on c.document_id = d.id
 where d.deleted_at is null
 group by d.id, d.title, d.indexed_at
having count(c.id) = 0;

The second query catches something no other check does. A document row with zero chunks passes every count in the ingest report's report — it was discovered, fetched, parsed and stored — and contributes nothing to any answer. For Atlas's first full run this found eleven documents: nine empty scans below the OCR threshold whose skip had not propagated, and two interrupted by a restart.

// Run both after every reconciliation, and treat a non-zero result as a
// failure rather than a statistic.
const orphans = await findOrphans()
if (orphans.chunksWithoutDocument > 0 || orphans.documentsWithoutChunks > 0) {
  throw new IngestAnomaly(
    `orphans: ${orphans.chunksWithoutDocument} chunks without a document, ` +
    `${orphans.documentsWithoutChunks} documents without chunks`,
  )
}

Testing deletion, which is the actual point of this chapter

Deletion is untested in most systems because it requires setting up a document and then removing it, which is more work than asserting that a search returns something.

// evals/deletion.test.ts — four cases, and they are worth the hour.
test('a deleted document is not retrievable', async () => {
  const id = await ingestFixture('returns-policy.md')
  expect(await retrieveIds('damaged goods return window')).toContain(id)

  await softDelete('policy_docs', 'returns-policy')
  expect(await retrieveIds('damaged goods return window')).not.toContain(id)
})

test('a superseded version is not retrievable and its replacement is', async () => {
  const v3 = await ingestFixture('returns-policy-v3.md')
  const v4 = await ingestFixture('returns-policy-v4.md')
  await markSuperseded(v3, v4)

  const ids = await retrieveIds('how long to return damaged goods')
  expect(ids).not.toContain(v3)
  expect(ids).toContain(v4)
})

test('reconciliation refuses a mass absence', async () => {
  const adapter = stubAdapter({ discovers: [] })     // source returns nothing
  await expect(reconcile(policySource, adapter)).rejects.toThrow(/Refusing to delete/)
})

test('erasure removes content, not just the flag', async () => {
  const id = await ingestFixture('returns-policy.md')
  await erase(id)
  const { rowCount } = await db.query(`select 1 from chunks where document_id = $1`, [id])
  expect(rowCount).toBe(0)
})

The third test is the most valuable of the four. It is the one that would have prevented the 400-contract incident, and it takes four lines because the stub adapter already exists for every other adapter test.

The failure mode: the answer from a document that was withdrawn

A specification is withdrawn by the supplier — a safety tolerance was wrong. It is deleted from the document store that afternoon.

Atlas's nightly sync has no delete event for this source, because the store does not emit them. Reconciliation runs weekly. For up to six days, the withdrawn specification is retrievable, cited, and gives the wrong tolerance to anyone who asks.

Everything behaved as designed. The design was wrong, in one parameter: reconciliation frequency was set by its cost rather than by its consequence.

// The fix is per-source, like freshness in chapter 19 — and for exactly the
// same reason. Deletion latency is a tolerance, not a schedule.
{
  id: 'product_specs',
  reconcileIntervalHours: 24,    // withdrawn specs are a safety matter
  expectedChurn: 0.02,
}
{
  id: 'ticket_notes',
  reconcileIntervalHours: 168,   // a stale closed ticket harms nobody
  expectedChurn: 0.01,
}

Set reconciliation frequency from the harm of a stale deletion, not from the cost of running it. They are different numbers and only one of them is about the business.

And where the harm is high and the source emits nothing, the honest options are to reconcile that source daily or to add a cheap existence check on the documents actually cited in answers — which costs one request per cited document and closes the window to zero for the documents anyone is relying on.

Key points

A stale document is out of date; a deleted one is an answer the company decided should not exist. The second is silent.
Five ways a document leaves and only one emits an event. Permission revocation and scope changes are indistinguishable.
Soft delete always, with a partial index. Erasure is a separate, harder operation with a tombstone.
Reconciliation must refuse to act on a mass absence. An empty listing read as "everything was deleted" is the most destructive bug available.
Use a grace period of two runs an hour apart for flaky sources, and zero where deletion matters more.
Supersession keeps the link, not just the flag, so "why can I not find it?" is one query.
Check orphans in both directions. Documents with zero chunks pass every ingest count and contribute nothing.
Test deletion, supersession, mass-absence refusal and erasure. Four tests, one hour.
Set reconciliation frequency from the harm of a stale deletion, not from its cost.

Next, and the last chapter of this section: running the first full ingest — and reading the numbers it produces.