Chapter 33 · Ingestion

Metadata: what to attach to every chunk

Seven fields, what each one is for, and which of them are not negotiable. The permission fields are the reason this chapter is in the ingestion section rather than the retrieval one.

11 min read

Metadata is the difference between a vector store and a retrieval system. A vector alone supports one operation — find things like this — and every other thing you need to do at query time is a metadata filter.

This chapter is in section 3 because metadata is written at ingestion and read at retrieval. Deciding it late means re-ingesting, which is the argument chapter 6 made about permissions and which applies to the whole set.

The seven fields

-- The full schema, consolidating chapters 8, 13, 17, 19 and 32.
create table documents (
  id             text primary key,
  source         text not null,
  source_ref     text not null,
  title          text not null,

  -- 1. Permission. Not negotiable. Chapter 13.
  classification text not null,
  account_id     text,

  -- 2. Freshness. Two distinct timestamps. Chapter 19.
  source_updated_at timestamptz not null,
  indexed_at        timestamptz not null default now(),

  -- 3. Provenance. Chapter 17 and 27.
  content_hash   text not null,
  parse_warnings text[] not null default '{}',

  -- 4. Lifecycle. Chapter 35.
  deleted_at     timestamptz,
  superseded_by  text references documents(id),

  -- 5. Residency and retention. Chapter 17.
  residency_region text not null,

  constraint documents_account_consistency
    check ((classification = 'account') = (account_id is not null))
);

create table chunks (
  id           bigserial primary key,
  document_id  text not null references documents(id) on delete cascade,

  -- 6. Position and structure. Chapters 30 and 32.
  ordinal      int not null,
  heading_path text[] not null default '{}',
  kind         text not null default 'prose',

  -- 7. The content, and its two forms.
  content      text not null,
  tokens       int not null,
  embedding    vector(1536) not null,

  -- Conversations only: per-chunk principals. Chapter 29.
  principals   text[],

  unique (document_id, ordinal)
);

The split between the two tables is the one structural decision here, and it is worth restating: permissions live on the document, never duplicated per chunk. A contract changing hands updates one row rather than forty, and the join is to a table in the same database, which is the only join the request path can afford.

The exception is principals on chunks, for conversations, and it is an exception precisely because email permissions are per message.

Which fields are filters and which are payload

A distinction that decides indexing, and which is easy to get wrong because every field looks equally important in a schema.

Field	Role	Needs an index
`classification`, `account_id`	Filter, every query	Yes, composite
`deleted_at`	Filter, every query	Partial index predicate
`source`	Filter, sometimes	Yes
`source_updated_at`	Payload, shown in answers	No
`heading_path`	Payload, and occasionally a filter	GIN, for test-set resolution
`content_hash`	Neither — used only by sync	Yes, for lookup by id
`parse_warnings`	Payload, shown as a caveat	No

-- The index that serves the hot path. Column order matters: the filter is
-- always on classification, usually on account_id, so classification leads.
create index documents_retrieval on documents (classification, account_id)
  where deleted_at is null;

Putting deleted_at is null in the index predicate rather than in the index columns keeps the index small and means the planner uses it for the only query shape that exists.

Attaching it, in one place

The temptation is to let each adapter set its own metadata. Resist it: seven fields set in five adapters is thirty-five opportunities for one to be missed, and a missing classification is a document nobody can see or a document everybody can.

// src/ingest/transform.ts — every document passes through here, and nothing
// writes to the documents table except this function's output.
export function toDocumentRow(
  parsed: ParsedDocument,
  source: SourceDefinition,
): DocumentRow {
  const classification = typeof source.classification === 'function'
    ? source.classification(parsed)        // derived from source metadata
    : source.classification

  const accountId = classification === 'account'
    ? requireAccountId(parsed)             // throws SkipDocument if absent
    : null

  return {
    id: `${source.id}:${parsed.sourceRef}`,     // namespaced, so ids cannot collide
    source: source.id,
    sourceRef: parsed.sourceRef,
    title: parsed.title,
    classification,
    accountId,
    sourceUpdatedAt: parsed.updatedAt,
    contentHash: hashBlocks(parsed.blocks),
    parseWarnings: parsed.parseWarnings,
    residencyRegion: source.residencyRegion,
    supersededBy: null,
  }
}

Namespacing the id with the source prefix is a small thing that prevents a real collision: ticket 4471 and contract 4471 are different documents, and a bare numeric id from two sources will eventually overwrite one with the other.

// And the test that keeps the single path single — the same shape as the
// unscoped-query test in chapter 13, for the same reason.
test('only transform writes to documents', async () => {
  const files = await readAll('src/**/*.ts')
  const writers = files.filter(
    (f) => /insert into documents/i.test(f.text) && !f.path.endsWith('transform.ts'),
  )
  expect(writers.map((f) => f.path)).toEqual([])
})

Hashing content, not bytes

content_hash gates re-embedding, which is the only expensive operation. Hashing the right thing matters.

// ❌ Hashing raw bytes re-embeds on every metadata touch: a re-save with no
//    edit, a PDF producer version bump, a changed modification date in the
//    file's own metadata.
const wrong = createHash('sha256').update(bytes).digest('hex')

// ✅ Hash the parsed content. Two files that parse identically ARE identical
//    for every purpose this system has.
export function hashBlocks(blocks: Block[]): string {
  const canonical = blocks
    .map((b) => `${b.type}:${blockText(b).replace(/\s+/g, ' ').trim()}`)
    .join('\n')
  return createHash('sha256').update(canonical).digest('hex')
}

Normalising whitespace inside the hash is what makes this robust to reformatting. A policy document re-exported with different line wrapping is the same document, and re-embedding 45,000 chunks because somebody opened a file in a different editor is a real and avoidable cost.

⚠️ The hash covers content only, so a metadata-only change — a reclassification, an account reassignment — must bypass it. Chapter 34 handles that explicitly, because it is exactly the case where "nothing changed" is wrong in the direction that matters.

Metadata you should not attach

Three that look useful and are not, and one that is actively harmful.

A tag list from the source system. Tempting as a filter, and almost always inconsistent — urgent, Urgent, URGENT, urgnet. If it is going to be a filter it has to be normalised and validated, and if it is not a filter it belongs in the content where retrieval can match it.

A summary generated at ingest. Costs a model call per document, goes stale silently when the document changes under an unchanged hash, and is usually worse than the first chunk. If you want a summary, generate it at query time from the chunks you actually retrieved.

A relevance or quality score computed at ingest. Relevance is relative to a question, and there is no question at ingest time.

A copy of the asker-side permission data — a materialised list of which users may see this document. This is the one that is actively harmful, for the reason chapter 14 gave: it turns every permission change into an index rewrite, and the window between the change and the rewrite is an access-control gap.

Using metadata at retrieval

The payoff, and it is more than the permission filter.

select c.id, c.content, c.heading_path, c.kind,
       d.title, d.source, d.source_updated_at, d.parse_warnings,
       1 - (c.embedding <=> $1) as score
  from chunks c
  join documents d on d.id = c.document_id
 where d.deleted_at is null
   and d.classification = any($2)
   and (d.account_id is null or d.account_id = any($3))
   -- Optional, from query analysis in chapter 45:
   and ($4::text[] is null or d.source = any($4))
 order by c.embedding <=> $1
 limit $5;

Every selected column has a consumer: heading_path renders the citation, kind lets a table be formatted as a table, source_updated_at produces the freshness stamp, and parse_warnings becomes the OCR caveat from chapter 27. None of it is available later, so selecting it now costs nothing and fetching it in a second query costs a round trip per answer.

The failure mode: the nullable permission column

The schema starts with classification text — nullable, because the first source is policy documents and everything is public_internal, and a not null with no default is awkward during the first migration.

It works. Months pass. A new adapter is added by someone who did not read chapter 14, and it does not set classification. Fifty documents land with classification = null.

Now trace the filter: d.classification = any($2) with a null classification evaluates to null, not true — so those fifty documents are invisible. That is the safe direction, and it is also why nobody notices for three months.

Then someone debugging a "document not found" complaint discovers the nulls and fixes the filter rather than the data:

-- ❌ The fix that looks right at 6pm and is a leak.
and (d.classification is null or d.classification = any($2))

Every null-classification document is now visible to everyone, and the commit message says "fix missing documents in retrieval".

-- The prevention, which is one line in the original migration.
alter table documents alter column classification set not null;
alter table documents add constraint documents_classification_valid
  check (classification in ('public_internal', 'account', 'financial'));

Make the invalid state unrepresentable at the database, not at the query. A constraint fails at insert, where the person who caused it is looking. A null fails at retrieval, months later, in front of someone whose fastest fix is a leak.

Key points

Metadata is what makes a vector store a retrieval system. It is written at ingestion and read at retrieval, so deciding it late means re-ingesting.
Permissions live on the document, never duplicated per chunk. The only exception is per-message principals for conversations.
Separate filters from payload. Only filters need indexes, and deleted_at is null belongs in the index predicate.
One function writes document rows, enforced by a test. Seven fields across five adapters is thirty-five chances to miss one.
Namespace document ids by source, or two sources' numeric ids collide.
Hash parsed content with normalised whitespace, not raw bytes. Metadata-only changes must bypass the hash deliberately.
Do not store source tags unnormalised, ingest-time summaries, ingest-time scores, or materialised per-user permissions.
Select every payload column in the retrieval query. The alternative is a round trip per answer.
Make permission columns not null with a check constraint. A nullable one fails safe at first and invites a leak as its fix.

Next: incremental sync and change detection — keeping the index current without re-reading everything.