Chapter 21 · The data layer

Building the source registry

Every decision in this section, collected into one file the code actually reads. The registry is not documentation — it is the thing that fails the build when a source is misconfigured.

11 min read

Section 2 has produced a lot of decisions: shapes, owners, permission predicates, exclusions, retention rules, residency, freshness routes. Right now they are spread across a YAML file per chapter and several tables in several heads.

This chapter collects them into one registry. The point is not tidiness. It is that the registry is read by the ingestion pipeline, the retrieval filter and the eval harness, so a wrong entry breaks a build rather than producing a wrong answer six weeks later.

The shape

// src/sources/registry.ts — one entry per source. Every field here was decided
// by a chapter in this section, and the comment says which.
export interface SourceDefinition {
  id: string

  /** Chapter 12. Decides the ingestion path and whether it is indexed at all. */
  shape: 'documents' | 'relational' | 'api' | 'conversations'

  /** Chapter 12 and 19. Index it, or query it live. */
  route: 'index' | 'live_query'

  /** Chapter 11. Who approves connecting it and who answers questions about it. */
  owner: string

  /** Chapter 13 and 14. The document-side classification, and how to derive it. */
  classification: Classification | ((doc: SourceDocument) => Classification)

  /** Chapter 16 and 17. */
  sensitivity: 'public_internal' | 'commercial' | 'financial' | 'personal'
  residencyRegion: string
  retentionDays: number | null

  /** Chapter 19. Both measured, not estimated. */
  changeRate: string
  toleranceHours: number
  syncIntervalMinutes?: number
  stampAnswers: boolean

  /** Chapter 18. Which of the question set this source answers alone. */
  answersAlone: string[]
}

And the registry itself, abbreviated to three entries of the seven:

export const SOURCES: SourceDefinition[] = [
  {
    id: 'policy_docs',
    shape: 'documents',
    route: 'index',
    owner: 'Head of Operations',
    classification: 'public_internal',
    sensitivity: 'public_internal',
    residencyRegion: 'us-east',
    retentionDays: null,
    changeRate: '~2 per month',
    toleranceHours: 24 * 7,
    syncIntervalMinutes: 60 * 24,
    stampAnswers: false,
    answersAlone: ['ret-001', 'ret-002', 'pol-001' /* … 12 total */],
  },
  {
    id: 'pricing_workbook',
    shape: 'documents',          // a spreadsheet, serialised. Chapter 15.
    route: 'index',
    owner: 'Head of Sales Operations',
    classification: 'financial',
    sensitivity: 'financial',
    residencyRegion: 'us-east',
    retentionDays: null,
    changeRate: '~2 per month',
    toleranceHours: 24,
    syncIntervalMinutes: 60 * 24,
    stampAnswers: true,          // ⚠️ rare changes, tight tolerance. Chapter 19.
    answersAlone: ['pri-001', 'pri-002', 'pri-004', 'pri-007'],
  },
  {
    id: 'orders_db',
    shape: 'relational',
    route: 'live_query',         // thousands of changes a day. Chapter 12.
    owner: 'Head of Operations',
    classification: 'account',
    sensitivity: 'commercial',
    residencyRegion: 'us-east',
    retentionDays: null,
    changeRate: 'thousands per day',
    toleranceHours: 0,
    stampAnswers: false,
    answersAlone: ['inv-001', 'ord-002', 'ord-005'],
  },
]

The validation that makes it worth having

A registry that is only read is documentation with type annotations. The value comes from validating it, and the rules below encode the decisions from earlier chapters so they cannot be contradicted by an entry.

// src/sources/validate.ts — runs in the build, not in a review.
export function validateRegistry(sources: SourceDefinition[]): string[] {
  const errors: string[] = []
  const seen = new Set<string>()

  for (const s of sources) {
    if (seen.has(s.id)) errors.push(`duplicate source id: ${s.id}`)
    seen.add(s.id)

    // Chapter 19's rule, enforced. A tolerance tighter than the sync interval
    // is a stale-answer bug waiting for the right Tuesday.
    if (s.route === 'index' && s.syncIntervalMinutes) {
      const toleranceMinutes = s.toleranceHours * 60
      if (toleranceMinutes < s.syncIntervalMinutes && !s.stampAnswers) {
        errors.push(
          `${s.id}: tolerance (${s.toleranceHours}h) is tighter than the sync ` +
          `interval (${s.syncIntervalMinutes}m) and answers are not stamped. ` +
          `Either sync faster, set stampAnswers, or use route 'live_query'.`,
        )
      }
    }

    // Chapter 12. Conversations are excluded, so they have no index route.
    if (s.shape === 'conversations' && s.route === 'index') {
      errors.push(`${s.id}: conversations must not be indexed. See chapter 12.`)
    }

    // Chapter 16. Personal data does not reach the index at all.
    if (s.sensitivity === 'personal' && s.route === 'index') {
      errors.push(`${s.id}: sensitivity 'personal' cannot be indexed. See chapter 16.`)
    }

    // Chapter 20. An excluded source must not also appear here.
    if (EXCLUDED.some((e) => e.id === s.id)) {
      errors.push(`${s.id}: present in both the registry and the exclusion list.`)
    }

    // Chapter 18. A source that answers nothing should not be connected yet.
    if (s.answersAlone.length === 0) {
      errors.push(`${s.id}: answers no question in the question set. Why is it here?`)
    }
  }

  // Chapter 18, from the other direction: every question needs a source.
  const covered = new Set(sources.flatMap((s) => s.answersAlone))
  const orphans = QUESTIONS.filter((q) => !covered.has(q.id) && !q.multiSource && !q.noSource)
  if (orphans.length) {
    errors.push(`questions with no source and no exemption: ${orphans.map((q) => q.id).join(', ')}`)
  }

  return errors
}

// The test that makes all of it load-bearing.
test('the source registry is valid', () => {
  expect(validateRegistry(SOURCES)).toEqual([])
})

The last check is the one that earns the chapter. It closes the loop from chapter 18: a question that is neither answerable, nor marked multi-source, nor marked as having no source, is an unnoticed gap. Without this check that gap is discovered by a user.

What the registry drives

The last edge is worth doing, and it takes twenty lines.

// scripts/registry-docs.ts — generate the human-readable table from the code,
// so the document and the behaviour cannot disagree. Chapter 137's versioning
// problem starts here.
const rows = SOURCES.map((s) => [
  s.id, s.shape, s.route, s.owner, s.sensitivity,
  s.toleranceHours ? `${s.toleranceHours}h` : 'live',
  String(s.answersAlone.length),
])

console.log('| Source | Shape | Route | Owner | Sensitivity | Tolerance | Questions |')
console.log('|---|---|---|---|---|---|---|')
for (const r of rows) console.log(`| ${r.join(' | ')} |`)

The reason to generate rather than maintain is the one from chapter 14: a description that is maintained separately from the mechanism drifts from it, and the drift is invisible until somebody relies on the description.

The ingestion loop, driven by the registry

With the registry in place, the top of the ingestion pipeline becomes short, and the shortness is the proof that the decisions are in data rather than scattered through code.

// src/ingest/run.ts — section 3 fills in the adapters. This stays this size.
export async function ingestAll(now = new Date()) {
  const errors = validateRegistry(SOURCES)
  if (errors.length) throw new Error(`invalid registry:\n${errors.join('\n')}`)

  for (const source of SOURCES) {
    if (source.route !== 'index') continue           // live-queried: nothing to do
    if (!isDue(source, now)) continue                // chapter 19's schedule

    const adapter = resolveSource(source.id)         // chapter 20 refuses excluded
    const result = await ingestSource(source, adapter)

    await recordSyncRun(source.id, result)           // chapter 19's heartbeat
    logger.info('ingest.complete', {
      source: source.id,
      seen: result.seen,
      changed: result.changed,
      skipped: result.skipped,                       // chapter 14's SkipDocument
    })
  }

  await enforceRetention()                            // chapter 17
}

Every call in that function is a decision from this section, and none of them is a literal. That is the test of whether the registry is doing its job: if a new source can be added by appending one entry, and a wrong entry fails the build, the section has produced something structural rather than a set of notes.

The failure mode: the registry that describes a system it does not control

This is the one to guard against, and it is not hypothetical — it is what happens to most configuration files of this kind.

The registry is written. It is accurate. Then a new source is needed urgently, and the adapter is written directly, with its schedule in a cron entry and its classification hard-coded in the adapter, because that is forty minutes instead of two hours. It works.

Now the registry describes six of seven sources. Nothing is broken. The seventh source has no retention rule, is absent from the staleness alert, has a classification nobody reviewed, and is invisible to the coverage check. It will not be noticed until it is the source of an incident.

The defence is to make the registry the only path in, which is what resolveSource throwing on an unknown id achieves — and to assert the inverse too:

// Every adapter must have a registry entry. The check in the other direction,
// which is the one that catches the urgent Friday source.
test('every adapter is registered', () => {
  const registered = new Set(SOURCES.map((s) => s.id))
  const implemented = Object.keys(ADAPTERS)
  expect(implemented.filter((id) => !registered.has(id))).toEqual([])
})

Two tests, in opposite directions, and between them a source cannot exist in the code without existing in the registry or vice versa. That property is worth more than any individual field in it.

Key points

One registry, read by ingestion, retrieval, tools, evals and the retention job. Not documentation.
Validate it in the build. Encode the earlier chapters' rules as checks: no indexed conversations, no indexed personal data, no tolerance tighter than the sync interval without stamping.
Check coverage in both directions — a source answering nothing, and a question answered by nothing.
Generate the human-readable table from the code. A separately maintained description drifts invisibly.
When the registry drives the pipeline, the top of the pipeline is twenty lines and every decision is data.
Assert that every adapter is registered and every registry entry resolves. Between them, a source cannot exist on one side only.

Next, and the last chapter of this section: a dry run by hand — answering three real questions with no system at all, which is how you find what the registry is still missing.