Chapter 20 · The data layer

The data you should refuse to connect

Five categories to decline, how to tell a hard no from a not-yet, and how to say it so it is heard rather than overruled in the next meeting.

9 min read

Every chapter in section 2 so far has been about handling a source correctly. This one is about declining. It is a short chapter and it is the one most likely to be useful in a meeting rather than at a keyboard.

The inventory will have surfaced at least one source that should not be connected. Somebody will want it connected, for good reasons, and the request will arrive as "can we just add…".

Five categories to decline

1. Records about people

Covered in chapter 16: HR files, performance notes, medical information, disciplinary records. The test is whether the document is about a person rather than how much personal data it contains, and redaction does not convert one into the other.

2. Data under a restriction you cannot satisfy

A supplier NDA restricting onward disclosure, a customer amendment requiring in-region processing you do not have, data a certification scope excludes. Chapter 17 is how you find these; this is the decision that follows.

The distinguishing feature is that this is not a judgement call. The restriction exists in writing, and connecting the source breaks it.

3. Sources with no authority signal

Email, chat, draft folders, anything where a proposal and a decision are textually indistinguishable. The argument is in chapter 12 and it is an engineering argument rather than a compliance one: a retrieved chunk carries no indication of whether its sentence was ever true, and no prompt fixes that.

4. Sources whose permission model you cannot read

If the four questions from chapter 14 did not produce a mechanism — if the answer remains "the system handles it" and nobody can say which column — then you cannot write the retrieval filter. The honest position is that this source is blocked pending an answer, not that it will be connected without one.

This is a not yet, and it is the most commonly mishandled category, because the blocker is somebody else's unavailability rather than a technical problem. Connecting it anyway means guessing at a permission rule, and a guessed permission rule passes your test set.

5. Data that is wrong

The category nobody expects. A source everyone knows is out of date, maintained by nobody, kept because deleting things is hard.

Connecting it does active damage, because retrieval launders it. A spreadsheet in a shared folder that people know not to trust becomes, after indexing, a cited source in a confident answer — and the citation makes it more credible than it was before.

# The fifth category, which is a data-quality problem wearing a scoping hat.
- id: legacy_product_catalogue
  reason: >
    Superseded by the specification store in 2024 and not maintained since.
    Still holds 1,900 entries, ~40% of which contradict current specifications.
    Indexing it would produce cited answers from a source people currently know
    to distrust.
  decided_by: "Head of Product"
  decided_on: 2026-11-03
  revisit: "If it is archived or corrected. Not otherwise."

Hard no against not yet

The distinction decides how the conversation goes, and conflating them is why these decisions get relitigated.

	Hard no	Not yet
Nature	The reason will not change	A specific blocker exists
Example	HR records; a legacy source nobody maintains	Permission model unread; OCR pipeline not built
What unblocks it	Nothing you will do	A named action with an owner
Recorded as	`revisit: never`	`revisit: <condition>`

# A not-yet, written so it can actually be revisited.
- id: scanned_contracts_pre_2023
  reason: "No text layer. 760 files need an OCR pass."
  revisit: "After chapter 27's OCR pipeline. Estimated 3 weeks."
  blocker_owner: "engineering"
  questions_blocked: [ctr-003, ctr-008]

Recording questions_blocked is what makes a not-yet decision reviewable on evidence. When somebody asks why the scanned contracts are not connected, the answer includes which two questions are affected, and that is a priority conversation rather than an argument about principle.

How to say it so it is heard

The common mistake is to say no on the grounds of risk. Risk arguments lose to opportunity arguments in most rooms, and they should, because they are usually unquantified.

What works is to state the consequence in the terms the requester already cares about.

Instead of	Say
"That's a privacy risk"	"That folder holds performance notes. The assistant would answer 'who should I talk to about this account' by disclosing a colleague's review status to twenty-four people."
"The data's not clean"	"40% of those entries contradict the current specifications. Connecting it means some answers will be confidently wrong and cited, which is worse than no answer."
"We don't have time"	"It needs an OCR pass — about three weeks — and it unblocks two of the thirty-four questions. Policy documents unblock twelve. I'd do those first."

Each of those is the same decision stated as a specific consequence with a specific scope. The third one is particularly worth noticing: it is not a refusal at all, it is a sequencing argument, and most "not yet" conversations are really sequencing conversations that got framed as refusals.

A decision recorded with its consequence survives the meeting it was made in. A decision recorded as a risk gets revisited by whoever next wants the data.

Make the exclusion enforced, not documented

A decision in a document is a decision until somebody writes a loader. Exclusions belong in the code path.

// src/ingest/sources/index.ts — the registry decides, and refuses by default.
import { EXCLUDED } from './excluded.js'

export function resolveSource(id: string): SourceAdapter {
  const exclusion = EXCLUDED.find((e) => e.id === id)
  if (exclusion) {
    // Not a warning. A source that was deliberately excluded must not be
    // ingestible by passing a flag, because the flag will be passed.
    throw new Error(
      `Source '${id}' is excluded: ${exclusion.reason}\n` +
      `Decided by ${exclusion.decided_by} on ${exclusion.decided_on}.\n` +
      `Revisit: ${exclusion.revisit}`,
    )
  }
  const adapter = ADAPTERS[id]
  if (!adapter) throw new Error(`Unknown source '${id}'. Register it first.`)
  return adapter
}

Two properties of that function matter. It throws rather than warns, and an unknown source is also an error — so connecting something new is a deliberate code change rather than a configuration value somebody sets at 6pm.

// And a test, so the exclusion list cannot be emptied quietly.
test('excluded sources cannot be resolved', () => {
  for (const { id } of EXCLUDED) {
    expect(() => resolveSource(id)).toThrow(/is excluded/)
  }
  expect(EXCLUDED.length).toBeGreaterThan(0)   // the list itself must not vanish
})

The final assertion looks paranoid and is not. An exclusion list that becomes empty through a merge, a refactor or a cleanup is indistinguishable from one that was never needed.

The failure mode: the exception that becomes the rule

A single document from an excluded source is needed. One contract amendment, one HR policy that is genuinely general, one row from the legacy catalogue. It is a reasonable request and the right answer is usually yes.

The wrong implementation is a flag:

// ❌ Within a month this is set on three sources and nobody remembers why.
await ingestSource('hr_records', { allowExcluded: true, onlyPath: 'policies/' })

The right implementation is to make the exception a source of its own, with its own entry in the registry, its own classification, and its own narrow definition:

# ✅ A new source, not an exception to an old one. It has a path, a reason, an
# owner and a classification, and it is as reviewable as everything else.
- id: hr_general_policies
  shape: documents
  path: "hr/policies/general/"      # a specific folder, not a flag
  owner: "Head of People"
  access_model: "All internal users"
  sensitivity: public_internal
  note: >
    Carved out of the excluded hr_records source: general policies only
    (leave, expenses, travel), which are published to all staff anyway.
    Individual records remain excluded. Approved 2026-11-03.

The difference is that the second version cannot drift. A flag is an invitation; a path is a boundary. And when somebody later asks why Atlas knows about the expenses policy but not about anyone's leave balance, the registry answers it.

Key points

Five categories to decline: records about people, data under restrictions you cannot meet, sources with no authority signal, sources whose permission model you cannot read, and data that is wrong.
Indexing a source people already distrust launders it. The citation makes it more credible than it was.
Separate a hard no from a not-yet. A not-yet needs a named blocker, an owner, and the list of questions it blocks.
State the consequence, not the risk. Risk arguments lose to opportunity arguments, and most not-yets are sequencing arguments in disguise.
Enforce exclusions in the loader: throw rather than warn, make unknown sources an error, and test that the list is non-empty.
Carve an exception out as a new source with its own path and classification. Never as a flag on an excluded one.

Next: building the source registry — turning everything decided in this section into one file the code reads.