Chapter 16 · The data layer

Personal data, and what never enters the system

A decision that is cheap before the first ingest and expensive after it. What personal data does to an index, and the three handling options with their real costs.

11 min read

Everything in this section so far has been about getting data in. This chapter is the first about keeping data out, and it is placed before ingestion for an asymmetry that is worth stating up front.

Excluding a source before the first ingest costs a line in a config file. Removing personal data from an index that already holds it costs a full re-ingest, a search of every log and trace, and a conversation about what was retrieved in the meantime.

It assumes the inventory, and it feeds chapter 20, which is about saying no to a whole source.

This chapter is engineering guidance, not legal advice. The obligations that apply to you depend on where you and your data subjects are, and that is a question for someone qualified. What follows is about what the obligations mean for the architecture.

What personal data does to an index

Three properties of a retrieval index interact badly with personal data, and they are properties, not bugs.

An index is a copy. Everything about retention, residency and deletion that applies to the source now applies to a second place the data lives. Chapter 17 covers what your contracts already promise about that.

Embeddings are derived data, and they are not anonymised. A vector computed from a sentence containing somebody's name and medical note is personal data. It is not reversible in a useful sense, but it is linked to an identifiable person and it sits beside the plaintext chunk in the same row. Treating the vector as "just numbers" is a mistake people make in writing.

Retrieval is recombination. This is the property with no equivalent in the source system. Three documents, each individually unremarkable, can be retrieved together and produce an answer nobody was ever authorised to assemble.

No document leaked. The index assembled something new. This is why "each of these documents is fine on its own" is not a sufficient test, and why the question for every source is what it enables in combination.

The four categories

A practical taxonomy. Classify every source in the inventory into one of these before ingestion.

Category	Examples	Handling
No personal data	Policies, product specifications, price lists	Ingest normally
Incidental, business-context	A signatory's name on a contract, an author's name	Ingest; usually fine, and removing it breaks the document
Substantive personal data	HR files, performance notes, medical, disciplinary	Exclude. Not a redaction problem
Mixed and inseparable	Email, chat, call notes	Exclude from version one

The distinction between rows two and three is the one that matters and it is not about volume. It is about whether the document is about a person. A contract that names the person who signed it is a business record with a name in it. A performance note is a record about a person, and no amount of redaction makes it something else — the whole document is the problem.

Row four is the realistic position on conversations, for the engineering reasons in chapter 12 as much as the privacy ones.

Three handling options, with honest costs

Exclude the source

The cheapest and most often correct. It costs a config entry and a recorded reason.

# sources/excluded.yaml — exclusions with reasons, because "why not?" gets
# asked in every review and the answer should not require a meeting.
- id: hr_records
  reason: "Substantive personal data. Not in scope for any planned slice."
  decided_by: "Head of People, Head of Sales Operations"
  decided_on: 2026-11-03
  revisit: never

- id: sales_email
  reason: >
    Mixed personal and business content, inseparable by folder. Also has no
    authority signal — a proposal and a decision are indistinguishable.
  decided_by: "Head of Sales Operations"
  decided_on: 2026-11-03
  revisit: "After version one ships, if the question log shows a real gap."

The revisit field distinguishes "never" from "not yet", which chapter 7 argued for and which is worth the discipline. Those are different decisions and they get conflated in conversation constantly.

Filter at the source boundary

Ingest the source but drop documents matching a rule. Workable when the personal data is in an identifiable subset.

// Allow-list by location, not deny-list by pattern. A deny-list fails open:
// a new folder nobody added a rule for gets ingested.
const ALLOWED_PREFIXES = [
  'policy/',
  'specs/',
  'contracts/active/',
] as const

export function shouldIngest(doc: SourceDocument): Decision {
  if (!ALLOWED_PREFIXES.some((p) => doc.path.startsWith(p))) {
    return { ingest: false, reason: `path not in allow-list: ${doc.path}` }
  }
  // A belt-and-braces content check, reported rather than silently dropped.
  // Chapter 36 surfaces these counts after every run.
  const found = scanForSensitivePatterns(doc.text)
  if (found.length) {
    return { ingest: false, reason: `matched ${found.join(', ')}` }
  }
  return { ingest: true }
}

The allow-list direction is the important part. A deny-list is a list of the mistakes you have thought of, and it fails open on every one you have not — including a folder created next quarter.

Redact during ingestion

The option that sounds best and is weakest. It is worth being specific about why rather than just advising against it.

// ⚠️ This is a defence in depth measure, NOT a control. Read the caveats below
// before relying on it for anything.
const PATTERNS: Array<[string, RegExp]> = [
  ['EMAIL', /\b[\w.+-]+@[\w-]+\.[\w.]{2,}\b/g],
  ['PHONE', /\b(?:\+?1[-. ]?)?\(?\d{3}\)?[-. ]?\d{3}[-. ]?\d{4}\b/g],
  ['SSN', /\b\d{3}-\d{2}-\d{4}\b/g],
  ['CARD', /\b(?:\d[ -]?){13,16}\b/g],
]

export function redact(text: string): { text: string; counts: Record<string, number> } {
  const counts: Record<string, number> = {}
  let out = text
  for (const [label, pattern] of PATTERNS) {
    out = out.replace(pattern, () => {
      counts[label] = (counts[label] ?? 0) + 1
      return `[${label}_REDACTED]`
    })
  }
  return { text: out, counts }
}

Three reasons this is not a control:

Names are not patterns. The highest-value personal data in most corpora is names in context, and there is no regex for a name. Named-entity recognition catches many and misses some, and the ones it misses are not random — they are unusual names, which are also the most identifying.
Recall is unmeasured. You know what the patterns matched. You have no idea what they missed, and an unmeasurable control cannot be relied on.
Recombination survives redaction. The inference in the diagram above still works with every email address removed.

Use it for structured identifiers — card numbers, national insurance numbers, account numbers — where the pattern genuinely is the data, and where its presence in an index is a compliance problem in itself. Do not use it to make a substantive personal-data source ingestible.

// Report what was redacted, per run. A count that jumps means the source
// changed shape, and that is worth a human looking.
logger.info('ingest.redaction', { source, documents: n, counts: totals })

The right-to-erasure path, designed now

If personal data is in scope at all, deletion has to work, and it has to work across three places. Building this later means finding out which of the three you forgot.

-- 1. Documents and chunks. The cascade from chapter 8 handles the chunks.
update documents set deleted_at = now() where source_ref = any($1);

-- 2. Anything derived: conversation logs, memory, eval fixtures. Chapter 98.
delete from question_log where subject_ref = any($1);

// 3. The part people miss: your traces and logs hold retrieved chunk text.
//    If generation requests are logged with their context, the deleted content
//    is in the log aggregator with its own retention policy.
export const LOGGING_RULES = {
  // Chunk IDs and scores: yes. Chunk content: never.
  logChunkContent: false,
  // A hard cap on log retention, set to the shortest useful window.
  retentionDays: 30,
} as const

Not logging chunk content is a decision best made before there is a production incident that would be easier to debug with it. The compromise that works is to log chunk IDs and scores — which is what chapter 6 recommended for diagnosis anyway — and to re-fetch content from the index when investigating, where deletion has already applied.

The failure mode: the helpful inference

An employee asks Atlas: "who should I talk to about the Henderson account?"

A reasonable question with a reasonable answer. But suppose the corpus included meeting notes, because meeting notes seemed like business records. Retrieval returns a note from a management meeting mentioning that the account's previous owner is "on a performance plan and the account has been reassigned pending review".

The answer names a colleague and discloses their employment status to a peer.

Every component worked. The document was genuinely a business record, it was in a folder everybody could read, retrieval found the most relevant passage, and the model summarised it accurately. The failure was a scoping decision made weeks earlier, in this chapter, by somebody who classified meeting notes as row two rather than row three.

The test that catches it is not technical:

For each source, ask: if the single most sensitive sentence in this source were read aloud to the whole user group, would that be acceptable?

Retrieval will eventually find that sentence for somebody. Not immediately, and not predictably — which is precisely why the decision cannot be deferred until it happens.

Key points

Exclusion before the first ingest is a config line. Removal afterwards is a re-ingest plus a log audit.
Embeddings derived from personal data are personal data, sitting beside the plaintext in the same row.
Retrieval recombines. Three innocuous documents can produce an inference none of them contained, so per-document review is not a sufficient test.
The line is whether the document is about a person, not how much personal data it contains. Records about people are excluded, not redacted.
Allow-list by location. A deny-list fails open on every folder you did not anticipate.
Pattern redaction is defence in depth for structured identifiers. It has unmeasurable recall and does not touch names in context.
Design erasure across three places: the index, derived data, and your logs. Log chunk IDs, never chunk content.
The test: if the most sensitive sentence in a source were read aloud to the whole user group, would that be acceptable?

Next: residency, retention and what your contracts already promise — obligations that already exist and that an index can silently break.