Chapter 14 · The data layer

Reading the permission model you already have

Getting from 'it is managed in the CRM' to a column name. The questions that produce an answer, and the three discoveries that change the design.

10 min read

The previous chapter decided that permissions go in the retrieval query and that the data they need is attached at ingestion. This chapter is about obtaining that data, which is the hardest work in section 2 and the part most likely to be done badly, because it is social rather than technical.

The deliverable is narrow and concrete: for every source in the inventory, a statement of the form "a given person may see exactly the rows/documents where X", expressed in columns and values that exist.

Why the answer you get is not the answer you need

Ask how permissions work and you will be told the policy. The policy is an aspiration maintained by people. What you need is the mechanism, which is maintained by software and is frequently different.

What you are told	What is actually true
"Reps see their own accounts"	…and any account where they are named as secondary contact, and the twelve house accounts
"Finance sees everything"	…except two folders restricted to the executive team
"Permissions are in the CRM"	…for accounts. Documents are in the document store, by folder, with inherited sharing
"Warehouse staff see inventory"	…including cost price, because the inventory screen shows it and nobody noticed

That last row is the one to be most alert to. It is not a documentation gap; it is a pre-existing permission bug that your system is about to amplify enormously. A cost column visible on a screen that supervisors rarely scroll is a latent problem. The same column surfaced by an assistant that answers "what does that part cost us?" in one sentence is an active one.

You will find at least one access path the company did not know it had. Reporting it is part of this job, and it is better found now than by an assistant announcing it to twenty-four people.

The four questions that produce a mechanism

For each source, in order. Each one is designed to make a vague answer impossible.

1. Show me two people who see different things, and show me the difference on screen.

Not a description — the actual screens, side by side. This converts a policy statement into an observation in about ten minutes and it finds exceptions immediately.

2. Where is that difference stored?

Follow it to a table and a column, or a group membership, or a folder ACL. "The system handles it" is not an answer; the system is software and the difference is a value somewhere.

3. Who can change it, and what happens when they do?

This determines your sync requirements. A permission changed in the CRM that takes effect immediately there, and takes effect in your copy after the next nightly sync, has a staleness window measured in hours. Chapter 19 sizes that window; the point here is to know it exists.

4. What happens when somebody leaves, or changes role?

The question that finds the stale grants. In most companies the offboarding process removes login access reliably and removes individual document grants unreliably. Your index is a copy of those grants.

Writing it down as something executable

The output is not prose. It is a predicate per source, stored in the repository.

# permissions/model.yaml — the mechanism, per source, in real columns.
sources:
  policy_docs:
    model: classification
    rule: "visible to all internal users"
    implementation: "documents.classification = 'public_internal'"
    verified_against: "2 users compared on screen, 2026-11-03"

  customer_contracts:
    model: ownership
    rule: "a rep sees contracts for accounts they own or co-own"
    implementation: |
      accounts.owner_id = :user_id
      or accounts.secondary_owner_id = :user_id
      or accounts.account_type = 'house'      -- discovered, not documented
    source_of_truth: "CRM, accounts table"
    change_latency: "immediate in CRM; our copy syncs hourly"
    verified_against: "3 users, including one with a co-owned account"

  inventory:
    model: role
    rule: "quantities to warehouse supervisors; cost to finance only"
    implementation: |
      on_hand, allocated   -> warehouse_supervisor, finance
      unit_cost, margin    -> finance
    note: |
      ⚠️ The existing inventory screen shows unit_cost to supervisors. Reported
      to the data owner on 2026-11-03; treated here as finance-only regardless
      of what the screen does. Our system does not inherit a known defect.

The verified_against field is the one that distinguishes this document from the one you were handed on day one. A rule with no verification line is a hypothesis.

The note on inventory shows the right handling of the discovery above: report it, and do not copy it. An assistant that is more restrictive than an existing screen is defensible. One that is less restrictive is an incident.

Turning the predicate into ingestion metadata

The predicate has to be evaluated at two different times, and separating them is the whole trick.

Document-side facts are stable and belong in the index. Asker-side facts change constantly and must be resolved per request. Putting an asker-side fact in the index — materialising "which users may see this document" as a list — is the mistake that produces the stale-grant problem, because every permission change then requires rewriting index rows.

// The ingestion side: derive document-side metadata from the source system.
export async function classifyDocument(doc: SourceDocument): Promise<DocMeta> {
  switch (doc.source) {
    case 'policy_docs':
    case 'product_specs':
      return { classification: 'public_internal', accountId: null }

    case 'customer_contracts': {
      // The account reference must come from the source system, never from
      // parsing the document text. A contract that names two companies, or
      // misspells one, would otherwise be classified wrongly and silently.
      const accountId = doc.metadata.accountId
      if (!accountId) {
        // Fail the document, not the batch. Chapter 36 reports these.
        throw new SkipDocument(`contract ${doc.id} has no account reference`)
      }
      return { classification: 'account', accountId }
    }

    case 'pricing_workbook':
      return { classification: 'financial', accountId: null }
  }
}

Deriving the account reference from source metadata rather than from the document text deserves emphasis. Text extraction is lossy and contracts name multiple parties. A classification derived from parsing is wrong occasionally, and a permission that is wrong occasionally is worse than one that is wrong consistently, because it will pass your test set.

Verifying the model rather than assuming it

Once the predicates are written, test them against reality before building on them.

// evals/permission-parity.ts — does our model agree with the source system?
// Run against real users on a sample, as a one-off, before the first ingest.
export async function checkParity(userId: string, sample: string[]) {
  const scope = await resolveScope(await getUser(userId))

  const ours = new Set(
    sample.filter((docId) => isPermitted(docId, scope)),
  )
  // The comparison set comes from the source system's own answer: what does
  // the document store say this user can open?
  const theirs = new Set(await sourceSystem.listVisible(userId, sample))

  return {
    userId,
    weAllowTheyDeny: [...ours].filter((d) => !theirs.has(d)),   // ❌ over-permissive
    weDenyTheyAllow: [...theirs].filter((d) => !ours.has(d)),   // ⚠️ under-permissive
  }
}

The two result lists are not equally serious, and the asymmetry should drive how you react.

weAllowTheyDeny is a leak. Any non-empty result here is a stop-work finding: your model grants something the company does not.

weDenyTheyAllow is a coverage gap. It makes the assistant less useful than it should be, which is a bug to fix rather than an incident. It is also where the undocumented exceptions show up — the house accounts in the YAML above were found exactly this way, as 12 documents the source system allowed and the first version of the predicate denied.

Run this on a sample of 200 documents against five users with different profiles. It takes an hour and it is the cheapest security work in the project.

The failure mode: the model that cannot express the reality

Sometimes the four questions produce an answer your schema cannot hold. A real example shape: a rep may see a contract for an account they do not own if the contract is for a product line they are the specialist for, unless the account is flagged as sensitive.

Three axes, with an exception. The temptation is to add two more columns and a boolean, and then another two when the next rule appears, until the filter is a 40-line where clause nobody can reason about.

The better move at that point is to stop modelling rules and start modelling decisions:

-- A resolved grant table, computed from the rules on a schedule rather than
-- expressed in the query. The query stays one join; the complexity moves to
-- a job that can be tested, logged, and diffed between runs.
create table document_visibility (
  document_id text not null references documents(id) on delete cascade,
  user_id     text not null,
  computed_at timestamptz not null default now(),
  primary key (document_id, user_id)
);

This reintroduces the staleness problem from the ACL case — so take it only when the rules genuinely do not fit a predicate, and then treat the recompute job as a security-critical path with its own alerting. The thing it buys is that a complicated rule becomes a tested function rather than a where clause, and that a visibility change can be diffed and reviewed.

⚠️ Do not reach for this early. Two axes covers most companies, and a predicate you can read is worth real complexity elsewhere.

Key points

You need the mechanism, not the policy. Ask to see two people's screens side by side.
Expect to find an access path the company did not know it had. Report it; do not inherit it.
Record a predicate per source in real columns, with a verified_against line. A rule with no verification is a hypothesis.
Document-side facts belong in the index. Asker-side facts are resolved per request. Materialising user lists creates the stale-grant problem.
Derive classification from source metadata, never from parsed document text.
Run a parity check against the source system before the first ingest. Over-permissive findings are stop-work; under-permissive ones are where the undocumented exceptions hide.

Next: structured against unstructured — the trap of treating a table and a paragraph as the same kind of thing.