Chapter 11 · The data layer

Inventorying what the company actually has

The exercise that reliably finds two sources nobody mentioned and one nobody is allowed to connect. How to run it in a week, and what to record for each source.

9 min read

Section 2 is about the data layer: everything that has to be true about your data before retrieval is worth attempting. It starts here, with finding out what there is.

This sounds like a formality. It is not. Run properly, an inventory changes the plan in most projects, and it changes it early, which is when changing the plan is cheap. It assumes the slice from chapter 7 so the exercise is bounded, and it feeds directly into the source registry in chapter 21.

Why the list you were given is wrong

You will have been told where the documents are. That list is produced by somebody senior describing the system as designed. The data is in the system as used, and the gap between the two is where the interesting material lives.

Three things are predictably missing from the list you were handed:

The shadow copy. A shared drive folder, or one person's local folder, holding the versions people actually use because the official store is slow or badly organised.
The thing in an inbox. Commitments, exceptions and approvals that were never written anywhere else. Chapter 4 made this argument; the inventory is where you find out how much of it there is.
The spreadsheet that is really a database. Someone's workbook that is the authoritative source for something, maintained by hand, referenced by a dozen people.

And one thing is predictably present that should not be connected at all. Finding it now rather than in week nine is most of the value of the exercise. Chapter 20 is about how to say so.

The method

A week, mostly conversations, with a bounded scope.

Starting from the questions is what keeps this from becoming a six-week data-governance project. You have 34 questions from chapter 7. For each one, ask somebody who answers it today: where did you look? Follow that answer to a real system, then ask what they would have looked at if that had not worked.

The second question is the one that finds the shadow copies.

The eight fields

Record these for every source. The set is chosen so that each field answers a decision made later in this section, and nothing is recorded for completeness.

Field	Decides	Chapter
Shape	Which ingestion path it needs	12
Owner	Who approves connecting it	13
Access model	How permissions are expressed today	14
Volume	Whether chunking strategy and cost change	30, 130
Change rate	Sync frequency, and live-query against index	19
Sensitivity	Whether it can be connected at all	16, 20
Questions answered	Priority	18
Retrieval route	Index it, or query it live	19

# A filled entry. Keep these in the repository, not in a wiki page.
- id: customer_contracts
  shape: documents            # documents | relational | api | conversations
  owner: "Head of Sales Operations"
  access_model: "Per-account, via CRM account ownership"
  volume: { count: 3100, pages: 18400 }
  change_rate: "~12 per week"
  sensitivity: commercial     # public_internal | commercial | financial | personal
  questions_answered: [ctr-001, ctr-002, ctr-004, pri-003, pri-007]
  retrieval_route: index      # index | live_query
  notes: >
    Signed PDFs, mostly scanned before 2023 — those need OCR and are excluded
    from the first ingest. Amendments are separate files with no link to the
    parent contract, which chapter 22 found by hand.

That notes field earns its place. The structured fields are what you query; the notes are where the thing that breaks the plan gets written down, and in practice it is the field most referred back to.

What it finds for Atlas

The list handed over on day one had three items: the document store, the orders database, and the ticket system. After a week of following questions:

Source	Status after the inventory
Policy and procedure documents	Confirmed. 340 documents, two of which are superseded and still present
Product specifications	Confirmed. 1,900 documents, 40% from supplier PDFs with poor text layers
Customer contracts	Confirmed. 3,100 — but amendments are unlinked separate files
Orders and inventory (Postgres)	Confirmed. Live query, never indexed
Support tickets (API)	Confirmed. Rate limited to 100 requests/minute
Sales email	Found. Significant real knowledge, and a personal-data problem
The pricing workbook	Found. A spreadsheet that is the authoritative source for regional discounts
A folder of supplier price lists	Found. Shadow copy, more current than the document store

Three sources were not on the list, and one of them — the pricing workbook — answers four of the 34 questions on its own. A project that did not run the inventory would have launched with four known-wrong answers and no idea why.

Volume, measured rather than estimated

The number people give you is wrong by a factor that is rarely small. Count.

# Documents, by extension and total size, under the export directory.
find ./export -type f | sed -n 's/.*\.\([A-Za-z0-9]*\)$/\1/p' \
  | tr 'A-Z' 'a-z' | sort | uniq -c | sort -rn | head

# How many PDFs have a usable text layer? The ones that do not need OCR,
# which is a different and slower pipeline. See chapter 27.
for f in ./export/specs/*.pdf; do
  chars=$(pdftotext "$f" - 2>/dev/null | tr -d '[:space:]' | wc -c)
  [ "$chars" -lt 200 ] && echo "needs OCR: $f"
done | wc -l

-- For a relational source: row counts and change rate in one query, which is
-- usually more informative than asking.
select 'orders' as t, count(*),
       count(*) filter (where updated_at > now() - interval '7 days') as changed_7d
  from orders
union all
select 'inventory', count(*),
       count(*) filter (where updated_at > now() - interval '7 days')
  from inventory;

The OCR count is the one that most often changes a timeline. "1,900 product specifications" and "1,900 product specifications, of which 760 are scans that need an OCR pass" are different projects by about three weeks.

Ranking what to connect

Not by size, and not by enthusiasm. By questions answered per unit of difficulty.

// A crude score is enough. The ranking it produces is usually uncontroversial,
// which is the point — it makes the obvious order explicit.
type Source = {
  id: string
  questionsAnswered: number
  difficulty: 1 | 2 | 3 | 4 | 5    // parse complexity + access + permissions
  blocked?: string                  // a reason, if it cannot be connected yet
}

export const priority = (sources: Source[]) =>
  sources
    .filter((s) => !s.blocked)
    .map((s) => ({ ...s, score: s.questionsAnswered / s.difficulty }))
    .sort((a, b) => b.score - a.score)

For Atlas that produces: policy documents (12 questions, difficulty 1), the pricing workbook (4 questions, difficulty 1), product specifications (9 questions, difficulty 3), contracts (5 questions, difficulty 4), and sales email blocked pending chapter 16.

The pricing workbook ranking second is the kind of result that only comes out of this exercise. It is a single file, it would be trivial to ingest, nobody mentioned it, and it answers more questions per unit of effort than the 3,100-document contract store.

The failure mode: inventorying systems instead of answers

The way this goes wrong is that it becomes thorough. Somebody suggests a complete data catalogue, and the exercise expands to every system the company runs, with a template of thirty fields, and after five weeks there is a beautiful document and no ingestion pipeline.

The guard is the one in the method: every source in the inventory is there because it answers at least one of the 34 questions. A system that answers none of them is recorded in a single line as out of scope and not investigated further. It can be investigated when a question needs it.

An inventory is complete when every question has a source, not when every system has an entry.

The second failure is subtler and more expensive: recording the access model as a sentence rather than as a mechanism. "Permissions are managed in the CRM" is a note. "Account ownership is a single owner_id column on the accounts table, and a rep sees exactly the accounts where it matches their user id" is a design input. Chapter 14 is about getting to the second kind of answer, and it is the hardest chapter in this section.

Key points

Start from the questions, not from the systems. It keeps the exercise to a week.
Ask where somebody looked, then ask where they would look if that failed. The second question finds the shadow copies.
Record eight fields, each answering a decision made later, and keep the inventory in the repository.
Count volumes rather than accepting the figure you are given. The OCR share is the one that moves timelines.
Rank by questions answered per unit of difficulty. A single overlooked spreadsheet often outranks a document store.
An inventory is complete when every question has a source, not when every system has an entry.

Next: documents, databases, APIs and inboxes — why these four are genuinely different engineering problems, and what each one costs.