Chapter 12 · The data layer

Documents, databases, APIs and inboxes

Four source shapes, four genuinely different engineering problems. What each one costs, which questions it can answer, and the mistake of treating them as variations on one.

10 min read

The inventory produced a list of sources with a shape field. This chapter is why that field exists and why it is the first thing recorded about any source.

The common mistake is to treat all four shapes as text to be chunked and embedded. It is technically possible for all of them, it is correct for one of them, and it is quietly wrong for the other three in ways that produce a system that works in a demo and is wrong in use.

The four shapes

Shape	Example	Answers	Route
Documents	Policies, contracts, specifications	"What is the rule for…"	Index and retrieve
Relational	Orders, inventory, customers	"How many…", "which…", "when did…"	Query live
API	Ticket system, shipping, CRM	"What is the status of…"	Call live
Conversations	Email, chat, call notes	"What did we agree with…"	Mostly: do not

The Route column is the consequential one. Two of these shapes should be indexed and two should be queried at request time, and getting that backwards is the single most common architectural error in systems like this.

Documents: index them

This is the shape retrieval was designed for. The content is prose, the question is semantic, and the answer is a passage. Everything in chapter 9 and in section 4 applies directly.

The problems are all about extraction, not retrieval:

PDFs that are images. 40% of Atlas's product specifications are supplier PDFs, and 760 of them have no text layer. Those need OCR, which is a slower pipeline with its own error modes. Chapter 27.
Structure that extraction destroys. A two-column layout read naively interleaves the columns. A table becomes a run of numbers with no headers.
Documents whose meaning depends on another document. Atlas's contract amendments are separate files with no link to the parent — found by hand in chapter 22.

# Separate the two populations before planning. These are different pipelines.
for f in ./export/specs/*.pdf; do
  chars=$(pdftotext "$f" - 2>/dev/null | tr -d '[:space:]' | wc -c)
  if [ "$chars" -lt 200 ]; then echo "$f ocr"; else echo "$f text"; fi
done | awk '{print $2}' | sort | uniq -c

Relational: query it, do not index it

This is where the serious mistake happens, and it is worth stating plainly.

Do not embed your database rows. A question with an exact answer deserves an exact query.

The reasoning is not aesthetic. Consider "how many units of SKU-4411 are in the Dallas warehouse?"

Indexed, the pipeline is: embed the question, find chunks that look like inventory rows, hope the right row is among the top twelve, and ask a model to read a number out of text. The answer is as fresh as the last sync. Similarity search over near-identical rows is close to random — every inventory row looks like every other inventory row to an embedding model, because they differ in exactly the tokens that carry no semantic weight.

Queried, the pipeline is: recognise the question, call a function, run select, return the number. It is correct, current, and auditable.

// The relational route is a tool, not an index. Section 6 builds this properly.
export const inventoryTool = {
  name: 'check_inventory',
  description:
    'Current on-hand quantity for a SKU, optionally at one warehouse. Use for any ' +
    'question about stock levels, availability or quantities on hand.',
  input: {
    sku: 'string, the SKU code, e.g. SKU-4411',
    warehouse: 'string, optional. One of: dallas, reno, columbus',
  },
  async run({ sku, warehouse }: { sku: string; warehouse?: string }, scope: Scope) {
    const { rows } = await db.query(
      `select w.name as warehouse, i.on_hand, i.allocated, i.updated_at
         from inventory i
         join warehouses w on w.id = i.warehouse_id
        where i.sku = $1
          and ($2::text is null or w.slug = $2)
          -- Permission filter, as in every query. Warehouse supervisors see
          -- quantities; nobody sees cost through this tool. Chapter 13.
          and w.id = any($3)`,
      [sku, warehouse ?? null, scope.warehouseIds],
    )
    return rows
  },
}

There is one legitimate exception, and it is narrow: free-text columns inside a relational source. A notes field on a support ticket is prose that happens to live in Postgres, and prose is a document-shaped problem. Index the column, not the row.

-- Index the prose, leave the structured fields to be queried.
select id, created_at, ticket_notes
  from tickets
 where ticket_notes is not null
   and length(ticket_notes) > 120;

APIs: call them, and respect their limits

Functionally similar to the relational route — a live call behind a tool — with three differences that are all operational.

Concern	Why it is different from a database
Rate limits	Atlas's ticket API allows 100 requests/minute, shared with other consumers
Latency	A third-party call is tens to hundreds of times slower than a local query
Availability	It will be down sometimes, and your system must degrade rather than fail

// The three differences, handled. Chapters 134 and 135 go further.
export async function callTicketApi(path: string, scope: Scope) {
  // 1. Latency: a hard timeout well inside the request's own budget.
  const signal = AbortSignal.timeout(2500)

  try {
    // 2. Rate limits: a shared limiter, because several tools may call this.
    return await limiter.schedule(() => fetchJson(path, { signal }))
  } catch (error) {
    // 3. Availability: return a degraded result the model can report honestly.
    //    Never return an empty list — the model cannot tell "none" from "unknown".
    return {
      unavailable: true,
      reason: 'The ticket system did not respond in time.',
    }
  }
}

That last comment is the part people get wrong. A failed API call that returns [] becomes "there are no open tickets for that account", stated with total confidence. Returning an explicit unavailable flag lets the answer say what is actually true: that it could not check.

Conversations: mostly, do not

Email, chat and call notes are the shape that looks most valuable and is most dangerous, and the recommendation for a first system is to exclude them. Chapter 4 made the case; here is what makes it an engineering problem rather than only a privacy one.

The fourth problem is the one that cannot be engineered around. In a policy document, a sentence is the rule. In an email, a sentence might be a rule, a proposal, a mistake somebody corrected two messages later, or one person's guess. A retrieved chunk carries no signal about which, and a model treats all four identically.

The practical consequence for Atlas: sales email is excluded from version one, the reason is recorded in the registry, and the institutional knowledge in it gets captured a different way — by noticing which questions the system cannot answer and writing the answers down as documents. That is slower and it produces a source with authority.

What this means for the architecture

The four shapes map onto exactly two routes, which is why the router in chapter 6 exists at all.

A question needing both — "is the Henderson contract's volume discount being applied to their current orders?" — needs the index for the contract term and a live query for the orders. That is a multi-source question, it is genuinely harder, and it is chapter 47.

The failure mode: the stale number

Here is how the architectural error actually manifests, because it does not look like an architecture problem.

Somebody indexes the inventory table, because it was easy and it made the demo answer stock questions. Retrieval returns a chunk reading SKU-4411 | Dallas | 1,240 | 2026-03-02. The model answers "1,240 units". The answer is well-formed, cited, and from a nightly sync — so it is between zero and twenty-four hours out of date, and nothing about it says so.

A rep quotes 1,240 units to a customer. 900 were allocated that morning.

The failure is not detectable by any eval that scores answer text, because the answer is correctly derived from the retrieved source. It is only detectable by asking a different question: should this source have been indexed at all?

The rule that prevents it:

If the answer changes faster than the sync interval, it cannot come from the index.

Chapter 19 turns that into a per-source decision with numbers rather than a principle.

Key points

Four shapes, two routes. Documents and free text are indexed; relational and API sources are queried live.
Do not embed database rows. Similarity search over near-identical rows is close to random, and the answer is only as fresh as the sync.
Index free-text columns inside a relational source. Index the column, not the row.
An API tool that fails must return an explicit unavailable flag. An empty list becomes a confident "there are none".
Conversations carry no signal about whether a sentence is a rule, a proposal or a mistake. Exclude them from version one and record why.
If the answer changes faster than the sync interval, it cannot come from the index.

Next: who is allowed to see what — and why this is decided before retrieval is built rather than after.