Chapter 6 · Foundations

The architecture at a glance

Seven components, what each one is responsible for, and the two boundaries that are expensive to move later. A map to return to whenever a chapter feels like it is about something unrelated.

9 min read

This chapter is a map. It assumes you have met Atlas and know the difference between a retriever, a workflow and an agent from chapter 1. Nothing is built here. What you get instead is a named place to hang every later chapter, and a warning about the two decisions in it that are painful to reverse.

The seven pieces

The split between the two boxes is the first thing to internalise. Everything above the line runs on a schedule, can take minutes, and is allowed to fail loudly and retry. Everything below the line runs while somebody waits, has a latency budget measured in seconds, and must degrade rather than crash.

Confusing the two is a common early mistake. Parsing a 400-page PDF at question time, or embedding a document on the request path because it was missed overnight, turns a four-second answer into a forty-second one for the unlucky person who asked first.

What each piece owns

#	Piece	Responsible for	Covered in
1	Ingestion	Fetching, parsing, chunking, change detection	Section 3
2	Index	Storing chunks, vectors and the metadata that filters them	Sections 3 and 4
3	Identity	Resolving the asker to a set of permissions	Section 2, enforced in section 10
4	Router	Sending cheap requests down cheap paths	Sections 5 and 7
5	Retrieval	Returning the right chunks, and only permitted ones	Section 4
6	Tools	Reading and changing the systems of record	Section 6
7	Generation	Turning context into a grounded, cited answer	Section 5

Two things are deliberately absent from that table, and their absence is the point.

There is no "agent" box. The agent is what you get when the router sends a request to a loop that calls pieces 5 and 6 repeatedly. It is a control-flow pattern over components that already exist, which is why section 7 comes so late: there is nothing for a loop to orchestrate until the tools and retrieval beneath it work.

There is also no evaluation box, because evaluation is not a component. It is a harness that wraps pieces 5, 6 and 7 independently, and it is section 9. Chapter 10 sets the numbers it will measure against.

The two expensive boundaries

Most of this architecture is cheap to change. You can replace the router, swap the model, add a reranker, or rewrite the chunker in an afternoon and only the quality numbers move. Two choices are not like that.

The permission boundary

The question is where you decide what somebody may see. There are three candidate places and only one correct one.

Permissions belong in the query, not in the prompt. A filter the model is asked to honour is a request; a filter in the where clause is a rule.

The third option is attractive because it is one sentence of prompt and no schema work. It is also the single most common cause of the leak described in chapter 3, and it fails in a way that is invisible until it matters: nothing logs that a model chose to comply.

The reason this boundary is expensive is that filtering at retrieval requires every chunk to carry the metadata that makes filtering possible, and that metadata is attached during ingestion. Deciding this after you have indexed a corpus means re-ingesting all of it.

So the ordering in this series is not arbitrary. Permissions are chapter 13, before retrieval exists, because the index has to be built with them in mind.

The chunk boundary

A chunk is the unit you retrieve. Its size and shape determine what a search can possibly find, and they are set during ingestion.

Choose chunks that are too large and every result wastes context on irrelevant text. Choose chunks that are too small and a clause loses the section heading that gave it meaning. Neither is visible from the retrieval code. Both look like "the search is bad".

Changing chunking means re-embedding, which costs real money on a large corpus and invalidates every retrieval measurement you have taken. This is why section 3 treats chunking as a decision to measure rather than guess, and why chapter 31 is about measuring it rather than about a recommended number.

The request path, concretely

Here is the whole online half as code. It is not the implementation — it is the shape that the implementation will fill in, and it is worth having in front of you now because every chapter in sections 4 to 7 expands exactly one line of it.

// The request path. Seven lines, each one a section of this series.
export async function ask(question: string, asker: User): Promise<Answer> {
  // 3. Identity: resolve the asker to concrete permissions.
  const scope = await resolveScope(asker)

  // 4. Router: decide which of the three paths this request takes.
  const route = await classify(question)

  if (route.kind === 'task') {
    // 6. Tools, in a fixed order you wrote.
    return runWorkflow(route.task, scope)
  }

  if (route.kind === 'planning') {
    // The agent loop: retrieval and tools, ordered by the model, bounded by you.
    return runAgent(question, scope, { maxSteps: 8, maxSeconds: 60 })
  }

  // 5. Retrieval, with the scope in the query rather than in the prompt.
  const chunks = await retrieve(question, { scope, limit: 12 })

  // 7. Generation, with citations required.
  return generate(question, chunks)
}

Notice that scope is threaded through every branch and is never optional. That is a deliberate API design choice: a retrieve() that can be called without a scope will eventually be called without one.

// Make the unsafe call impossible to write, rather than discouraged.
type Scope = { userId: string; accountIds: string[]; classifications: string[] }

// Not: retrieve(question, options?: { scope?: Scope })
export function retrieve(question: string, options: { scope: Scope; limit?: number }) {
  // ...
}

This is a small thing that pays for itself. A required parameter is checked by the compiler on every call site, including the ones a model writes for you and the ones added at speed six months from now.

Where the money goes

Worth knowing before you design, because it changes what you optimise.

Component	Share of cost	Why
Generation	Usually the largest	Every answer pays for its context, and context is mostly retrieved chunks
Ingestion embeddings	Large once, small thereafter	A full re-index is the expensive event; incremental sync is cheap
Retrieval	Near zero	A vector search over a few hundred thousand chunks is a database query
Agent loops	Highly variable	A request that takes eight steps costs roughly eight times one that takes one

The two levers that matter are both routing decisions rather than optimisations: send questions to the retriever instead of the loop, and put fewer, better chunks in context instead of more. Chapter 60 and section 11 return to this with numbers.

The failure you will hit first

You will build this in roughly the order of the diagram, and around the time retrieval starts returning plausible results you will wire up generation and get an answer that is confidently wrong. The investigation will go to the prompt, because that is where the wrong sentence appeared.

It is almost never the prompt. In the first weeks of a system like this, a wrong answer is a retrieval problem about four times in five: the right chunk was not returned, or was returned ranked eighth out of a window that held six.

The habit that saves the most time is to make this diagnosable from the start. Log, for every answer, the chunk IDs that went into context and their scores. When a wrong answer arrives, look at that list before looking at anything else. If the supporting chunk is not in it, no amount of prompt work will help.

// The one piece of observability worth having on day one.
logger.info('answer', {
  questionId,
  route: route.kind,
  chunkIds: chunks.map((c) => c.id),
  scores: chunks.map((c) => Number(c.score.toFixed(3))),
  tokensIn,
  tokensOut,
})

Section 11 builds this out properly. One structured log line gets you most of the value now.

Key points

Offline and online halves have different budgets and different failure behaviour. Never do ingestion work on the request path.
There is no "agent" component. The agent is a loop over retrieval and tools, which is why it comes last.
The permission filter belongs in the retrieval query. Metadata must be attached during ingestion to make that possible, so permissions are decided before retrieval is built.
Chunking and permissions are the two expensive boundaries. Everything else can be swapped.
Log the chunk IDs behind every answer. Most wrong answers are retrieval failures wearing a prompt's clothes.

Next: choosing the first slice — how to pick the narrowest version that still proves the hard parts, and the four tests a candidate slice has to pass.