Chapter 9 · Foundations

A hello-world agent

One document, one question, one cited answer, under a hundred lines. The end-to-end path working badly is worth more than any component working well.

11 min read

This chapter builds the whole path end to end, at the smallest possible scale: one document, a handful of chunks, one question, one answer with a citation. It assumes the repository from chapter 8 and a running Postgres with pgvector. The source shapes it does not yet handle are the subject of chapter 12.

The point is not the quality of the answer. It will be mediocre. The point is that every seam between components has been crossed once, so from here every chapter improves something measurable rather than building something unknown.

The whole thing

Four steps, and the shape never changes no matter how large the system gets.

Step 1: chunk one document

A deliberately naive chunker. Chapter 30 explains why this is the decision that quietly determines quality and how to do it properly; today it needs to exist, not to be good.

// src/ingest/chunk.ts — the naive version, replaced in section 3.
export interface Chunk {
  ordinal: number
  content: string
  headingPath: string[]
}

/** Splits on markdown headings, then on paragraph boundaries at ~800 tokens. */
export function chunkMarkdown(markdown: string, targetTokens = 800): Chunk[] {
  const chunks: Chunk[] = []
  const headingPath: string[] = []
  let buffer: string[] = []

  const flush = () => {
    const content = buffer.join('\n\n').trim()
    if (content) {
      chunks.push({ ordinal: chunks.length, content, headingPath: [...headingPath] })
    }
    buffer = []
  }

  for (const block of markdown.split(/\n{2,}/)) {
    const heading = /^(#{1,6})\s+(.*)$/.exec(block.trim())
    if (heading) {
      flush()
      const depth = heading[1].length
      headingPath.length = depth - 1
      headingPath[depth - 1] = heading[2].trim()
      continue
    }
    buffer.push(block)
    // ~4 characters per token is close enough to split on and wrong enough
    // to measure properly later. See chapter 31.
    if (buffer.join('\n\n').length > targetTokens * 4) flush()
  }

  flush()
  return chunks
}

Note what it already does right: it tracks the heading path, so a chunk knows it came from Returns → Damaged goods. That is the one thing that is annoying to add later, and it is why the schema in the previous chapter has a column for it.

Step 2: embed

// src/ingest/embed.ts
import { config, secrets } from '../config.js'

/** Batched, because one request per chunk is slow and rate-limited.
 *  Retries and backoff arrive in chapter 134; this version fails loudly. */
export async function embed(texts: string[]): Promise<number[][]> {
  const response = await fetch('https://api.openai.com/v1/embeddings', {
    method: 'POST',
    headers: {
      'content-type': 'application/json',
      authorization: `Bearer ${secrets.openaiKey}`,
    },
    body: JSON.stringify({ model: config.embedding.model, input: texts }),
  })

  if (!response.ok) {
    throw new Error(`embedding failed: ${response.status} ${await response.text()}`)
  }

  const json = await response.json()
  // The API returns results in request order, but the index is returned too —
  // sort by it rather than trusting the order. This has bitten people.
  return json.data
    .sort((a: { index: number }, b: { index: number }) => a.index - b.index)
    .map((d: { embedding: number[] }) => d.embedding)
}

/** pgvector's text input format. */
export const toVector = (v: number[]) => `[${v.join(',')}]`

Step 3: store and search

// src/ingest/run.ts — ingest one file.
export async function ingestFile(filePath: string, meta: DocumentMeta) {
  const markdown = await fs.readFile(filePath, 'utf8')
  const chunks = chunkMarkdown(markdown)
  const vectors = await embed(chunks.map((c) => contextualise(c)))

  await db.transaction(async (tx) => {
    await tx.query(
      `insert into documents (id, source, source_ref, title, updated_at, classification, account_id)
       values ($1, $2, $3, $4, now(), $5, $6)
       on conflict (id) do update set updated_at = now(), title = excluded.title`,
      [meta.id, meta.source, meta.sourceRef, meta.title, meta.classification, meta.accountId],
    )
    // Replace rather than append: re-ingesting a changed document must not
    // leave the old chunks behind. Chapter 34 does this incrementally.
    await tx.query(`delete from chunks where document_id = $1`, [meta.id])

    for (const [i, chunk] of chunks.entries()) {
      await tx.query(
        `insert into chunks (document_id, ordinal, heading_path, content, tokens, embedding)
         values ($1, $2, $3, $4, $5, $6)`,
        [meta.id, chunk.ordinal, chunk.headingPath, chunk.content,
         Math.ceil(chunk.content.length / 4), toVector(vectors[i])],
      )
    }
  })

  console.log(`ingested ${meta.title}: ${chunks.length} chunks`)
}

/** What gets embedded is not the same as what gets stored. Prefixing the
 *  heading path measurably improves retrieval on fragments that read as
 *  ambiguous alone — "within 30 days" means nothing without its heading. */
function contextualise(chunk: Chunk): string {
  return chunk.headingPath.length
    ? `${chunk.headingPath.join(' > ')}\n\n${chunk.content}`
    : chunk.content
}

That contextualise function is a small thing with a disproportionate effect, and it is the first example of a pattern that recurs throughout section 4: what you embed and what you show the model do not have to be the same text.

Search is one query.

// src/retrieve/search.ts
export interface Retrieved {
  id: number
  documentId: string
  title: string
  headingPath: string[]
  content: string
  score: number
}

export async function retrieve(
  question: string,
  options: { scope: Scope; limit?: number },
): Promise<Retrieved[]> {
  const [vector] = await embed([question])

  // The permission filter is in the query. Not in the prompt. See chapter 13.
  const { rows } = await db.query(
    `select c.id, c.document_id, d.title, c.heading_path, c.content,
            1 - (c.embedding <=> $1) as score
       from chunks c
       join documents d on d.id = c.document_id
      where d.deleted_at is null
        and (d.classification = 'public_internal'
             or (d.classification = 'account' and d.account_id = any($2)))
      order by c.embedding <=> $1
      limit $3`,
    [toVector(vector), options.scope.accountIds, options.limit ?? 12],
  )

  return rows.map(toRetrieved)
}

<=> is pgvector's cosine distance operator. Lower is closer, which is why the order by has no desc and the score is 1 - distance. Getting this backwards returns the least relevant chunks in the corpus, confidently, and the symptom is answers that are not merely wrong but bizarre.

Step 4: generate with a citation

<!-- src/generate/prompts/answer.md -->
You are an internal assistant for a distribution company. Answer the question
using only the sources below.

Rules:
- Every factual claim must cite a source, as [1], [2], matching the numbers below.
- If the sources do not contain the answer, say exactly: "I could not find this
  in the documents available to you." Do not answer from general knowledge.
- Quote figures, dates and thresholds exactly as written.

Sources:
{{sources}}

Question: {{question}}

// src/generate/answer.ts
export async function generate(question: string, chunks: Retrieved[]) {
  if (chunks.length === 0) {
    // Not a model call. There is nothing to ground an answer in, and asking
    // anyway is how an unsourced answer gets produced.
    return { text: 'I could not find this in the documents available to you.', sources: [] }
  }

  const sources = chunks
    .map((c, i) => `[${i + 1}] ${c.title}${c.headingPath.length ? ` — ${c.headingPath.join(' > ')}` : ''}\n${c.content}`)
    .join('\n\n')

  const response = await callModel(prompt('answer', { sources, question }))

  return {
    text: response,
    // Only the sources actually cited are returned to the interface. Chapter 62
    // makes this robust; a regex is enough today.
    sources: [...new Set([...response.matchAll(/\[(\d+)\]/g)].map((m) => Number(m[1])))]
      .map((n) => chunks[n - 1])
      .filter(Boolean),
  }
}

The early return on an empty result set is the most valuable four lines in the file. It costs nothing and it eliminates the entire class of failure where the retriever found nothing and the model answered anyway from what it happened to know about returns policies in general.

Running it

$ npm run ingest -- ./samples/returns-policy.md
ingested Returns and Credits Policy: 9 chunks

$ npm run ask -- "how long do customers have to return damaged goods?"
Damaged goods may be returned within 30 days of delivery, provided the damage is
reported within 48 hours of receipt [1]. Items returned after 30 days are
assessed case by case and may attract a restocking fee of 15% [2].

Sources:
  [1] Returns and Credits Policy — Returns > Damaged goods
  [2] Returns and Credits Policy — Returns > Late returns

That is the system. Everything from here makes it work on 2,400 documents instead of one, for 24 people with different permissions instead of one scope object, and measurably rather than by looking at it.

The failure mode you will hit in the next hour

Ask it a question the document does not answer. Something adjacent, like "what is the restocking fee for electronics?" when the policy mentions a restocking fee but never mentions electronics.

A naive build answers it. The chunk about restocking fees is still the closest match in the corpus — distance is relative, and the nearest chunk to a question is returned whether or not it is relevant at all. The model receives a plausible-looking source about restocking fees and writes a confident sentence about electronics.

This is the single most common defect in systems like this, and it has two causes worth separating:

Cause	Fix	Where
Irrelevant chunks returned because nothing better exists	A score floor, so low-similarity results are discarded	Chapter 41
The model filling gaps from general knowledge	Explicit abstention instruction, and measuring it	Chapters 63 and 111

The score floor is the one you can add right now:

// A floor, chosen by looking at the scores on questions you know the answers to.
// 0.3 is a starting point for this embedding model, not a universal constant.
const relevant = rows.filter((r) => r.score >= 0.3)

⚠️ Do not treat that number as given. The right floor depends on the embedding model and the corpus, and chapter 50 derives it from a test set instead of intuition. A floor chosen by guessing is still better than no floor.

Key points

Cross every seam once at the smallest scale. Mediocre end to end beats excellent in one component.
Track the heading path through the chunker from the first commit.
What you embed and what you show the model are allowed to differ, and prefixing the heading path helps.
The permission filter goes in the SQL where clause, never in the prompt.
Return early when retrieval finds nothing. Do not let the model answer unsourced.
Distance is relative: the nearest chunk is returned even when nothing is relevant. Add a score floor.

Next: what success looks like — turning the five targets from chapter 5 into numbers that can be measured on a Tuesday, before the scope grows.