Chapter 53 · The model layer

The system prompt as an engineering artifact

Not a string literal and not a text box. A versioned, reviewed, tested file with a known behaviour profile — and the seven sections a retrieval prompt actually needs.

11 min read

Chapter 8 established that prompts live in files. This chapter is the argument for that and what goes in them.

The claim: a system prompt has the properties of code and none of the tooling, so you have to supply the tooling. It determines behaviour, it breaks when changed, it has to be reviewed, and a regression in it is indistinguishable from a model regression unless both are versioned.

The seven sections

A retrieval prompt needs these, in roughly this order. Each is doing a specific job.

<!-- src/generate/prompts/answer.md -->

<!-- 1. ROLE. One or two lines. Enough to set register; no biography. -->
You are an internal assistant for a distribution company. You answer questions
from the company's own documents for staff who are working, quickly.

<!-- 2. THE TASK, stated once and unambiguously. -->
Answer the question using only the sources below.

<!-- 3. HARD RULES. Numbered, imperative, testable. Each one maps to an eval. -->
Rules:
1. Every factual claim must cite a source as [1], [2], matching the numbers below.
2. If the sources do not contain the answer, reply with exactly:
   "I could not find this in the documents available to you."
   Do not answer from general knowledge.
3. Quote figures, dates, thresholds and identifiers exactly as written.
4. If two sources disagree, say so and cite both. Do not choose between them.
5. If a source is marked superseded, use the current text and say it was amended.
6. If a source carries a "last updated" note older than the question implies,
   give the date and suggest confirming.

<!-- 4. FORMAT. What the output looks like, shown rather than described. -->
Answer in one or two short paragraphs. Lead with the answer. No preamble.

<!-- 5. EDGE CASES, as explicit branches. These are the ones measurement found. -->
If the question asks for a total, average or count across many records, say that
you can only see a selection of sources and a complete lookup is needed.

If the question assumes a value ("is it still 12%?"), state the current value
from the sources even when it differs from the one in the question.

<!-- 6. THE SOURCES. Delimited, numbered, last before the question. -->
Sources:
{{sources}}

<!-- 7. THE QUESTION. Last, so it is adjacent to the generation point. -->
Question: {{question}}

Four of those sections exist because a measurement produced them, not because they seemed wise.

Rule 4 came from the two live policy versions. Before it, the model silently picked one.

Rule 5 came from the 340 contract amendments.

The aggregate branch came from the average of six rows.

The assumption branch came from the rewrite that answered the question.

Every rule in a mature prompt should be traceable to a failure it prevents. A rule nobody can name a failure for is a rule nobody can test, and it is diluting the ones that matter.

Ordering, and why it is not arbitrary

Two forces pull in opposite directions, and the resolution is measurable.

Caching wants everything stable at the front, which the ordering above gives: role, task and rules are identical for every question, so they are a cacheable prefix. On Atlas that is roughly 600 tokens cached across 800 questions a day.

Instruction recency wants the rules near the generation point. With six chunks and 2,520 tokens the distance is small and the effect is negligible — the position test showed nothing at six chunks.

// Measure it rather than reasoning about it. Two orderings, same cases.
// Atlas, 6 chunks: rules-first 0.91 correct, rules-last 0.91 correct.
// At 20 chunks the gap opened to 0.88 against 0.91 — so for a system that
// sends many chunks, repeat the critical rules after the sources.

The pragmatic answer for a large context is to do both: the full rules as a cached prefix, and a two-line reminder of the two most important ones after the sources.

Testing a prompt

A prompt change is a behaviour change, and it gets the same treatment as a code change.

// evals/prompt.test.ts — fast, mechanical, no judge. These run on every PR.
describe('answer prompt', () => {
  test('cites every claim', async () => {
    const answer = await generate(Q.returns, SOURCES.returns)
    expect(answer.text).toMatch(/\[\d+\]/)
    expect(validCitations(answer, SOURCES.returns)).toBe(true)
  })

  test('uses the exact abstention phrase', async () => {
    const answer = await generate(Q.electronics, SOURCES.returns)
    expect(answer.text.trim()).toBe(
      'I could not find this in the documents available to you.')
  })

  test('reports conflicting sources rather than choosing', async () => {
    const answer = await generate(Q.returns, [SOURCES.v3, SOURCES.v4])
    expect(answer.text).toMatch(/\[1\]/)
    expect(answer.text).toMatch(/\[2\]/)
    expect(answer.text).toMatch(/differ|conflict|two versions|both/i)
  })

  test('prefers the source over its prior', async () => {
    // Counterfactual, from chapter 51.
    const answer = await generate(Q.returns, [SOURCES.ninetyDays])
    expect(answer.text).toContain('90')
    expect(answer.text).not.toContain('30 days')
  })

  test('refuses to aggregate from a sample', async () => {
    const answer = await generate(Q.averageDiscount, SOURCES.sixRows)
    expect(answer.text).toMatch(/selection|complete lookup|cannot count/i)
  })
})

⚠️ These are non-deterministic, so they need care. Chapter 58 covers temperature; the practical rule is temperature 0 for tests and n=3 with a majority requirement for anything that still varies.

// A flaky prompt test is a signal, not a nuisance: a rule followed two
// times in three is a rule that fails a third of the time in production.
export async function majorityOf(n: number, fn: () => Promise<boolean>) {
  const results = await Promise.all(Array.from({ length: n }, fn))
  const passes = results.filter(Boolean).length
  if (passes > 0 && passes < n) {
    logger.warn('prompt.nondeterministic', { passes, of: n })
  }
  return passes > n / 2
}

Versioning, and attributing a regression

Without a prompt version in the logs, a quality drop has two candidate causes and no way to distinguish them.

// Every answer records what produced it.
const PROMPT_VERSION = createHash('sha256')
  .update(fs.readFileSync(promptPath, 'utf8')).digest('hex').slice(0, 8)

logger.info('answer', {
  questionId,
  promptVersion: PROMPT_VERSION,
  model: config.generation.model,
  retrievalConfigVersion: RETRIEVAL_CONFIG.measuredOn,
  grounded, citationsValid,
})

-- Which makes the question answerable in one query.
select prompt_version, model, count(*),
       round(avg(grounded::int)::numeric, 3) as grounded,
       round(avg(citations_valid::int)::numeric, 3) as citations_valid
  from answer_log
 where answered_at > now() - interval '30 days'
 group by 1, 2 order by 3 desc;

A hash rather than a hand-maintained number, because a hand-maintained version is wrong the first time somebody edits a prompt without bumping it.

What does not belong in a prompt

Not in the prompt	Where instead
Permissions	The `where` clause
Source precedence	A filter before generation
Arithmetic	Computed in code
Freshness logic	A stamp on the chunk
Secrets or internal URLs	Nowhere

The pattern across all five: anything that must be true belongs somewhere deterministic. The prompt is for the parts that are genuinely judgement — how to phrase a conflict, when a caveat is warranted, how long an answer should be.

Length, and the cost of a rule

Every rule costs tokens on every request and competes for attention with the other rules.

Prompt	Tokens	Correct	Grounded	Citations valid
Minimal (3 rules)	180	0.84	0.93	0.91
Current (6 rules + 2 branches)	610	0.91	0.98	0.99
Expanded (14 rules + 6 branches)	1,840	0.89	0.97	0.98

The expanded version is worse. Eight of its fourteen rules were added speculatively, and they diluted the six that were measured.

// So: a rule enters the prompt with a test, or it does not enter.
// The test IS the justification, and a rule whose test passes without it
// comes back out.
test('rule 5 is load-bearing', async () => {
  const withRule = await generate(Q.paymentTerms, SOURCES.amended, { prompt: 'answer' })
  const without = await generate(Q.paymentTerms, SOURCES.amended, { prompt: 'answer-no-rule-5' })
  expect(withRule.text).toContain('45')
  expect(without.text).toContain('30')   // ⚠️ if this passes, remove rule 5
})

The failure mode: the prompt that was tuned to the model

The prompt reaches 0.91 correctness over several weeks of refinement. Some of those refinements are general — citation discipline, the conflict rule — and some are compensating for one model's quirks: an instruction not to start with "Based on the provided sources", a nudge about paragraph length, a phrase that stopped it hedging.

Then the model is upgraded. The general rules still work. The compensations are now instructions about behaviour that no longer exists, and two of them actively make the new model worse.

Quality drops and the prompt looks innocent, because every line in it was added for a reason that was true at the time.

<!-- Mark compensations, so they can be audited at upgrade time. -->
Rules:
1. Every factual claim must cite a source as [1], [2].    <!-- general -->
2. If the sources do not contain the answer, reply exactly: "..."  <!-- general -->
...
<!-- MODEL-SPECIFIC: mid-tier 2026-11. Re-test on every model change.
     Without this it opens with "Based on the provided sources" ~40% of
     the time. Delete if the new model does not. -->
Do not begin with a preamble referring to the sources.

Chapter 54 is about writing prompts that need fewer of these in the first place. The discipline that makes the comment worth writing:

Separate what the task requires from what this model needs. Only the first survives an upgrade, and only labelling tells you which is which.

Key points

A prompt has the properties of code and none of the tooling. Supply the tooling.
Seven sections: role, task, hard rules, format, edge cases, sources, question.
Every rule should be traceable to a failure it prevents. An untestable rule dilutes the ones that matter.
Put stable content first for caching. At six chunks instruction recency does not matter; at twenty it does, so repeat the critical rules after the sources.
Test prompts mechanically on every PR. A flaky prompt test means a rule that fails that often in production.
Version by content hash and log it with every answer, or a regression cannot be attributed between prompt and model.
Permissions, precedence, arithmetic and freshness do not belong in a prompt. Anything that must be true belongs somewhere deterministic.
Longer is not better: fourteen rules scored worse than six. A rule enters with a test or not at all.
Label model-specific compensations. Only task requirements survive an upgrade.

Next: prompt structure that survives a model upgrade.