Chapter 54 · The model layer

Prompt structure that survives a model upgrade

A model upgrade should be a config change, not a rewrite. Four structural properties that make prompts portable, and the upgrade procedure that takes an afternoon.

11 min read

The previous chapter ended on compensations: instructions that exist because one model behaves a certain way. This chapter is about writing prompts that need fewer of them, and about the procedure for the upgrade when it comes.

Model upgrades are frequent and usually improvements. A system where upgrading means a week of prompt rework will stay on an old model for a year, which costs more in quality than any prompt refinement will recover.

Four properties of a portable prompt

1. Describe the outcome, not the mechanism

<!-- ❌ Mechanism: tells the model how to think. Model-specific, and it
     becomes wrong when the model's default behaviour changes. -->
Think step by step. First identify which sources are relevant. Then extract
the relevant facts. Then compose your answer. Finally add citations.

<!-- ✅ Outcome: tells the model what must be true of the result. Portable,
     because it is a property of the answer rather than of the process. -->
Every factual claim must cite a source as [1], [2]. If the sources do not
contain the answer, reply with exactly: "..."

Outcome instructions are also the only kind that can be tested, which is not a coincidence. A rule you can write an assertion for is a rule about the output.

2. Show the format, do not describe it

<!-- ❌ A description. Each model interprets "concise" differently, and the
     interpretation changes between versions. -->
Answer concisely in a professional tone with appropriate citations.

<!-- ✅ One short example. Unambiguous, and it costs about 40 tokens. -->
Answer in this shape:

  Damaged goods may be returned within 30 days of delivery, provided the
  damage is reported within 48 hours [1]. Later returns are assessed
  individually and may attract a restocking fee [2].

Atlas's measurement: replacing the description with the example reduced length variance across three models by about two thirds and removed four format-related compensations from the prompt.

3. Separate the portable core from the compensations

src/generate/prompts/
  answer.core.md          # task, rules, format. Never model-specific.
  answer.compat.md        # model-specific compensations, dated and labelled
  answer.md               # generated: core + compat for the active model

// Composition, so an upgrade is deleting a file rather than editing a prompt.
export function buildPrompt(name: string, model: string): string {
  const core = read(`${name}.core.md`)
  const compat = readIfExists(`${name}.compat.${model}.md`)
  // ⚠️ A missing compat file is normal and must not be an error. A new model
  // starts with none, and the upgrade procedure adds them only if a test fails.
  return compat ? `${core}\n\n${compat}` : core
}

<!-- answer.compat.mid-tier-2026-11.md -->
<!-- Added 2026-11-03. Without this, opens with "Based on the provided
     sources" in ~40% of answers. Test: prompt.test.ts > 'no preamble'. -->
Do not begin with a preamble referring to the sources.

Keeping the reason, the date and the test name in the file is what makes the upgrade audit possible. A compensation whose test now passes without it gets deleted, and that judgement takes seconds instead of an argument.

4. Make the output machine-checkable

A prompt whose output can only be judged by reading it cannot be regression-tested across models.

<!-- Citations as [n] rather than as prose. Parseable, so a test can assert
     on them, so an upgrade can be verified mechanically. -->
Cite as [1], [2]. Do not write "according to the returns policy".

This is the quiet reason the citation format in chapter 9 was chosen as [n]: not because readers prefer it, but because validCitations() can check it.

The upgrade procedure

An afternoon, and it is the same shape as every other evaluation in this series.

// scripts/model-upgrade.ts
export async function evaluateUpgrade(current: string, candidate: string) {
  // 1. The candidate against the CORE prompt only, no compensations.
  //    This is the honest baseline: what does the new model do unaided?
  const core = await scoreEndToEnd(cases, { model: candidate, prompt: 'core-only' })

  // 2. The candidate with the current compensations applied.
  const withCompat = await scoreEndToEnd(cases, { model: candidate, prompt: 'full' })

  // 3. The incumbent, for comparison.
  const baseline = await scoreEndToEnd(cases, { model: current, prompt: 'full' })

  return { baseline, core, withCompat, perCompat: await auditCompensations(candidate) }
}

// Each compensation, tested with and without, against the new model.
async function auditCompensations(model: string) {
  return Promise.all(COMPENSATIONS.map(async (c) => {
    const withIt = await runTest(c.testName, { model, include: [c.id] })
    const without = await runTest(c.testName, { model, include: [] })
    return {
      id: c.id,
      stillNeeded: withIt.pass && !without.pass,
      nowHarmful: !withIt.pass && without.pass,    // ⚠️ the dangerous case
      irrelevant: withIt.pass && without.pass,
    }
  }))
}

model upgrade: mid-tier-2026-11 -> mid-tier-2027-02

                    baseline   core only   with compat
correct               0.91       0.93         0.92
grounded              0.98       0.99         0.99
citations valid       0.99       1.00         1.00
abstains correctly    0.89       0.94         0.91

compensations (4):
  no-preamble        irrelevant    — the new model does not do this. DELETE.
  paragraph-length   irrelevant    — DELETE.
  hedging-nudge      NOW HARMFUL   — suppresses correct hedging. DELETE.
  exact-abstention   still needed  — KEEP.

The core only column beating with compat is the normal result and the reason this audit exists. Three of four compensations were worse than useless against the new model, and hedging-nudge was actively reducing correct abstention — the capability chapter 51 identified as the one worth paying for.

// The rule that falls out of it.
if (core.correct >= withCompat.correct) {
  logger.info('upgrade.compat_obsolete', {
    note: 'Ship the core prompt. Re-add compensations only where a test fails.',
  })
}

On a model upgrade, start from the core prompt and add compensations back only where a test fails. Carrying them forward by default is how a prompt accumulates instructions for behaviour that no longer exists.

Pinning, and knowing when the floor moved

// Pin a dated identifier. An alias that silently moves is the same problem
// as the embedding drift in chapter 39, with a larger blast radius.
export const config = {
  generation: {
    model: 'claude-sonnet-5-5',   // ⚠️ dated and pinned, never an alias
    pinnedOn: '2026-11-03',
    evaluatedAt: { correct: 0.91, grounded: 0.98, citationsValid: 0.99 },
  },
}

// And a canary for the case where a pinned model still changes underneath
// you — same pattern as chapter 39's embedding canary.
export async function modelCanary() {
  const answer = await generate(CANARY_QUESTION, CANARY_SOURCES, { temperature: 0 })
  const expected = await getStoredCanaryAnswer()
  if (!expected) { await storeCanaryAnswer(answer.text); return }

  if (normalise(answer.text) !== normalise(expected)) {
    logger.warn('model.canary_drift', {
      note: 'A pinned model produced different output at temperature 0. ' +
            'Re-run the eval suite before trusting quality numbers.',
    })
  }
}

⚠️ Temperature 0 is not a guarantee of identical output — chapter 58 covers why — so the canary is a warning rather than a gate. It is still the cheapest signal that something changed.

The portability test

Worth running once, before the first upgrade is needed: score the prompt against a model you are not using.

// A prompt that works on three models is portable. One that works on one
// is a prompt plus a dependency nobody has written down.
for (const model of ['small', 'mid', 'large', 'other-vendor']) {
  const scored = await scoreEndToEnd(cases, { model, prompt: 'core-only' })
  console.log(model, round(scored.correct), round(scored.citationsValid))
}

Model	Correct	Citations valid	Reading
Small	0.79	0.88	Below the bar, as in chapter 51
Mid (current)	0.91	0.99	—
Large	0.93	1.00	—
Other vendor, mid	0.89	0.97	Portable

The last row is the useful one. A core prompt scoring 0.89 on a model it was never written for means the prompt encodes the task rather than one model's habits — and it means a migration is available if pricing, residency or availability ever forces one.

The failure mode: the upgrade that was never attempted

The least visible failure in this chapter, because nothing breaks.

The model is pinned in 2026 and the pin works. Upgrading means re-running evals, auditing compensations and a deploy, and there is always something more urgent. Two years later the system is on a model that is three generations old, costs more per token than its successors, and abstains correctly 20 points less often than the current mid tier would.

Nobody decided this. It is the default outcome of an upgrade having no owner and no schedule.

// Make it scheduled work rather than a decision somebody has to initiate.
export const MODEL_REVIEW = {
  everyDays: 90,
  // The review is cheap because everything it needs already exists: the
  // eval suite from section 9, the compensation audit above, and the
  // config's recorded baseline.
  procedure: [
    'Score the current model. Confirm it still matches config.evaluatedAt.',
    'Score each available successor on the CORE prompt.',
    'Audit compensations against the best successor.',
    'If it wins on correctness, groundedness and abstention: upgrade.',
    'If it does not: record the numbers and the date, and stop.',
  ],
} as const

Recording the numbers when you decline an upgrade is the part that makes the next review cheap. "Evaluated 2027-02, successor scored 0.89 against 0.91, stayed" is a complete answer to "why are we still on this model?" and it takes one line.

Key points

An upgrade should be a config change. If it means a week of rework, you will stay on an old model for a year.
Describe the outcome, not the mechanism. Outcome rules are the only testable kind.
Show the format with a short example rather than describing it. It cut Atlas's cross-model length variance by two thirds.
Split the prompt into a portable core and dated, labelled compensations in separate files, each naming its test.
Make the output machine-checkable. [n] citations exist so an assertion can verify them.
On upgrade, score the core prompt first and add compensations back only where a test fails. Three of four were obsolete and one was harmful.
Pin a dated model identifier and run a canary at temperature 0.
Run the portability test early. A core prompt scoring well on an unused model encodes the task rather than one model's habits.
Schedule the model review. An upgrade with no owner does not happen, and the decline is worth recording too.

Next: few-shot examples — what they fix, what they cost, and where they stop helping.