Chapter 136 · Production

Deployment, environments and promoting a prompt change

Four things deploy on four cadences, so a staging environment needs a corpus. The prompt promotion pipeline, flags over releases, and why the rollback has to be one action.

8 min read

Chapter 127 said four things version independently and "roll back" is ambiguous. This chapter is the deployment machinery that makes the ambiguity manageable, and the hardest part is the environment question.

Staging needs a corpus, and that is the whole problem

// A conventional staging environment runs the same code against a small
// synthetic database. Here, the data IS the behaviour: against 400
// documents, retrieval has no competition, no near-misses and no
// ambiguity, so it passes everything and predicts nothing.

Staging corpus	Catches	Costs
Empty / synthetic	Crashes only	Nothing, and tells you nothing
1% sample	Crashes, gross retrieval errors	Nothing
Snapshot of production	Nearly everything	A copy of restricted data
Production, read-only	Everything	Nothing to copy, but it is production

Atlas uses the fourth, and it is the arrangement worth arguing for: staging reads the production corpus, read-only, through the same permission filter, under a service identity with no write access.

// Which works because the corpus is already read-only at serving time.
// The risk being avoided is not reading production data — the assistant
// does that constantly — it is writing to it, and that is a credential
// property rather than an environment property.
const STAGING = {
  corpus: 'production, read-only credential',
  vectorIndex: 'production, read replica',
  tools: 'sandbox backends only — ⚠️ no production writes, enforced by the allowlist',
  model: 'production model id, separate quota pool',
  users: 'the team, with their real groups',
}

The third line is the one that must be enforced rather than configured. Atlas's staging tool registry is built from a different allowlist, so a production write tool is not registerable rather than merely not configured — chapter 67's argument that the durable way to close a path is to remove it.

Four cadences

const CADENCE = {
  code:   'PR, review, CI, deploy. Ordinary.',
  prompt: 'its own pipeline — below. Several times a week.',
  model:  'rare, deliberate, chapter 109. Twice a year.',
  corpus: 'continuous, nobody deploys it. Chapter 138.',
}

The mismatch is the point. Code review gates code and nothing gates the corpus, which changes hourly and is 51% of failures. Chapter 137 is about making them legible together; here the consequence is that a deploy is never the only thing that changed between two observations.

Promoting a prompt change

// The prompt is an artifact (chapter 53) and gets a pipeline, not a
// deploy. Five stages, and each has a gate that can fail.
const PROMOTION = [
  { stage: 'author',    gate: 'the change names the failure it fixes' },
  { stage: 'smoke',     gate: 'the 24-case tier, 40 s' },
  { stage: 'full eval', gate: 'chapter 108 paired run, slice table read' },
  { stage: 'red team',  gate: 'chapter 116, 120 cases, no new effects' },
  { stage: 'canary',    gate: '10% of traffic for 24 h, guardrails only' },
]

// Prompts are versioned content, not code. They live in the repo, are
// reviewed in a PR, and ship as a flag value rather than a release —
// so the promotion and the rollback are both one action.
const active = await flags.get('prompt.version')      // 'v37'
const prompt = PROMPTS[active]

// ⚠️ All versions ship in the artifact. A rollback that requires a
// deploy is not a rollback, and the one time Atlas needed one at 9pm,
// the deploy pipeline was the slowest part of the incident.

The canary stage is where the guardrail distinction from chapter 113 matters: during 24 hours at 10% there is nowhere near the sample to detect a quality change, so the canary watches error rate, latency, refusal rate, escalation rate and the hazard count — things a day of traffic can move visibly — and the quality question was already answered by the full eval run.

Flags, and what they cost

// Every model-facing surface is a flag, which is what makes same-minute
// response possible.
const FLAGS = {
  'prompt.version': 'v37',
  'model.primary': 'resolved-model-id',
  'model.routing.enabled': true,
  'tools.writes.enabled': true,
  'memory.enabled': true,
  'mode': 'full',
}

// And the discipline that stops this becoming unmanageable: every flag
// has an owner, a default, and a removal date. Atlas reviews them
// monthly and has removed nine. A flag with no removal date is a
// permanent branch in the system's behaviour that nobody tests both
// sides of.

The eval suite runs against the flag state, not against the code defaults. That sounds obvious and was not: Atlas's suite ran against defaults for a month while production ran a different prompt version, so the suite was measuring a configuration nobody used.

Deploying code safely around a non-deterministic system

// Three properties that make an ordinary deploy safe here.

// 1. The prompt and the code version together. A code change that
//    assumes a new prompt field, shipped before the prompt is promoted,
//    breaks in exactly the way flags were supposed to prevent.
assert(PROMPTS[flags.get('prompt.version')].schemaVersion === CODE_SCHEMA_VERSION)

// 2. In-flight runs survive the deploy. Chapter 87's resumability is
//    load-bearing at deploy time: a run suspended for confirmation
//    must resume on the new version, which means the stored run format
//    is a compatibility surface.

// 3. Drain rather than cut. A 30 s drain on shutdown, because a
//    multi-step run in progress is 3-30 seconds of work the user is
//    waiting for.

The second is the one that bites, and chapter 90 catalogued the shape: a tool renamed in the code, a suspended run resumed against the new surface, and a stored run referencing a tool that no longer exists. The stored run format gets the same compatibility discipline as a database schema.

The deployment record

// Written on every change to any of the four, into one stream. It is
// the artifact chapter 112's drift analysis and chapter 125's incident
// response both read.
interface DeploymentEvent {
  at: Date
  kind: 'code' | 'prompt' | 'model' | 'corpus-migration' | 'flag'
  from: string
  to: string
  actor: string
  evalRunId?: string          // the run that justified it
}

One stream for all four is the design decision. A code deploy log, a flag audit and a prompt history in three places means an investigator correlating a quality step on Tuesday has to check three systems and will check two.

Key points

Staging needs a corpus, because the data is the behaviour. Four hundred synthetic documents have no near-misses and predict nothing.
Run staging against the production corpus read-only, through the same permission filter, with sandbox tool backends enforced by a different allowlist.
The risk is writing to production, not reading it. That is a credential property, not an environment property.
Four cadences: code per PR, prompt several times a week, model twice a year, corpus continuously and ungated. A deploy is never the only thing that changed.
Promote a prompt through five gates: named failure, smoke tier, paired eval with the slice table, red team, 24-hour canary.
Ship all prompt versions in the artifact and select by flag. A rollback needing a deploy is not a rollback.
The canary watches guardrails, not quality. A day at 10% cannot detect a quality change; the eval run already did.
Give every flag an owner, a default and a removal date. Review monthly.
Run the eval suite against the flag state. Atlas measured code defaults for a month while production ran something else.
Assert the prompt's schema version against the code's, keep the stored run format compatible across deploys, and drain for 30 seconds.
Write code, prompt, model, corpus and flag changes into one deployment stream. Three systems means an investigator checks two.

Next: versioning prompts, models and the data together — the problem this chapter has been working around.