Chapter 137 · Production

Versioning prompts, models and the data together

Four axes that move independently, the composite version that makes any observation reproducible, and the honest limit: you can version three of them and only snapshot the fourth.

8 min read

Two chapters have deferred to this one. Chapter 112 needed provenance to tell drift from regression; chapter 136 needed a definition of "roll back". Both need the same thing: a single identifier that says what the system was.

The composite version

// Four axes, one identifier. Every observation — an answer, an eval run,
// an audit row, a telemetry span — carries it.
interface SystemVersion {
  code: string               // git sha
  prompt: string             // content hash of the active prompt
  model: string              // RESOLVED id, never an alias
  corpus: string             // snapshot id
  embedding: string          // the embedding model; moves rarely, breaks everything
}

const composite = (v: SystemVersion) =>
  `${v.code.slice(0, 7)}.${v.prompt.slice(0, 7)}.${v.model}.${v.corpus}.${v.embedding}`

// Five, not four, because the embedding model is the axis people forget
// and the one with the worst failure mode. Changing it invalidates every
// vector in the index, and until a full re-embed completes, retrieval is
// comparing vectors from two different spaces — which does not error.

Only three of the five are versions

An honest account, because the usual presentation implies more control than exists.

Axis	Can you pin it?	Can you roll back?
Code	Yes	Yes
Prompt	Yes	Yes, by flag
Embedding model	Yes	Expensive — a full re-embed
Model	Only as a resolved id	No. The provider may retire it
Corpus	Only as a snapshot	No. It is a living system of record

// The corpus is the one that defeats the metaphor. You cannot roll back
// a contract amendment, and you would not want to — the new terms are
// the correct answer. A "snapshot" is a read-only copy for
// reproduction, not a state the system can return to.

That distinction resolves a lot of confused incident conversations. When quality falls because the corpus changed, the remedy is never to revert the corpus; it is to fix retrieval, chunking or freshness so the system handles the new state. Chapter 112's snapshot diagnostic exists to tell you which situation you are in, not to offer a rollback.

Corpus snapshots

// Cheap, and the thing that makes everything else reproducible. Atlas's
// monthly snapshot is a few hundred megabytes.
interface CorpusSnapshot {
  id: string                 // '2026-11-01'
  at: Date
  documentCount: number
  chunkCount: number
  embeddingModel: string
  contentHash: string        // over sorted chunk ids + content hashes
}

// Taken monthly, retained 13 months, plus one before every migration.
// The content hash is what makes "the corpus changed" detectable
// without diffing 340,000 chunks, and the per-source breakdown is what
// makes it actionable.
const bySource = await snapshotDiff('2026-10-01', '2026-11-01')
// -> contracts: 41 changed, 3 added
//    price-list: 1,204 changed        <- the one that moved
//    policies: 0

A monthly snapshot gives month-granularity reproduction, which is enough for drift analysis and not enough for a four-day-old incident. Atlas's compromise: the audit record stores a content hash of the specific chunks retrieved on every run, so a single run's inputs are reproducible exactly even when the snapshot is weeks old.

// Which is the general technique: snapshot the whole thing coarsely,
// and fingerprint the specific inputs finely. The second is cheap
// because a run touches twelve chunks.
retrieved: [{ chunkId, contentHash, score, aclGroup }]

Compatibility between axes

// Not every combination is valid, and the invalid ones fail in
// interesting ways rather than loudly.
const CONSTRAINTS = [
  'prompt.schemaVersion must match code.schemaVersion',
  'index.embeddingModel must match query.embeddingModel',   // ⚠️ silent if violated
  'prompt.toolNames must be a subset of code.toolRegistry',
  'storedRun.formatVersion must be readable by code',       // chapter 87's resume
]

// Checked at startup, so an incompatible combination refuses to serve
// rather than serving degraded. The embedding one is the only check in
// this series that Atlas made fatal with no override.
if (index.embeddingModel !== config.embeddingModel) {
  throw new Fatal(`index built with ${index.embeddingModel}, configured ${config.embeddingModel}`)
}

The reason it is fatal: a mismatch produces plausible-looking results with badly degraded recall and no error anywhere. It is the single worst silent failure in the stack, and the cost of refusing to start is an obvious outage instead of a subtle one.

Migrating the embedding model

// The expensive axis, done once in Atlas's history. The procedure, in
// the order that keeps the system serving.
const MIGRATION = [
  '1. Build the new index alongside the old. Days of ingestion spend.',
  '2. Evaluate retrieval on the chapter 49 test set against both.',
  '3. Shadow: query both, serve the old, record divergence.',
  '4. Flip by flag. Keep the old index for a week.',
  '5. Re-baseline every eval number. They are not comparable across spaces.',
]

Step five is the one that surprises people. Retrieval scores, groundedness, the memory recall figures from section 8 — all of them were measured against one vector space, and a migration moves the floor under every historical number. Chapter 112's rule about marking chart boundaries applies here more than anywhere.

Atlas's migration: recall@12 went 0.94 to 0.96, which took three weeks and $1,100 of embedding spend for two points. Worth it for a corpus that will run for years, and it is a number worth having before anyone proposes it casually.

Reading a version in an incident

// What the composite buys, concretely. Someone reports a bad answer
// from four days ago.
const v = auditRun.version
// a1f3c2e.9b4d711.model-x-2026-08.2026-11-01.embed-v3

// Each field answers a question immediately:
//   code differs from current?    -> a deploy is a candidate
//   prompt differs?               -> check the promotion record
//   model differs?                -> the provider rolled; chapter 109
//   corpus differs?               -> expected; check the per-source diff
//   embedding differs?            -> a migration happened; nothing is comparable

Without the composite, each of those is a separate investigation against a separate system. With it, the first three are a string comparison, which is the difference between a ten-minute triage and an afternoon.

Key points

Five axes: code, prompt, model, corpus, embedding model. One composite identifier on every observation.
The embedding model is the fifth axis people forget, and a mismatch compares vectors from two spaces without erroring.
Only three of the five are genuinely versions. The model can be retired by the provider; the corpus is a living system of record.
You cannot roll back a contract amendment, and you would not want to. A corpus snapshot is for reproduction, not restoration.
Snapshot the corpus coarsely — monthly, a few hundred megabytes, 13 months — and fingerprint the specific retrieved chunks on every run.
Store a per-source diff between snapshots. "The price list moved 1,204 chunks" is actionable; "the corpus changed" is not.
Check cross-axis compatibility at startup and make the embedding mismatch fatal with no override. An obvious outage beats a subtle one.
An embedding migration means re-baselining every historical number. Atlas's took three weeks and $1,100 for two points of recall.
In an incident, the composite turns three separate investigations into a string comparison.

Next: scaling ingestion as the corpus grows — the axis nobody deploys and everybody depends on.