Chapter 127 · Production

What production means for a system that is non-deterministic

The operational definitions that do not transfer: what an SLO looks like when the same input gives different output, what a rollback restores, and which four properties you can actually promise.

8 min read

Everything in section 10 was about building correctly. This section is about running the result, and it opens with a definition problem that has to be settled before any of the operational work makes sense.

Production practice assumes determinism in places it never states. "Reproduce the bug", "roll back to the last good version", "this request succeeded" — each assumes the same input yields the same output. None of that holds here.

What is actually non-deterministic

Worth being precise, because the answer is "less than people assume" and the deterministic parts are where the operational grip is.

// Deterministic, given the same inputs and versions.
const DETERMINISTIC = [
  'Retrieval. Same query, same index, same filter -> same chunks.',
  'The prompt assembled from those chunks.',
  'Tool argument validation.',
  'Permission decisions.',
  'Every policy-layer check from section 10.',
]

// Not deterministic.
const NOT = [
  'The model’s output, even at temperature 0.',
  'Which tools it decides to call, and in what order.',
  'How many steps a loop takes.',
  'Whether it refuses.',
]

The first list is most of the system by line count, and it is where reproduction, rollback and testing work normally. Chapter 107 separated the layers for evaluation; the same separation is what makes the thing operable.

// Temperature 0 is not determinism, and the gap costs people a day.
// Floating-point reduction order varies with batch composition on the
// provider's hardware, so the same request can take a different path at
// a near-tie. Atlas, 500 repeats of one question at temperature 0:
//   identical output:              94.2%
//   differed in wording only:       5.4%
//   differed in a factual claim:    0.4%

Two in five hundred differed in substance. Small, non-zero, and it means "it worked when I ran it" is evidence rather than proof.

Four properties you can promise

// An SLO on answer quality is not operable: it needs a judge, it arrives
// hours late, and nobody can be paged on it. These four can be measured
// per request, in real time, and acted on.
const SLOS = {
  availability: 'a response or an honest failure, 99.5% of requests',
  latency:      'p50 < 3 s, p95 < 10 s, p99 < 30 s',
  groundedness: 'every factual claim carries a resolvable citation, 99.9%',
  safety:       'zero cross-user disclosures; zero unconfirmed writes',
}

The third is the one that converts a quality property into an operational one. Is the answer correct is unanswerable in real time; does every claim carry a citation that resolves to a chunk this caller may see is a deterministic check on every response, which chapter 122 already built. It does not guarantee correctness and it guarantees checkability, which is the operable half.

The fourth is reported as a count. A rate implies a tolerable number.

// And the one deliberately absent.
//   ❌ "answer quality above 0.85"
// It cannot be measured per request, it depends on a judge version
// (chapter 112), and nobody can be paged on it. Quality is tracked on
// the chapter 112 weekly run, which is a different instrument with a
// different cadence.

What a rollback restores

// Four things version independently, and a "rollback" usually moves one.
interface SystemVersion {
  code: string              // git sha
  prompt: string            // chapter 53's artifact
  modelId: string           // resolved, not an alias
  corpusSnapshotId: string  // the data
}

Reverting the code does not revert the prompt if the prompt ships separately, does not revert the model if the provider moved it, and never reverts the corpus. Chapter 137 is the whole problem; the point here is that "roll back" is ambiguous and the ambiguity is the incident.

In an ordinary service the deployed artifact is the system. Here it is one of four, and the other three change without a deploy.

Reproduction

// Three levels, and only the first is available in the usual sense.
const LEVELS = {
  exact:        'impossible. Accept it.',
  contextual:   'same prompt, same context, same config -> the same class of answer',
  deterministic:'the retrieval, the policy decisions, the tool arguments -> identical',
}

// Which makes the deterministic prefix the thing to capture. Given the
// audit record from chapter 124, retrieval and every policy decision
// replay exactly, and most bad answers are explained there rather than
// in the generation.
const replay = await replayDeterministic(auditRun)
// -> the same 12 chunks, the same permission decisions, the same tool
//    arguments. Chapter 129 is built on this.

Chapter 107 attributed 51% of failures to retrieval. Those reproduce perfectly, which is a far better operational position than the framing "you cannot reproduce anything" suggests.

What "healthy" means

// A health check that calls the model is expensive, slow and flaky. One
// that does not is useless. Atlas splits it.
export async function health(): Promise<Health> {
  return {
    // Cheap, every 10 s. Dependencies only.
    liveness: await Promise.all([pingDb(), pingVectorIndex(), pingCache()]),
    // Expensive, every 5 min, one fixed question with a known answer.
    readiness: await canary(),
  }
}

// The canary is a real end-to-end run against a fixed document, asserting
// a citation resolves rather than asserting exact text. It catches an
// expired credential, a provider outage, an empty index after a bad
// ingest, and a prompt that no longer produces valid structured output --
// which are the four things that have actually taken Atlas down.

Asserting a citation resolves rather than matching text is the detail that makes the canary stable. A canary asserting exact output flaps on the 5.4% wording variance and gets muted within a fortnight.

The mental model

A deterministic prefix, a non-deterministic middle, a deterministic suffix. Everything operable lives in the green boxes, and the engineering move throughout this section is to make the yellow box as narrow as possible — which is the same move section 10 made for different reasons.

Key points

Most of the system is deterministic: retrieval, prompt assembly, validation, permissions. Only the model's output and the control flow it chooses are not.
Temperature 0 is not determinism. 500 repeats gave 94.2% identical output and 0.4% differing in a factual claim.
Promise availability, latency, citation integrity and safety. Quality cannot be measured per request and nobody can be paged on it.
Report safety violations as a count. A rate implies a tolerable number.
Four things version independently: code, prompt, model, corpus. "Roll back" is ambiguous, and the ambiguity is the incident.
Exact reproduction is unavailable; deterministic-prefix replay is not. Most bad answers are explained there, and 51% of failures are retrieval.
Split the health check: cheap dependency liveness, and a slower canary that runs end to end.
Have the canary assert that a citation resolves, not that text matches. An exact-match canary flaps on wording variance and gets muted.

Next: observability: traces, spans and what to log — instrumenting the yellow box.