Chapter 58 · The model layer

Temperature, sampling, and determinism you can test

Temperature 0 is not deterministic, and testing a non-deterministic system needs a different method. What the sampling parameters actually do and how to make a flaky test informative.

10 min read

Every test in this section so far has assumed a stable answer. This chapter examines that assumption, which does not hold, and gives the testing method that works anyway.

What the parameters do

A model produces a probability distribution over the next token. Sampling parameters reshape it.

// temperature: flattens or sharpens the distribution.
//   0   — always the highest-probability token (greedy)
//   1   — the distribution as the model produced it
//   >1  — flatter; unlikely tokens become plausible
//
// top_p: nucleus sampling. Keep the smallest set of tokens whose
//   probabilities sum to p, renormalise, sample from those.
//
// top_k: keep the k highest-probability tokens.

For a grounded-answer task the guidance is short and the reasoning matters more than the number.

export const generation = {
  temperature: 0,
  // ⚠️ Do not set top_p alongside temperature 0. They interact, the
  // interaction differs by provider, and at temperature 0 top_p has nothing
  // to do. Setting both is how two providers behave differently for reasons
  // nobody can find.
} as const

Temperature 0 is right here because there is no creative latitude in the task. The answer is determined by the sources; variation between runs is variation in how faithfully the sources were read, and none of it is wanted.

Why temperature 0 is not deterministic

Three causes, and knowing which one you are seeing changes what you do.

Floating-point non-associativity

(a + b) + c ≠ a + (b + c) in floating point. GPU kernels sum in an order that depends on batch composition and scheduling, so logits differ in the last bits between runs. When the top two tokens are within that margin, greedy decoding picks differently.

// Which makes divergence sharply non-uniform: it happens at the points
// where the model was genuinely undecided, and then compounds.

Batching and infrastructure

Your request is batched with others. Batch size and composition change the kernel path, and therefore the rounding. Nothing about your request changed.

Provider-side changes

A pinned model is the weights, not necessarily the serving stack. A kernel optimisation or a different accelerator generation changes outputs at the margin without changing the version string — which is why chapter 54's canary is a warning rather than a gate.

// Measure your own rate before designing around it.
export async function divergenceRate(question: string, sources: Retrieved[], n = 20) {
  const answers = await Promise.all(
    Array.from({ length: n }, () => generate(question, sources, { temperature: 0 })),
  )
  const distinct = new Set(answers.map((a) => a.text.trim()))
  return { distinct: distinct.size, n }
}
// Atlas, mid tier, 20 runs across 30 questions:
//   22 questions: 1 distinct answer
//    6 questions: 2 distinct answers
//    2 questions: 3+ distinct answers

Eight of thirty questions produced more than one answer at temperature 0. The useful follow-up is that the variation was almost never semantic: the facts and citations matched; the wording differed. That distinction is what the testing method below is built on.

Testing a non-deterministic system

Three rules.

Assert on properties, not on strings

// ❌ Brittle, and it will fail on a wording change that is not a regression.
expect(answer.text).toBe('Damaged goods may be returned within 30 days [1].')

// ✅ Properties that must hold whatever the wording.
expect(answer.found).toBe(true)
expect(answer.text).toContain('30 days')
expect(answer.citations).toEqual([1])
expect(answer.text).not.toMatch(/14 days/)

The exceptions are strings the system depends on exactly: the abstention phrase is matched exactly on purpose, because a paraphrased refusal breaks the isAbstention check downstream. Asserting on it exactly is how a paraphrase gets caught.

Run n times and require a majority

// A property that holds 2 runs in 3 is a property that fails a third of the
// time in production. The test should say so rather than passing or
// flaking at random.
export async function assertStable<T>(
  n: number, fn: () => Promise<T>, predicate: (v: T) => boolean,
) {
  const results = await Promise.all(Array.from({ length: n }, fn))
  const passes = results.filter(predicate).length

  if (passes === n) return
  if (passes === 0) throw new Error(`property never held (0/${n})`)
  // ⚠️ The informative case. A partially-holding property is a real finding
  // and must not be reported as a flaky test.
  throw new Error(
    `property held ${passes}/${n} times — non-deterministic behaviour, ` +
    `expect roughly ${Math.round((1 - passes / n) * 100)}% failures in production`,
  )
}

test('abstains on an unanswerable question', async () => {
  await assertStable(5,
    () => generate(Q.electronics, SOURCES.returns),
    (a) => a.found === false && isExactAbstention(a.text))
})

Separate the eval suite from the test suite

	Tests	Evals
Question	Did behaviour break?	How good is it?
Cases	~20	120+
Runs	n=3 or 5, majority	n=1
Gate	Blocks the PR	Threshold, per chapter 48
Time	Under a minute	Minutes

Conflating them produces a suite too slow to run on every commit and too small to measure quality — the worst of both.

Where a higher temperature is right

Three places in a system like this, and the answer path is not one of them.

// 1. Generating eval cases or test data: variation is the point.
const variants = await generate(prompt, { temperature: 0.9, n: 5 })

// 2. Self-consistency on a hard judgement: sample several, take the
//    majority. Expensive, and genuinely improves hard multi-hop planning.
const plans = await Promise.all([1, 2, 3].map(() =>
  planNextHop(question, context, { temperature: 0.7 })))
const chosen = majorityVote(plans)

// 3. Suggesting follow-up questions, where identical suggestions every time
//    are a worse experience and nothing depends on them.

Self-consistency is worth measuring on the multi-hop subset. On Atlas it improved sequential-hop correctness from 0.86 to 0.93 for three times the planning cost — applied only to the 6% of requests that plan, which is an acceptable trade and would not be at 100%.

Seeds, and what they buy

Some providers accept a seed and promise best-effort reproducibility.

// Useful for debugging a specific answer. ⚠️ Not a guarantee, and not a
// substitute for property-based assertions — a seeded test that passes
// tells you nothing about the unseeded production path.
const answer = await generate(question, sources, { temperature: 0, seed: 42 })

Use a seed to reproduce a reported bad answer. Do not use one in the eval suite, because it measures one sampling path rather than the behaviour.

The failure mode: the flaky test that was deleted

A test asserting the exact abstention phrase passes locally and fails in CI roughly one run in six. It is annoying, it blocks unrelated work, and after the third occurrence it is marked skipped with a note saying "flaky, model non-determinism".

The test was correct. One answer in six was not using the exact phrase — which means isAbstention() returned false for those, which means they were logged as answers rather than abstentions, which means the abstention metric was overstating by the same amount. And found: false with non-matching text is the consistency failure that returns a fallback, so one abstention in six was being shown as a validation failure to the person who asked.

The flake was the finding.

// Three defences.

// 1. Never skip a non-deterministic test. Convert it to a rate assertion,
//    which records the real behaviour instead of hiding it.
test('abstention phrase is exact', async () => {
  const rate = await measureRate(20, () => generate(Q.electronics, SOURCES.returns),
    (a) => isExactAbstention(a.text))
  // The committed number is what it currently does, raised deliberately.
  expect(rate).toBeGreaterThan(0.95)
})

// 2. Make the downstream check tolerant where exactness is not required,
//    so a near-miss degrades rather than breaking.
export const isAbstention = (text: string) =>
  /^i could not find this in the documents available to you/i.test(text.trim())

// 3. Fix the cause: an abstention is a fixed string, so emit it in code
//    rather than asking the model to reproduce it.
if (!answer.found) {
  return { ...answer, answer: ABSTENTION_TEXT }   // ⚠️ the real fix
}

The third is the one that removes the problem rather than measuring it. Any output that must be exact should be produced by your code, with the model deciding only whether to produce it — found: false is a boolean the model is good at, and the sentence is a constant.

If a string must be exact, do not ask a sampler for it. Ask for the decision and write the string yourself.

Key points

Temperature 0 for grounded answers: the task has no creative latitude, and variation is unfaithfulness.
Do not set top_p alongside temperature 0. The interaction is provider-specific and buys nothing.
Temperature 0 is not deterministic — floating-point non-associativity, batching, and provider-side serving changes.
Measure your own divergence rate. Atlas: 8 of 30 questions varied, almost always in wording rather than facts.
Assert on properties, not strings, except where the string is a contract.
Run n times and require a majority. A property holding 2 in 3 is a finding, not a flaky test.
Keep tests (20 cases, n=5, blocking) separate from evals (120 cases, n=1, threshold).
Higher temperature belongs in test-data generation, self-consistency on hard planning, and follow-up suggestions.
Use seeds to reproduce a reported bug, never in the eval suite.
If a string must be exact, emit it from code and ask the model only for the decision.

Next: streaming responses — and what it changes for validation, citations and error handling.