Chapter 114 · Evaluation

Making the eval suite fast enough that people run it

A suite nobody runs is a suite that does not exist. Concurrency, caching, tiering and the numbers that took Atlas's run from 34 minutes to 6 — plus the report that makes the result readable in thirty seconds.

8 min read

Everything in this section is worth nothing if the suite takes forty minutes. At forty minutes it is run before releases; at six it is run before pushing; at nine seconds — which is what chapter 107's retrieval harness costs — it is run while editing.

The behavioural threshold is sharp and worth stating as a design target: a suite run must be shorter than the attention span of the person who triggered it. Past about five minutes they switch tasks, and a result that arrives after a context switch gets skimmed.

Atlas's suite went from 34 minutes to 6. Here is where the 28 went.

Where the time was

	Before	After	How
Sequential execution	19 min	2.5 min	Concurrency with a rate limiter
Judging everything	9 min	1.5 min	Mechanical scorers for 71%
Re-embedding the corpus	4 min	0	Snapshot with pre-computed embeddings
Re-running unchanged cases	2 min	0.5 min	Content-addressed cache
Report generation	20 s	20 s	Left alone
Total	34 min	6 min

Concurrency, which is the whole first half

// The naive loop is the single biggest cost and the easiest fix. The only
// subtlety is that the provider rate limit is the real constraint, so the
// pool has to be bounded and has to back off rather than fail.
export async function runAll(cases: EvalCase[], config: Config): Promise<CaseResult[]> {
  const limit = pLimit(CONCURRENCY)      // 12 for Atlas, set by the rate limit
  return Promise.all(cases.map((c) => limit(() => runWithRetry(c, config))))
}

// Backoff on 429 only, and never on a content error. Retrying a malformed
// request is chapter 77's lesson: an error that is not 'unavailable' or
// 'correctable' is a bug, and retrying it hides the bug and wastes the run.
async function runWithRetry(c: EvalCase, config: Config, attempt = 0): Promise<CaseResult> {
  try {
    return await run(c, config)
  } catch (e) {
    if (isRateLimit(e) && attempt < 5) {
      await sleep(2 ** attempt * 500 + Math.random() * 400)
      return runWithRetry(c, config, attempt + 1)
    }
    throw e
  }
}

Finding the right concurrency is an experiment, not a guess. Atlas's scaling:

Concurrency	Wall clock	429 rate
1	19 min	0%
8	3.1 min	0%
12	2.5 min	0.4%
24	2.4 min	11%
48	3.8 min	31%

Past 12 there is nothing left to gain and the retries start costing more than the parallelism buys. Forty-eight is slower than twelve, which is the shape every rate-limited workload has and which teams routinely discover by setting the number to 100.

Caching what cannot have changed

// Content-addressed on everything that could affect the result. A hit is
// only valid if literally nothing relevant differs, which makes the key
// long and the correctness obvious.
function cacheKey(c: EvalCase, config: Config): string {
  return sha256([
    c.id, hashCase(c),
    config.promptHash, config.modelId,
    config.embeddingModelId, config.corpusSnapshotId,
    config.retrievalParamsHash,
  ].join('|'))
}

On a prompt change, every case misses — the prompt hash is in the key — so the cache buys nothing on the run that matters most. It pays on the runs around it: re-running after a flaky failure, running the same config against a second case set, and the chapter 112 weekly drift run, where the config is identical and only the corpus moved.

// One rule that keeps it trustworthy: the cache is off by default on the
// run that produces a shipping decision. A stale hit that changes a ship
// decision costs more than every minute the cache has ever saved.
const useCache = process.env.EVAL_CACHE === '1' && !opts.forDecision

Tiering

// Not every run needs every case. Three tiers, chosen by what changed.
const TIERS = {
  smoke:  { cases: 24,  time: '40 s',  when: 'while editing' },
  core:   { cases: 120, time: '3 min', when: 'before pushing' },
  full:   { cases: 240, time: '6 min', when: 'before releasing, and nightly' },
}

// The smoke tier is not a random 24. It is the 24 cases that have
// historically failed first — selected by how often each case has been
// the earliest detector of a regression that the full suite later
// confirmed.
const smoke = cases
  .map((c) => ({ c, earlyDetections: history.earlyDetectionCount(c.id) }))
  .sort((a, b) => b.earlyDetections - a.earlyDetections)
  .slice(0, 24)
  .map((x) => x.c)

Atlas's smoke tier catches 71% of the regressions the full suite catches, in 40 seconds. That is the number that got the suite into people's editing loop, and the selection is recomputed monthly from the run history rather than curated by hand.

The report

Run time is half the problem. A six-minute run whose output takes fifteen minutes to interpret has not solved anything.

eval: prompt-v37 vs prompt-v36   240 cases   6m02s   $2.14

  OVERALL          0.859   +1.2  [-1.1, +3.4]   no evidence of change

  BY INTENT
    lookup         0.913   +0.3  [-2.0, +2.6]
    policy         0.841   +1.6  [-2.2, +5.3]
    comparison     0.794   +0.9  [-4.1, +6.0]
    out-of-scope   0.887  +15.1  [+8.2, +21.7]  ████ improved
    action         0.791   -9.4 [-16.1, -2.8]   ████ REGRESSED

  INVARIANTS       5/5 pass
  HAZARDS          0

  12 responses changed materially -> artifacts/diff-prompt-v37.html

Five properties make it readable in thirty seconds:

// 1. The headline carries the interval and the verdict in words, so
//    "+1.2" is never read as a win on its own.
// 2. Slices always, sorted by absolute change.
// 3. Invariants and hazards reported separately, never averaged in.
// 4. A pointer to the response diffs, which is where the understanding is.
// 5. Cost and wall clock, so the suite's own budget stays visible.

What not to optimise

// Three optimisations that cost more than they save.

// 1. Shrinking the case set. The interval widens, and a wider interval
//    means more changes land in "no evidence", which is the outcome that
//    wastes the most human time.

// 2. A cheaper judge. Chapter 106's calibration is specific to the judge
//    model. A cheaper one needs a full recalibration and usually comes
//    back with a lower kappa on exactly the criteria that matter.

// 3. Skipping the both-orders pairwise comparison. It halves the cost of
//    comparisons and restores a 12-point position bias.

Optimise the plumbing, never the measurement. Every minute saved by weakening the instrument is paid back the first time someone ships on a number that was not real.

That closes section 9. Everything from here on has a harness behind it: section 10's guardrails are measured by the invariants in this suite, and section 11's production work is monitored by the drift run.

Key points

A suite run must be shorter than the attention span of whoever triggered it. Past five minutes the result gets skimmed.
Atlas went 34 minutes to 6: concurrency 19→2.5, mechanical scorers 9→1.5, a corpus snapshot 4→0, caching 2→0.5.
Find the concurrency empirically. Atlas's 12 was optimal; 48 was slower than 12 because of retries.
Back off on rate limits only. Retrying a content error hides a bug and wastes the run.
Content-address the cache on everything that could change the result, and turn it off for runs that produce a shipping decision.
Tier the suite. A 24-case smoke tier selected by historical early detection catches 71% of regressions in 40 seconds.
Recompute the smoke selection monthly from run history rather than curating it.
Make the report readable in thirty seconds: interval and verdict in the headline, slices sorted by change, invariants separate, a link to the response diffs.
Never optimise by shrinking the set, cheapening the judge, or dropping the both-orders comparison.

Next, and the start of section 10: guardrails and security. Section 9 built the instrument that tells you whether the system is behaving. Section 10 is what stops it misbehaving in the first place — input validation, prompt injection, output filtering, rate limits, audit, and the controls that have to hold when the model does something nobody anticipated.