Chapter 113 · Evaluation

A/B testing agent changes with real users

The suite measures the answer; users measure whether it helped. Unit of assignment, the metrics that survive contact with reality, sample sizes you will not like, and what to do when the test is underpowered.

8 min read

Chapter 105 ruled out asking a judge whether a user would be satisfied. This is where that question gets answered, and it is a different discipline from everything before it.

The honest framing: an internal assistant at a 200-person company has a few hundred weekly users and about 1,300 questions a week. That is a small experiment, and most of this chapter is about being useful at that scale rather than pretending to a sample size that does not exist.

The unit of assignment

// Three options, and the choice changes what the test can measure.
type Unit = 'request' | 'session' | 'user'

Unit	Power	Problem
Request	Highest	A user sees both variants mid-session; the experience is incoherent
Session	Good	Carryover across sessions for anything with memory
User	Lowest	The only valid unit when behaviour can change

// Atlas assigns by user for anything touching memory, tone or refusal
// policy, and by session for retrieval and model changes. The rule:
// assign by user whenever the change could alter how people ASK, not
// only what they get back.
function unitFor(change: ChangeKind): Unit {
  return MEMORY_OR_BEHAVIOUR.has(change) ? 'user' : 'session'
}

Assigning by request is tempting for its power and is almost always wrong here. A user who gets a terse answer and then a verbose one to the next question is being shown an inconsistent system, and whatever they report is about the inconsistency.

Metrics that survive contact with reality

// One primary, chosen before the test runs. Everything else is a guardrail
// or a diagnostic, and none of them can rescue a failed primary.
interface Experiment {
  primary: 'task-completion' | 'follow-up-rate' | 'escalation-rate'
  guardrails: ['over-answer-rate', 'p95-latency', 'cost-per-answer']
  diagnostics: string[]
}

Metric	Why it works	Why it fails
Thumbs up rate	Direct	4% response rate, and only extremes respond
Follow-up rate	Behavioural, no asking	Confounded by harder questions
Escalation to a human	The real outcome	Slow, needs the channel instrumented
Task completion	Closest to value	Needs a definition per intent
Session length	Easy	Ambiguous — longer can be good or bad

Atlas uses escalation rate as primary: the share of assistant sessions where the user subsequently emails the operations team about the same thing. It is slow, it needs the mailbox matched against sessions, and it is the only metric on the list that measures whether the assistant did the job it was built to replace.

-- Escalation, matched on user and a 24-hour window. Crude matching, and
-- the crudeness is symmetric across variants, which is what matters.
select e.variant,
       count(distinct s.id)                                     as sessions,
       count(distinct m.session_id)                             as escalated,
       round(100.0 * count(distinct m.session_id) / count(distinct s.id), 1) as pct
  from sessions s
  join experiment_assignment e on e.user_id = s.user_id
  left join ops_mailbox m
         on m.from_user = s.user_id
        and m.received_at between s.ended_at and s.ended_at + interval '24 hours'
 where s.started_at > now() - interval '6 weeks'
 group by 1;

Sample size, stated before the test

// Run this before the experiment, not after. Finding out afterwards that
// the test could never have detected the effect is the most common waste
// in this whole area.
export function weeksRequired(baseline: number, mde: number, weeklyUnits: number): number {
  const p1 = baseline, p2 = baseline - mde
  const pBar = (p1 + p2) / 2
  // Two-proportion, 80% power, alpha 0.05.
  const nPerArm = (2 * pBar * (1 - pBar) * (1.96 + 0.84) ** 2) / (p1 - p2) ** 2
  return Math.ceil((2 * nPerArm) / weeklyUnits)
}

Atlas: baseline escalation 14%, 340 weekly sessions.

Effect to detect	Weeks needed
5 points (14% → 9%)	3
3 points	8
2 points	18
1 point	70

Eighteen weeks for two points is the number that reshapes how the team works. It means small improvements are not measurable with real users at this scale, and pretending otherwise produces a quarter of inconclusive tests.

The consequence is a division of labour that is worth stating plainly:

// What each instrument is actually for.
const DIVISION = {
  evalSuite: 'every change. Detects 2-3 point effects in six minutes.',
  abTest:    'large changes only. A new capability, a redesigned interface, a model family switch.',
  shadow:    'anything where the risk is a tail failure rather than an average one.',
}

An eval suite detecting a two-point effect in six minutes against an A/B test needing eighteen weeks is not a close call. The suite is the primary instrument and the A/B test is reserved for the question the suite genuinely cannot answer: whether people's behaviour changes.

What an underpowered test is still good for

// Three legitimate uses that do not need significance.
const USES = [
  'Guardrails. A 5x latency regression is obvious in two days.',
  'Catastrophe detection. Escalation doubling needs no interval.',
  'Qualitative. 40 sessions read by hand, by variant, blind.',
] as const

The third is underrated. Atlas ran a two-week test on a retrieval change that came back inconclusive on every quantitative metric, and reading 40 sessions blind surfaced that users in the treatment arm were pasting order numbers into follow-ups because the first answer had not resolved them. That is a finding, it is actionable, and no amount of additional sample would have produced it.

At a few hundred users, the honest output of most A/B tests is a set of transcripts. Read them.

Running one safely

// Four properties, each of which has a failure story behind it.
const PROTOCOL = {
  // 1. Assignment is deterministic and sticky, hashed on a stable id. A
  //    user who flips arms mid-test contaminates both.
  assign: (userId: string) => (hash(userId + EXPERIMENT_ID) % 100 < 50 ? 'a' : 'b'),

  // 2. The primary metric and the duration are written down before the
  //    test starts. Otherwise the test ends when the numbers look good.
  preRegistered: true,

  // 3. Guardrails are checked daily and can stop the test; the primary is
  //    looked at once, at the end. Peeking at the primary daily inflates
  //    the false-positive rate well past the stated alpha.
  peeking: 'guardrails only',

  // 4. Users can see which variant they are on and opt out. At 200 people
  //    they will find out anyway, and discovering it accidentally costs
  //    more trust than being told.
  transparent: true,
}

The fourth is specific to internal tools and is the one that surprises people from consumer backgrounds. At this scale the users are colleagues who talk to each other, and an undisclosed experiment that gets noticed becomes a story about the team rather than about the assistant.

Key points

Assign by user whenever the change could alter how people ask, by session otherwise. Request-level assignment shows one person an inconsistent system.
Choose one primary metric before the test. Atlas uses escalation to a human: slow, awkward to instrument, and the only one that measures the job.
Thumbs-up rates have a 4% response rate and only extremes respond.
Compute the required duration first. Atlas needs 3 weeks for 5 points and 18 for 2.
Small improvements are not measurable with real users at a few hundred people. The eval suite detects 2-3 points in six minutes.
Reserve A/B tests for large changes and for behaviour questions the suite cannot answer.
An underpowered test still catches catastrophes, enforces guardrails, and produces transcripts worth reading. Reading 40 blind sessions found what no metric did.
Make assignment sticky, pre-register the primary, peek only at guardrails, and tell people the test is running.

Next: making the eval suite fast enough that people run it — the property that decides whether any of this gets used.