Chapter 111 · Evaluation

Measuring refusal: did it decline when it should have

The criterion two careful people agreed on only 79% of the time. Five reasons to decline, the policy that makes them separable, and the asymmetric cost that decides every threshold.

8 min read

Chapter 106 found appropriate-scope at the bottom of the rubric: human-to-human kappa 0.66, the lowest of five criteria. The diagnosis there was that the refusal policy is underspecified, and this chapter is the repair.

The reason it matters more than its score suggests: chapter 63 established that abstention is the capability that separates a system people trust from one they stop using, and chapter 109 showed it is the first thing a model upgrade breaks. It is the most fragile property in the system and the hardest to score.

Five reasons, and they are not interchangeable

"Should it have refused?" is unanswerable because it bundles five questions with different answers.

Reason	Trigger	Correct response
Not in the corpus	Retrieval found nothing relevant	Say so, suggest where it might be
Not permitted	Permission filter excluded the source	Say it exists and is not visible to them
Not authoritative	The system holds an answer, another system owns it	Point at the owner
Not authorised	An action beyond the asker's rights	Name who can authorise
Ambiguous	Several readings, materially different answers	Clarify rather than refuse

// Once separated, each is independently assertable, which is what makes
// agreement possible. Two raters disagreeing about "should it refuse" very
// often agree completely about which of these five applies.
type RefusalReason =
  | 'not-in-corpus' | 'not-permitted' | 'not-authoritative'
  | 'not-authorised' | 'ambiguous' | 'none'

Re-running the calibration with the reason enumerated rather than a yes/no: human kappa went from 0.66 to 0.88, and the judge from 0.61 to 0.81. The same structural move as chapter 110 — replacing a judgement with a classification — and it worked for the same reason.

The 12% of remaining human disagreement concentrated almost entirely on not-authoritative against not-in-corpus, which is a genuine policy question rather than a rating problem: when the corpus contains a stale copy of something another system owns, is answering from it grounded or wrong? Atlas's policy now says the owner wins, and the raters agree.

Four outcomes, two of which are failures

// The confusion matrix for refusal. The two off-diagonal cells have very
// different costs, and every threshold in this area is set by that gap.
//
//                     should answer       should refuse
//   answered        ✅ correct          ❌ over-answer
//   refused         ❌ over-refuse      ✅ correct

Cell	Atlas	Cost
correct answer	78.1%
correct refusal	15.4%
over-refuse	4.9%	Annoying, visible, self-correcting — the user rephrases
over-answer	1.6%	A confident wrong answer, invisible, acted on

Over-refusal is three times as common and a fraction as costly. A user who is wrongly declined knows immediately and rephrases; a user who is wrongly answered takes the answer to a customer.

// Which gives the single rule that decides every threshold in this area:
// when uncertain, refuse. Atlas's abstention threshold is deliberately set
// to trade roughly three over-refusals for one avoided over-answer, and
// that ratio is a product decision written down rather than an accident
// of tuning.

The eval cases this needs

// Over-answering is detected by cases that LOOK answerable and are not.
// A case that is obviously out of scope is passed by any system and
// measures nothing — chapter 103's discrimination test rejects it.
{
  id: 'near-answerable-credit-limit',
  question: 'what is Northline’s credit limit?',
  tags: ['out-of-scope', 'not-authoritative', 'adversarial'],
  // The corpus contains a 2024 credit memo quoting a limit. The finance
  // system owns the current one. Retrieval WILL return something relevant,
  // which is exactly what makes this case hard and worth having.
  expectations: [
    { kind: 'behaviour', value: { abstains: true, refusalReason: 'not-authoritative' } },
    { kind: 'must-contain', value: 'finance', why: 'it must name the owner' },
    { kind: 'must-not-contain', value: '2024', why: 'the stale memo must not be quoted as current' },
  ],
}

Atlas's refusal set is 44 cases across the five reasons, and 31 of them are of this shape: retrieval returns something, and the right move is still to decline. Cases where retrieval returns nothing are easy and three of them suffice.

Measuring refusal quality, not just the decision

A correct refusal that is useless is a near-failure.

// Three properties of a good refusal, each independently checkable.
interface RefusalQuality {
  statesReason: boolean     // which of the five, in plain language
  namesNextStep: boolean    // the owner, the person, the system
  avoidsBlame: boolean      // not "you do not have permission" to a colleague
}

Atlas's refusals, before and after making these explicit in the prompt:

	Before	After
States a reason	71%	96%
Names a next step	34%	89%
User rephrased after a refusal	52%	23%

The last row is the outcome measure. Halving the rephrase rate means the refusals started telling people what to do instead of only what the assistant would not do — and it cost one sentence in the prompt plus the measurement that showed it was missing.

Watching it in production

-- Refusal rate by reason, weekly. A rate that moves with no deploy is the
-- corpus or the permission model changing underneath, which chapter 112
-- treats as its main subject.
select date_trunc('week', answered_at) as week,
       refusal_reason,
       count(*) as n,
       round(100.0 * count(*) / sum(count(*)) over (partition by date_trunc('week', answered_at)), 1) as pct
  from answer_log
 where answered_at > now() - interval '12 weeks'
 group by 1, 2 order by 1, 2;

Two patterns worth alerting on:

// 1. A refusal rate falling with no deploy. The usual cause is a model
//    provider rolling a version forward silently, and it shows here days
//    before it shows anywhere else. Chapter 109's upgrade measured -8.2
//    points on exactly this axis.

// 2. 'not-permitted' rising for one team. Someone's access changed, or a
//    source was re-classified, and the assistant is the first place anyone
//    notices. That is a useful signal to route to an owner rather than
//    treat as a quality regression.

Refusal is the only metric where a sudden improvement is as suspicious as a sudden fall. A system that stopped declining things did not get smarter.

Key points

"Should it have refused?" bundles five questions. Enumerating the reason took human kappa from 0.66 to 0.88 and the judge from 0.61 to 0.81.
The five: not in the corpus, not permitted, not authoritative, not authorised, ambiguous.
Over-refusal was 4.9% and over-answering 1.6%. The rarer one is far more costly: it is invisible and gets acted on.
Set the threshold to trade about three over-refusals for one avoided over-answer, and write the ratio down as a product decision.
Build cases where retrieval returns something relevant and the right move is still to decline. 31 of Atlas's 44 are that shape; obviously out-of-scope cases measure nothing.
Score refusal quality separately: states a reason, names a next step, avoids blame. Naming a next step went 34% to 89% and halved the rephrase rate.
Track refusal rate by reason weekly. A fall with no deploy usually means the model moved underneath you.
A sudden improvement in refusal rate is as suspicious as a fall.

Next: tracking quality over time as the data moves under you — the regressions nobody deployed.