Chapter 63 · The model layer

Refusal and abstention

Saying 'I do not know' is the capability that separates model tiers and the one most systems get wrong. Five kinds of not-knowing, and why over-abstention is also a failure.

10 min read

Chapter 51 found abstention separating model tiers more than correctness did: 0.52 against 0.89 against 0.94. This chapter is about the 0.89 and how to move it.

The framing: a system that always answers is a system whose answers carry no information. If Atlas answers every question, the fact that it answered tells you nothing about whether it knew — and a reader cannot distinguish the 90% that are right from the 10% that are invented.

Five kinds of not-knowing

They need different responses, and conflating them produces a system that says the same unhelpful thing to five different situations.

Kind	Cause	Right response
Not in the corpus	The answer does not exist in any source	Say so; name who would know
Not retrieved	It exists; search missed it	Say so; offer to rephrase
Not permitted	It exists; the asker may not see it	Say so without revealing it exists
Not answerable by retrieval	A count or aggregate	Route to a tool or a person
Ambiguous	The question has several readings	Ask which

// Distinguishing them is mostly free, because retrieval already knows.
export function abstentionKind(
  question: string, result: RetrievalResult, scope: Scope,
): AbstentionKind {
  if (result.kind === 'question_too_short') return 'ambiguous'
  if (classifyHops(question) === 'aggregate') return 'not_answerable'

  // ⚠️ Did an UNSCOPED search find something this person cannot see? That
  // is 'not_permitted', and it is handled very differently from absence.
  if (result.results.length === 0) {
    return hasPermittedAnswerElsewhere(question, scope) ? 'not_permitted' : 'not_in_corpus'
  }
  // Results exist but all weak: retrieval probably missed the right one.
  return result.results[0].score < config.retrieval.floor + 0.05
    ? 'not_retrieved' : 'answered'
}

// And the responses, which are written once rather than generated.
export const ABSTENTIONS: Record<AbstentionKind, string> = {
  not_in_corpus:
    'I could not find this in the documents available to you.',
  not_retrieved:
    'I could not find this in the documents available to you. ' +
    'Try naming the document or product code if you know it.',
  // ⚠️ Deliberately identical to not_in_corpus. See below.
  not_permitted:
    'I could not find this in the documents available to you.',
  not_answerable:
    'This needs a count across records, which I cannot do from document ' +
    'search. The operations team can run this report.',
  ambiguous: '',   // handled by asking, not by refusing
}

The permission case, and why it says less

not_permitted returns the same sentence as not_in_corpus, and that is a security decision rather than laziness.

// ❌ Helpful, and a disclosure.
'You do not have access to the Henderson contract.'
// The asker now knows: a Henderson contract exists, and its contents are
// restricted. On a corpus of customer contracts, existence is often the
// sensitive fact — a competitor's name in the corpus reveals a relationship.

// ✅ Indistinguishable from absence.
'I could not find this in the documents available to you.'

The phrasing "available to you" is doing careful work: it is honest — the documents available to this person do not contain it — without confirming or denying that others exist. Chapter 121 returns to this as a leakage class.

⚠️ This is a case where being less helpful is correct, and it will be questioned. Record the reason next to the string, because somebody will propose improving it.

Over-abstention is also a failure

The error nobody measures. A system tuned hard against invention becomes a system that declines when it knows.

// Both errors, tracked separately. One number cannot represent a trade
// between two error types.
interface AbstentionMetrics {
  correctAbstention: number    // declined, and the answer was not available
  falseAbstention: number      // ⚠️ declined, and the answer WAS retrieved
  correctAnswer: number
  falseAnswer: number          // answered, and it was not supported
}

// False abstention is measurable exactly, because the test set says what
// was supposed to be found.
export function isFalseAbstention(
  testCase: EvalCase, answer: Answer, chunks: Retrieved[],
): boolean {
  if (answer.found) return false
  if (testCase.expect.abstains) return false
  // It declined, the case expected an answer, and the supporting chunk was
  // in context. The information was there and it did not use it.
  return chunks.some((c) => testCase.supportingIds.includes(c.id))
}

Atlas's measurements across three prompt versions:

Prompt	Correct abstention	False abstention	Correct answer	False answer
No abstention rule	0.31	0.01	0.88	0.12
Rule only	0.71	0.04	0.89	0.04
Rule + example	0.97	0.06	0.91	0.01
Rule + example + "be cautious"	0.98	0.19	0.78	0.01

The fourth row is the trap. Adding a caution instruction improved correct abstention by one point and tripled false abstention, taking correct answers from 0.91 to 0.78. The system became noticeably more reliable and substantially less useful, and only the two-error view shows it.

Tune abstention on both errors. A single "refuses appropriately" number improves monotonically as the system becomes useless.

Making abstention cheap for the reader

An abstention that ends the interaction is a worse outcome than one that moves it forward.

// Each abstention carries what to do next, derived rather than generated.
export function abstentionResponse(kind: AbstentionKind, ctx: Context): Response {
  const base = ABSTENTIONS[kind]

  switch (kind) {
    case 'not_in_corpus': {
      // The known-gap routing from chapter 18. A named person beats an apology.
      const gap = KNOWN_GAPS.find((g) => g.pattern.test(ctx.question))
      return { text: gap ? `${base} ${gap.response}` : base, kind }
    }
    case 'not_retrieved':
      // Near-misses, offered as a starting point rather than as an answer.
      return { text: base, kind, nearest: ctx.results.slice(0, 3).map(toCitation) }

    case 'not_answerable':
      return { text: base, kind, suggestedTool: findAggregateTool(ctx.question) }
  }
}

The nearest field is worth the three lines. "I could not find this — these three passages were closest" lets the reader judge whether the question was phrased badly, and it converts a dead end into a navigable one.

Asking instead of refusing

For ambiguity, a question beats a refusal — and it has to be a specific question.

// ❌ Useless. The person already said what they wanted.
'Could you be more specific?'

// ✅ Name the readings. The reader picks in one click.
export function clarify(question: string, chunks: Retrieved[]): Clarification | null {
  const topics = distinctTopics(chunks)     // from heading paths
  if (topics.length < 2 || topics.length > 4) return null
  return {
    text: `That could mean a few things. Which did you mean?`,
    options: topics.map((t) => ({ label: t.label, refine: `${question} (${t.label})` })),
  }
}

⚠️ Gate it tightly. A system that asks a clarifying question on 15% of requests is more annoying than one that guesses — Atlas fires this on 2.4%, only when retrieved chunks split cleanly across two or three distinct heading paths with similar scores.

Abstention and streaming

Chapter 59 put <found> in the header for this reason. An abstention decided after three sentences have streamed is not an abstention.

// found: false is known before any prose is shown, and the sentence is
// emitted from code per chapter 58 — not reproduced by a sampler.
if (!header.found) {
  yield { kind: 'abstention', text: ABSTENTIONS[abstentionKind(...)], nearest }
  return
}

The failure mode: the abstention that was not recorded as one

Atlas's dashboard shows abstention at 12% and correctness at 0.91. Both look healthy.

The reality: abstention is 23%. Half of the declines are not counted, because the model phrases them in ways isAbstention() does not match — "I'm not able to find that in the available documents", "That doesn't appear in the sources provided". Those are recorded as answers, and since they contain no citations they fail the uncited check and return a fallback.

So: a quarter of questions get an abstention, half of those are shown a validation-failure message instead, and the metrics say everything is fine.

// Three fixes, and the third makes the first two unnecessary.

// 1. Measure phrasing drift explicitly.
const LOOKS_LIKE_ABSTENTION =
  /\b(could not find|couldn'?t find|not able to find|doesn'?t appear|no information|not in the (sources|documents))\b/i

if (!answer.found && !isExactAbstention(answer.text)) {
  logger.warn('abstention.phrasing_drift', { text: answer.text.slice(0, 120) })
}
if (answer.found && LOOKS_LIKE_ABSTENTION.test(answer.text)) {
  // ⚠️ The dangerous direction: it declined in prose while claiming found.
  logger.error('abstention.unflagged', { text: answer.text.slice(0, 120) })
}

// 2. Make the structured field authoritative, not the text.
const abstained = !answer.found      // never a string match on the answer

// 3. Emit the sentence from code when found is false. The model decides
//    WHETHER, your code decides WHAT. Chapter 58's rule, applied.
if (!answer.found) answer.text = ABSTENTIONS[kind]

The second check is the one that found this on Atlas. A model claiming found: true while writing a refusal in prose is a consistency failure that neither field reveals alone — and it is invisible to every metric that looks at one of them.

Make the decision a field and the wording a constant. Anything that reads the wording to infer the decision will be wrong at a rate nobody is measuring.

Key points

A system that always answers carries no information in the fact that it answered.
Five kinds of not-knowing, needing five different responses. Retrieval already knows which.
The permission case returns the same sentence as absence, deliberately. Existence is often the sensitive fact.
Track false abstention separately. A "be cautious" instruction improved abstention by one point and cost thirteen points of correct answers.
A single abstention metric improves monotonically as the system becomes useless.
Carry next steps: the named owner for a known gap, the nearest passages, the tool that could answer.
Clarify with named options, never "be more specific", and gate it to a few percent of requests.
Decide abstention in the streaming header, before any prose.
Make found authoritative and emit the sentence from code. Inferring a decision from wording fails at an unmeasured rate.

Next, and the last chapter of this section: comparing two models on your own task.