Chapter 104 · Evaluation

Exact-match, fuzzy and semantic scoring

Three mechanical scorers, what each gets wrong, and the measured case for spending nothing on 70% of assertions before reaching for a model.

8 min read

Chapter 103's expectations have to become numbers. The temptation is to hand every one of them to a model, which is slow, costs money, and introduces a second system whose correctness is now a question. This chapter is the case for not doing that wherever possible.

Atlas's 240 cases carry 816 expectations. Seventy-one percent are scored with no model call at all.

Exact match, and where it actually belongs

// The scorer everyone dismisses as too brittle. It is correct for a narrow
// and important class: values with one canonical form.
function exactMatch(response: string, expected: string): boolean {
  return normalise(response).includes(normalise(expected))
}

function normalise(s: string): string {
  return s.toLowerCase().replace(/[‘’]/g, "'").replace(/\s+/g, ' ').trim()
}

Expectation	Exact match is	Why
An identifier (`PO-4471`)	Right	One form
A document id in a citation	Right	One form
A tool name	Right	One form
A number (`45`)	Mostly right	Beware `145`
A policy statement	Wrong	Many phrasings

The 145 problem is real and cost Atlas two weeks of a falsely-green case.

// Numbers need a boundary, which plain inclusion does not give you.
function containsNumber(response: string, n: string): boolean {
  return new RegExp(`(?<![\\d.])${escapeRe(n)}(?![\\d.])`, 'u').test(response)
}

With plain includes, a case asserting '45' passed on an answer saying "145 days". The lookbehind and lookahead fix it, and the lesson generalises: mechanical scorers fail silently in the passing direction, which is the worst direction available.

Fuzzy matching, and why Atlas mostly removed it

// Edit distance or token overlap, to tolerate phrasing differences.
function fuzzyContains(response: string, expected: string, threshold = 0.8): boolean {
  return bestWindowSimilarity(normalise(response), normalise(expected)) >= threshold
}

It sits between the two useful behaviours and does neither well. Measured against 180 hand-labelled response-expectation pairs:

Scorer	Precision	Recall	Cost
Exact	1.00	0.61	Free
Fuzzy (0.8)	0.79	0.84	Free
Semantic (0.75)	0.91	0.93	~$0.00002
A judge	0.94	0.95	~$0.002

Fuzzy's 0.79 precision is the problem. A scorer that is wrong one time in five, in both directions, produces an aggregate nobody can act on — and its failures are arbitrary rather than systematic, so they cannot be reasoned around.

Atlas kept fuzzy for exactly one job: tolerating OCR noise in citations drawn from scanned PDFs, where the expected string genuinely varies by a character or two and the variation is mechanical rather than semantic.

Semantic scoring

// Embed the expectation and the candidate sentences, take the best match.
// This is the chapter 38 embedding model doing a second job, on text short
// enough that it is close to free.
export async function semanticContains(
  response: string,
  expected: string,
  threshold = 0.75,
): Promise<{ passed: boolean; score: number; sentence: string }> {
  const sentences = splitSentences(response)
  const [e, ...cands] = await embedBatch([expected, ...sentences])
  let best = { score: -1, sentence: '' }
  cands.forEach((c, i) => {
    const s = cosine(e, c)
    if (s > best.score) best = { score: s, sentence: sentences[i] }
  })
  return { passed: best.score >= threshold, score: best.score, sentence: best.sentence }
}

Two failure modes worth knowing before trusting it.

// 1. Negation. Embeddings are notoriously weak at it: "Northline is on net
//    45" and "Northline is not on net 45" score 0.94 against each other.
//    Any expectation whose meaning turns on a negation must not use this.

// 2. Topical drift. A sentence about payment terms in general scores 0.78
//    against a specific term, which clears the threshold without stating
//    the fact.

The first is severe enough to be a hard rule rather than a caveat.

// Enforced at case-definition time, so it fails when the case is written
// rather than producing a wrong score forever.
if (/\b(not|never|no|without|except|unless)\b/i.test(expectation.value) &&
    expectation.scorer === 'semantic') {
  throw new Error(`Case ${id}: semantic scoring cannot express a negation. Use exact or a judge.`)
}

Choosing a scorer per expectation

// The scorer belongs on the expectation, not on the case and not globally.
// A single case routinely needs all three.
type Scorer = 'exact' | 'number' | 'semantic' | 'fuzzy-ocr' | 'judge'

const DEFAULT_SCORER: Record<Expectation['kind'], Scorer> = {
  'must-contain': 'semantic',      // overridden to 'exact' for identifiers
  'must-not-contain': 'exact',     // a trap string is a literal
  'must-cite': 'exact',            // a document id
  behaviour: 'exact',              // structured fields, not prose
}

must-not-contain defaulting to exact is deliberate and the opposite of the obvious choice. The trap is a specific string — the superseded 30 from the 2024 agreement — and a semantic scorer fires on any sentence about payment terms, turning a precise assertion into a near-guaranteed failure. A trap is literal by construction.

What the 71% buys

	Model-scored everything	Mixed
Full-run wall clock	19 min	6 min
Full-run cost	$8.40	$2.10
Scorer disagreement with a reviewer	6%	4%

Faster, four times cheaper, and more accurate — because the mechanically-scored assertions are the ones with exact answers, and a judge asked to confirm that PO-4471 appears in a response will occasionally say no.

Chapter 114 argues that run time is what decides whether a suite gets used. The 71% is most of that chapter's answer, and it costs nothing but choosing the scorer per assertion.

Reach for a judge when the property genuinely requires judgement. An identifier does not, a citation does not, and a tool call certainly does not.

Reporting a score honestly

// Per-expectation results, not a single pass/fail per case. A case with
// four expectations that fails one has information in it that a boolean
// throws away.
interface CaseResult {
  caseId: string
  expectations: { kind: string; passed: boolean; score?: number; detail: string }[]
  passed: boolean           // all expectations passed
  partial: number           // share that passed
}

Aggregating on partial rather than passed made Atlas's suite considerably more sensitive. A prompt change that stopped citations appearing dropped partial by six points while passed dropped by thirty-one — the first is a readable signal about what broke, the second says only that something did.

Key points

71% of Atlas's 816 expectations are scored with no model call. That is most of why a full run is six minutes and two dollars.
Exact match is correct for identifiers, document ids and tool names. For numbers it needs a digit boundary: '45' passed on "145 days" for two weeks.
Mechanical scorers fail silently in the passing direction, which is the worst one.
Fuzzy matching scored 0.79 precision and was removed except for OCR noise in citations. Arbitrary errors cannot be reasoned around.
Semantic scoring hits 0.91 precision for near-free and fails on negation: "is" and "is not" score 0.94 against each other. Forbid it at case-definition time.
must-not-contain defaults to exact. A trap string is literal, and a semantic scorer fires on the topic rather than the trap.
The mixed approach was four times cheaper, three times faster and more accurate than judging everything.
Aggregate on per-expectation partial credit. It moved six points where pass/fail moved thirty-one, and the six were readable.

Next: LLM-as-judge, and its specific traps — the other 29%.