Chapter 57 · The model layer

Schemas, validation and the malformed response

Shape-valid and wrong is the common case. Four validation layers, what each catches, and the repair strategy that does not make things worse.

12 min read

The previous chapter ended on the gap: constrained decoding guarantees shape, never content. citations: [7] against six sources parses perfectly and is wrong.

This chapter is the validation layer, and the organising idea is that validation is not one check. It is four, each catching a different class, and only the first is about parsing.

Four layers

The split between the two failure paths is the point of the chapter. Layers 1 and 2 are format problems and can be repaired. Layers 3 and 4 are content problems, and repairing them means inventing — which is the thing the whole system exists to avoid.

Layer 1 and 2: parsing and schema

// src/generate/validate.ts
import { z } from 'zod'

export const AnswerSchema = z.object({
  answer: z.string().min(1).max(4000),
  citations: z.array(z.number().int().positive()).max(20),
  found: z.boolean(),
  caveats: z.array(z.string().max(300)).max(5),
})

export type Answer = z.infer<typeof AnswerSchema>

export function parseAnswer(raw: string): Result<Answer> {
  // Tolerate the two things models do to JSON, and nothing else. A lenient
  // parser that accepts anything hides a prompt problem.
  const cleaned = raw
    .replace(/^```(?:json)?\s*/i, '')     // fenced output
    .replace(/\s*```$/, '')
    .trim()

  let parsed: unknown
  try {
    parsed = JSON.parse(cleaned)
  } catch {
    return { ok: false, layer: 'syntactic', raw }
  }

  const result = AnswerSchema.safeParse(parsed)
  return result.success
    ? { ok: true, value: result.data }
    : { ok: false, layer: 'schema', issues: result.error.issues, raw }
}

Tolerating exactly two deviations — a code fence and surrounding whitespace — is deliberate. A parser that strips trailing commas, fixes quotes and extracts the first JSON-looking substring will succeed on output that indicates the prompt is wrong, and you will never find out.

Layer 3: semantic validation

The layer that matters, and the one usually missing.

export function validateSemantics(answer: Answer, sources: Retrieved[]): Issue[] {
  const issues: Issue[] = []

  // ⚠️ The most important check in this file. A citation to a source that
  // was not sent is a fabricated reference, and it is invisible to a reader.
  for (const n of answer.citations) {
    if (n > sources.length) {
      issues.push({ kind: 'citation_out_of_range', value: n, max: sources.length })
    }
  }

  // Inline [n] markers must match the citations array. A mismatch means one
  // of the two is wrong and a UI rendering links will show the wrong source.
  const inline = new Set([...answer.answer.matchAll(/\[(\d+)\]/g)].map((m) => Number(m[1])))
  const declared = new Set(answer.citations)
  for (const n of inline) {
    if (!declared.has(n)) issues.push({ kind: 'inline_not_declared', value: n })
  }
  for (const n of declared) {
    if (!inline.has(n)) issues.push({ kind: 'declared_not_inline', value: n })
  }

  return issues
}

// Quoted figures must appear in a cited source. Narrow and high-value:
// numbers are what people act on, and a transposed figure is undetectable
// by reading.
export function validateFigures(answer: Answer, sources: Retrieved[]): Issue[] {
  const citedText = answer.citations
    .map((n) => sources[n - 1]?.content ?? '')
    .join(' ')
    .replace(/\s+/g, ' ')

  const figures = answer.answer.match(/\b\d+(?:[.,]\d+)?\s*(?:%|days?|hours?|months?)\b/gi) ?? []

  return figures
    .filter((f) => !citedText.toLowerCase().includes(normaliseFigure(f)))
    .map((f) => ({ kind: 'figure_not_in_sources', value: f }))
}

Atlas's measurement on this check: it fired on 1.4% of answers. Of those, two thirds were the model correctly rephrasing ("a month" from "30 days") and one third were genuine errors — a rate of roughly one fabricated figure in 200 answers, which is invisible without the check and material at 800 questions a day.

// So it is a warning that triggers a review queue, not a hard rejection.
// ⚠️ A check with a 67% false-positive rate must not block an answer.
if (figureIssues.length) {
  await queueForReview(questionId, figureIssues)
  answer.caveats.push('Figures in this answer could not be matched to a source automatically.')
}

Layer 4: consistency

Fields that must agree with each other.

export function validateConsistency(answer: Answer): Issue[] {
  const issues: Issue[] = []

  // found: false means no citations and the exact phrase.
  if (!answer.found) {
    if (answer.citations.length > 0) issues.push({ kind: 'abstained_but_cited' })
    if (!isExactAbstention(answer.answer)) issues.push({ kind: 'abstained_wrong_phrase' })
  }

  // found: true means at least one citation. An uncited claim is the
  // failure mode the whole prompt exists to prevent.
  if (answer.found && answer.citations.length === 0) {
    issues.push({ kind: 'answered_without_citation' })
  }
  return issues
}

answered_without_citation is the one to treat most seriously. It is an ungrounded answer that passed every other layer, and it is exactly what chapter 51 measured as the capability separating model tiers.

Repair: when, and how not to make it worse

// Format failures only, one retry, and the retry must be cheap and specific.
export async function generateValidated(
  question: string, sources: Retrieved[], attempt = 0,
): Promise<Answer> {
  const raw = await callModel(buildPrompt(question, sources))
  const parsed = parseAnswer(raw)

  if (!parsed.ok) {
    if (attempt >= 1) {
      // ⚠️ One retry. A loop here is a cost incident waiting for a bad
      // prompt deploy, and the second failure is almost never transient.
      logger.error('answer.unparseable', { layer: parsed.layer, raw: raw.slice(0, 400) })
      throw new UnparseableResponse(parsed.layer)
    }
    // Repair by telling it what was wrong, not by asking again.
    return generateValidated(question, sources, attempt + 1)
  }

  const issues = [
    ...validateSemantics(parsed.value, sources),
    ...validateConsistency(parsed.value),
  ]

  // ❌ Never retry a semantic failure with "that citation was wrong, try
  //    again". The model will produce a different citation, and a citation
  //    chosen to satisfy a validator is worse than an absent one.
  if (issues.some(isFatal)) {
    logger.error('answer.invalid', { issues, questionId })
    return FALLBACK_ANSWER      // below
  }

  return { ...parsed.value, warnings: issues.filter((i) => !isFatal(i)) }
}

The comment on the semantic path is the substantive advice in this chapter. A retry prompt saying "citation [7] does not exist, please correct it" reliably produces a citation that does exist and is not the source of the claim. The validator passes, the answer is now worse, and nothing records it.

// The fallback: honest, and it is a better outcome than a repaired answer.
const FALLBACK_ANSWER: Answer = {
  answer: 'I found relevant documents but could not produce a reliable answer. ' +
          'The sources are listed below — please check them directly.',
  citations: [],
  found: false,
  caveats: ['An internal validation check failed on this answer.'],
}

Returning the retrieved sources alongside the fallback is what makes it useful rather than merely safe: retrieval worked, so the person gets the passages and does the last step themselves.

What the numbers look like

Atlas, 30 days, 24,000 answers:

Layer	Fired	Action	Notes
Syntactic	0.0%	—	Constrained decoding, as expected
Schema	0.0%	—	—
Citation out of range	0.08%	Fallback	19 answers
Inline/declared mismatch	0.31%	Warning	Usually a trailing citation
Figure not in sources	1.4%	Review queue	~1/3 genuine
Answered without citation	0.12%	Fallback	29 answers

Forty-eight fallbacks in 24,000 answers. Each one is a person who got sources instead of an answer, which is a small cost; each one is also an answer that would otherwise have been confidently wrong with a fabricated reference.

Validation is also your drift detector

-- The rate is a leading indicator. It moves before correctness does,
-- because a model or prompt change degrades discipline before it degrades
-- comprehension.
select date_trunc('day', answered_at) as day, prompt_version, model,
       round(100.0 * count(*) filter (where validation_failed) / count(*), 2) as fail_pct
  from answer_log
 where answered_at > now() - interval '30 days'
 group by 1, 2, 3 order by 1;

// Alert on the rate, not on individual failures. Individual failures are
// expected; a doubling is a deploy.
if (todayFailRate > baselineFailRate * 2 && todayCount > 200) {
  alert('validation failure rate doubled', { today: todayFailRate, baseline: baselineFailRate })
}

The failure mode: the validator that was quietly disabled

The validator is strict at launch. It fires on 3% of answers, mostly the figure check's false positives, and each one is a person getting a fallback instead of a reasonable answer.

Someone widens the figure tolerance. Then the inline-mismatch check becomes a warning instead of fatal. Then — during an incident where the fallback rate spiked — the whole semantic layer is put behind a flag, and the flag is left off.

Six months later validation_failed is 0.0% and nobody can remember whether that is because the system is good or because nothing is checked.

// Three defences.

// 1. A validator that never fires is broken. Assert on the floor as well
//    as the ceiling.
test('validation is live', async () => {
  const result = await generateValidated(Q.any, [FABRICATED_CITATION_SOURCE])
  expect(result.caveats.length).toBeGreaterThan(0)
})

// 2. Make the flag visible in the health report rather than in config.
logger.info('validation.config', {
  semanticEnabled: config.validation.semantic,
  figureCheckEnabled: config.validation.figures,
  consistencyEnabled: config.validation.consistency,
})

// 3. ⚠️ Separate severity from the check. Downgrading a check to a warning
//    is a reasonable response to false positives; DISABLING it is not, and
//    the two should not be the same switch.
export const VALIDATION = {
  citationRange:  { enabled: true, severity: 'fatal' },
  inlineMismatch: { enabled: true, severity: 'warn' },
  figures:        { enabled: true, severity: 'review' },
  uncited:        { enabled: true, severity: 'fatal' },
} as const

The third is the structural fix. With one switch, the only available response to noise is to turn the check off; with two, the response is to lower its severity and keep the signal — and the signal is what tells you the model changed.

A check that is too noisy should become quieter, never absent. The difference between a 0% failure rate and no validator is invisible in every dashboard that matters.

Key points

Four layers: syntactic, schema, semantic, consistency. Only the first two are format problems.
Tolerate exactly two JSON deviations. A lenient parser hides a prompt problem permanently.
Citation range is the single most important check — a fabricated reference is invisible to a reader.
Check that inline markers and the citations array agree, and that quoted figures appear in a cited source.
The figure check has a high false-positive rate, so it queues a review rather than blocking.
Repair format failures once. Never repair a semantic failure: a citation chosen to satisfy a validator is worse than none.
Fall back to an honest message plus the retrieved sources. Retrieval worked; let the person finish.
The validation failure rate is a leading drift indicator. Alert on a doubling, not on individuals.
Separate "enabled" from "severity", or the only response to a noisy check is to delete the signal.

Next: temperature, sampling and determinism — what you can and cannot reproduce.