Chapter 56 · The model layer

Structured output

The moment an answer stops being prose and becomes data. Four mechanisms ranked by reliability, and the schema design choices that decide how often parsing fails.

10 min read

So far Atlas returns prose with [n] citations, parsed by a regex. That works for an answer a person reads. It stops working the moment anything downstream needs the parts separately: a UI rendering sources as links, a confidence signal, a flag that the answer was a refusal.

This chapter is about getting structure out reliably. Chapter 57 is about what to do when it fails anyway.

Four mechanisms, ranked

Mechanism	Reliability	Cost	When
Constrained decoding / tool schema	~100%	None	Always, if available
JSON mode	High	None	If constrained decoding is not
Prompted JSON + repair	Medium	A retry sometimes	Fallback
Delimited prose, parsed	High for simple shapes	None	Often the right answer

The fourth deserves its place, and it is the one people skip past.

Constrained decoding

The model is prevented from emitting tokens that would violate the schema. Not persuasion — the sampler cannot produce invalid output.

// Expressed as a tool the model must call. The schema is enforced by the
// decoder, so a malformed response is not a possible outcome.
const ANSWER_TOOL = {
  name: 'provide_answer',
  description: 'Return the answer to the user question with its citations.',
  input_schema: {
    type: 'object',
    properties: {
      answer: { type: 'string', description: 'The answer, with [n] citations inline.' },
      citations: {
        type: 'array',
        items: { type: 'integer', minimum: 1 },
        description: 'Source numbers cited, in order of first appearance.',
      },
      found: {
        type: 'boolean',
        description: 'False when the sources do not contain the answer.',
      },
      caveats: {
        type: 'array',
        items: { type: 'string' },
        description: 'Freshness or conflict warnings. Empty when none apply.',
      },
    },
    required: ['answer', 'citations', 'found', 'caveats'],
  },
} as const

⚠️ Constrained decoding guarantees the shape, not the content. citations: [7] against six sources is schema-valid and wrong, which is why chapter 57 exists.

Delimited prose

For a shape this simple, a delimiter is as reliable as JSON and has two real advantages.

Answer the question, then on a new line give the metadata.

<answer>
Damaged goods may be returned within 30 days of delivery [1].
</answer>
<found>true</found>
<caveats>none</caveats>

// Parsing is five lines and cannot fail on an escaped quote.
const section = (tag: string, text: string) =>
  new RegExp(`<${tag}>([\\s\\S]*?)</${tag}>`).exec(text)?.[1]?.trim() ?? null

Two advantages over JSON, and both matter.

It streams. The answer tag opens first, so tokens can be shown to the reader as they arrive — which JSON cannot do without a partial-JSON parser, and which chapter 59 depends on.

No escaping. A quotation mark inside the answer is the most common JSON failure for this task, because answers quote policy text constantly.

Reach for JSON when the structure is nested or variable. For a flat shape with one long prose field, delimiters are more reliable and they stream.

Atlas uses delimited prose for the answer path and a tool schema for everything in section 6.

Schema design decides the failure rate

The schema is not neutral. These choices change how often you get something unusable.

Flat beats nested

// ❌ Nesting invites a plausible-looking wrong shape.
{ result: { answer: { text: '...', meta: { citations: [...] } } } }

// ✅ Flat. Fewer places to get it wrong, and a partial parse is still useful.
{ answer: '...', citations: [1, 2], found: true, caveats: [] }

Required beats optional

// ❌ Optional fields produce three shapes for "no caveats": the key absent,
//    null, and []. Every consumer then handles three cases.
caveats?: string[]

// ✅ Required, with the empty case shown in the example. One shape.
caveats: string[]    // [] when none

This is the same point chapter 55 made about showing the empty array: the fix is half schema and half example.

Enums beat free strings

// ❌ 'high', 'High', 'very high', 'medium-high', 'I am fairly confident'
confidence: { type: 'string' }

// ✅ Three values, and a consumer can branch on them.
confidence: { type: 'string', enum: ['high', 'medium', 'low'] }

Ask for what you will use

// ❌ Fields nobody reads. Each one costs output tokens on every request and
//    is a chance to fail validation.
{ answer, citations, found, caveats, confidence, reasoning, alternatives,
  relatedQuestions, suggestedFollowUps, sentiment, topics }

// ✅ Four fields, all consumed.
{ answer, citations, found, caveats }

A field nobody reads is pure cost, and reasoning is the usual offender — it doubles output tokens to produce text that is logged and never looked at.

Structure changes what you can measure

The real payoff, and it is larger than the parsing convenience.

// Prose: inferring intent from text. Fragile and approximate.
const abstained = answer.text.startsWith('I could not find this')

// Structured: the model states it. Exact, and testable.
const abstained = !answer.found

// Which makes the metrics from chapter 10 cheap and precise.
logger.info('answer', {
  found: answer.found,
  citationCount: answer.citations.length,
  caveatCount: answer.caveats.length,
  // ⚠️ The check a regex on prose cannot do reliably: does every cited
  // number correspond to a source that was actually sent?
  citationsValid: answer.citations.every((n) => n >= 1 && n <= sources.length),
})

-- And the dashboard question becomes a query rather than a text search.
select date_trunc('day', answered_at) as day,
       count(*),
       round(avg((not found)::int) * 100) as abstention_pct,
       round(avg(citation_count), 1) as avg_citations,
       count(*) filter (where not citations_valid) as invalid_citations
  from answer_log
 where answered_at > now() - interval '30 days'
 group by 1 order by 1;

The invalid_citations column is the one to watch. It is the clearest early signal of model or prompt drift, and it is only available because the citations are a field rather than a pattern in prose.

What it costs

	Prose	Delimited	Tool schema
Output tokens	220	240	280
Parse failures	3% (citation regex)	0.2%	0%
Streams	✅	✅	⚠️ needs partial parsing
Correct	0.91	0.91	0.90

The tool-schema row is worth noting: correctness fell by a point. Constrained decoding occasionally forces a token the model would not have chosen, and on a prose-heavy field that shows up as slightly worse writing. For a four-field answer it is a fine trade; for a long-form answer it is a reason to prefer delimiters.

The failure mode: the schema that changed the answer

The subtle one. Atlas's schema gains a field:

confidence: { type: 'string', enum: ['high', 'medium', 'low'] }

Harmless-looking. Measured, correctness fell from 0.91 to 0.87 and abstention from 0.97 to 0.82.

The cause: asking for a confidence rating gave the model a way to hedge within an answer. Where it previously declined — found: false — it now answers with confidence: 'low', because the schema made that an available and apparently reasonable response.

The field did not record a behaviour. It created one.

// Three defences.

// 1. Score every schema change end to end. A schema change is a behaviour
//    change, exactly like a prompt change.
test('schema v2 does not reduce abstention', async () => {
  const v1 = await scoreEndToEnd(cases, { schema: 'v1' })
  const v2 = await scoreEndToEnd(cases, { schema: 'v2' })
  expect(v2.abstention).toBeGreaterThanOrEqual(v1.abstention - 0.02)
})

// 2. Make the mutually exclusive cases exclusive in the schema, not in
//    guidance. If found is false, there is no answer to be confident about.
//    A union type says so; two independent fields do not.

// 3. ⚠️ Do not ask the model to rate itself at all. Self-reported confidence
//    correlates weakly with correctness and strongly with fluency — so the
//    answers it marks 'high' are the well-written ones, not the right ones.

Atlas removed the field. The signal that replaced it is computed rather than reported:

// Derived from things that are actually observable.
export function answerConfidence(answer: Answer, chunks: Retrieved[]): Confidence {
  if (!answer.found) return 'none'
  const topScore = chunks[0]?.score ?? 0
  const cited = chunks.filter((_, i) => answer.citations.includes(i + 1))
  const agreement = cited.length >= 2 && !hasConflict(cited)
  if (topScore > 0.55 && agreement) return 'high'
  if (topScore > 0.40) return 'medium'
  return 'low'
}

Ask the model for facts it has, not for judgements about itself. Every field you add is an invitation to a behaviour, and a field asking "how sure are you?" invites hedging instead of declining.

Key points

Four mechanisms: constrained decoding, JSON mode, prompted JSON with repair, and delimited prose. The last is often right.
Delimiters beat JSON for a flat shape with one long prose field: they stream, and they do not break on quoted policy text.
Constrained decoding guarantees shape, never content. citations: [7] against six sources is schema-valid.
Flat beats nested, required beats optional, enums beat free strings, and every field must have a consumer.
Structure makes abstention and citation validity exact rather than inferred, which turns the quality dashboard into a SQL query.
A tool schema cost one point of correctness on prose-heavy output. Fine for four fields, a reason for delimiters on long-form.
A schema change is a behaviour change. Adding a confidence field cut abstention by fifteen points by giving the model a way to hedge instead of decline.
Ask the model for facts, not for judgements about itself. Compute confidence from retrieval scores and citation agreement.

Next: schemas, validation and malformed responses — what to do when the structure is valid and the content is not.