Chapter 55 · The model layer

Few-shot examples, and where they stop helping

Examples fix format and edge-case handling, not knowledge. What three examples buy, what twenty cost, and the two ways a well-meant example makes the system worse.

10 min read

Chapter 54 used one example to pin the output format. This chapter is about when more of them help.

The short version: examples teach shape, not substance. They are excellent for format and for edge-case behaviour, useless for knowledge, and they cost tokens on every single request forever.

What examples fix

Three things, in descending order of how well they work.

Format, which they fix completely

<!-- One example removes more format variance than any amount of description,
     and it is the single highest-value example you will add. -->
Answer in this shape:

  Damaged goods may be returned within 30 days of delivery, provided the
  damage is reported within 48 hours [1]. Later returns are assessed
  individually and may attract a restocking fee [2].

Edge-case behaviour, which they fix well

A rule says what to do; an example shows what it looks like. For behaviour that is unusual, the example is more reliable than the rule — and the combination is more reliable than either.

<!-- Two examples covering the behaviours that are hardest to get from a
     rule alone: declining, and reporting a conflict. -->
When the sources do not answer the question:

  Question: What is the restocking fee for electronics?
  Sources: [1] Returns Policy > Late returns — "...a restocking fee of 15%..."
  Answer: I could not find this in the documents available to you.

When two sources disagree:

  Question: How long do customers have to return damaged goods?
  Sources: [1] Returns Policy v4 — "within 30 days"
           [2] Returns Policy v3 — "within 14 days"
  Answer: The current policy allows 30 days [1]. An earlier version of the
  same policy says 14 days [2]; confirm which applies before quoting it.

The abstention example is the one that earns its tokens. Atlas's measurement: the rule alone produced the exact phrase 71% of the time; the rule plus this example, 97%.

Tone, which they fix adequately

Worth one example at most. Tone is the least consequential thing in this system and the most tempting to iterate on.

What examples do not fix

Problem	Why examples do not help	What does
Missing knowledge	The answer is not in the model	Retrieval
Wrong retrieval	Examples do not change what was fetched	Section 4
Arithmetic errors	Shape, not capability	Compute it
Permissions	Not a prompt concern	The `where` clause

An example cannot teach the model something it does not know. It can only show it what to do with what it has.

The cost curve

Every example is tokens on every request, and it competes for attention with the rules.

// Measure, as always. Same cases, same retrieval, examples varied.
for (const n of [0, 1, 2, 3, 5, 8, 12, 20]) {
  const scored = await scoreEndToEnd(cases, { examples: EXAMPLES.slice(0, n) })
  console.log(n, round(scored.correct), round(scored.formatValid),
              round(scored.abstention), scored.promptTokens)
}

Examples	Correct	Format valid	Abstains	Prompt tokens
0	0.86	0.72	0.71	420
1	0.89	0.96	0.74	500
2	0.90	0.97	0.94	580
3	0.91	0.98	0.97	660
5	0.91	0.98	0.97	820
8	0.90	0.98	0.96	1,060
12	0.89	0.97	0.95	1,380
20	0.87	0.96	0.93	2,100

Three readings.

The first example buys the format, 0.72 to 0.96. Nothing else in this chapter has that return.

The second and third buy abstention, 0.74 to 0.97. Those are the two edge-case examples above.

Everything past three is flat and then negative. At twenty examples correctness is below where it was at one, and the prompt is five times longer.

The shape is the same as the context-window curve and for the same reason: more material competing for the same attention.

Choosing which three

Not three random good answers. Three examples covering the behaviours that rules alone do not reliably produce.

// The selection criterion, applied to candidate examples.
export function exampleValue(example: Example, cases: EvalCase[]) {
  // An example earns its place only if the behaviour it demonstrates fails
  // without it. Same test as a prompt rule in chapter 53.
  return {
    withIt: scoreSubset(cases, example.covers, { examples: [example] }),
    without: scoreSubset(cases, example.covers, { examples: [] }),
  }
}

Atlas's three, each traceable to a measured failure:

Example	Covers	Lift
A normal cited answer	Format	+0.24 format validity
An abstention	Declining when sources do not answer	+0.23 abstention
A conflict between sources	Two live policy versions	+0.19 on that subset

A fourth candidate — an example of handling a superseded amendment — tested at +0.02 and was left out. The rule alone was sufficient.

Dynamic examples, and why they usually are not worth it

Retrieving examples similar to the current question, rather than fixing three.

// Plausible, and it breaks prompt caching, which is the hidden cost.
const similar = await retrieveExamples(question, { limit: 3 })

Three problems:

It breaks the cacheable prefix from chapter 53. Variable examples at the front mean nothing is cached.
It adds a retrieval — latency and another failure mode on the request path.
It measured at +0.01 on Atlas against three fixed examples.

// If you do it: put dynamic examples AFTER the stable prefix, so the rules
// and format example stay cached.
const prompt = [
  STABLE_PREFIX,        // role, rules, format example — cached
  dynamicExamples,      // varies
  sources,
  question,
].join('\n\n')

It is worth it when the task has genuinely distinct modes — a system answering both policy questions and troubleshooting questions may want different examples for each. For one mode, three fixed examples is the answer.

Examples in the structured-output case

Where examples are most valuable, and chapter 56 returns to it: a model producing JSON against a schema benefits far more from one correct example than from a longer schema description.

// One example of the exact output, including the awkward cases: an empty
// array, a null, an escaped quote. These are where format failures live.
const EXAMPLE_OUTPUT = {
  answer: 'Damaged goods may be returned within 30 days [1].',
  citations: [1],
  confidence: 'high',
  caveats: [],              // ⚠️ show the empty case, or it emits null
}

Showing the empty array rather than omitting the field is the detail. A model that has only seen populated examples will often emit null or omit the key, and the parser then has two shapes to handle.

The failure mode: the example that taught the wrong lesson

Atlas's conflict example, written early:

  Question: How long do customers have to return damaged goods?
  Sources: [1] Returns Policy v4 — "within 30 days"
           [2] Returns Policy v3 — "within 14 days"
  Answer: The current policy allows 30 days [1].

Reasonable, and it taught something nobody intended. The example shows the model resolving a conflict by picking the newer source and not mentioning the other — so on a question where two sources disagreed and neither was clearly newer, the model picked one and presented it as settled.

Rule 4 said to report both. The example demonstrated otherwise, and the example won.

<!-- The corrected example, which matches the rule it accompanies. -->
  Answer: The current policy allows 30 days [1]. An earlier version of the
  same policy says 14 days [2]; confirm which applies before quoting it.

Two defences, and the second is the general one.

// 1. Every example must satisfy every rule. Assert it mechanically —
//    this is cheap and it catches the class.
test('examples obey the rules', () => {
  for (const example of EXAMPLES) {
    if (example.sources.length > 1 && hasConflict(example.sources)) {
      // Rule 4: report both.
      expect(countCitations(example.answer)).toBeGreaterThan(1)
    }
    // Rule 1: every example answer cites, unless it is the abstention.
    if (!isAbstention(example.answer)) {
      expect(example.answer).toMatch(/\[\d+\]/)
    }
  }
})

// 2. Test the behaviour the example might have over-generalised, on a case
//    the example does NOT cover. This is what found it.
test('reports a conflict where neither source is newer', async () => {
  const answer = await generate(Q.leadTime, [SOURCES.supplierA, SOURCES.supplierB])
  expect(countCitations(answer.text)).toBeGreaterThan(1)
})

An example is a stronger instruction than a rule. When they disagree, the example wins — so every example has to be a correct instance of every rule, and the cases an example does not cover have to be tested separately.

Key points

Examples teach shape, not substance. They cannot supply knowledge the model does not have.
The first example buys format, 0.72 to 0.96. Nothing else has that return.
The second and third buy abstention, 0.74 to 0.97. Those are the edge cases rules handle poorly.
Past three, the curve is flat and then negative. Twenty examples scored below one.
Select examples by testing whether the behaviour fails without them, exactly as with prompt rules.
Dynamic example retrieval breaks prompt caching, adds latency, and measured at +0.01. Use it only for genuinely distinct task modes.
For structured output, show the awkward cases — empty arrays, nulls — or the parser gets two shapes.
An example is a stronger instruction than a rule, so every example must satisfy every rule. Assert it.
Test the behaviour an example might over-generalise, on a case the example does not cover.

Next: structured output — making the model return something a program can rely on.