Chapter 125 · Guardrails and security

Incident response when an agent does something wrong

The first hour, the kill switches that have to exist before you need them, blast-radius queries that work because of the audit index, and the two findings that only come from a real one.

9 min read

Everything so far has been prevention. This chapter assumes it failed.

Agent incidents have one property that shapes the whole response: the blast radius is unknown at the start and is almost always larger than the report. Someone reports one wrong answer. The same defect produced answers for everyone who asked a similar question since it was introduced, and nobody else noticed because the answers were fluent.

Decide the switches before you need them

// Four, in increasing severity. Each must be one action, by one person,
// with no deploy. A kill switch that requires a release is not one.
const SWITCHES = {
  disableTool:     (name: string) => flags.set(`tool.${name}.enabled`, false),
  disableWrites:   () => flags.set('tools.writes.enabled', false),
  readOnlyMode:    () => flags.set('mode', 'retrieval-only'),   // search, no generation
  fullStop:        () => flags.set('assistant.enabled', false),
}

// Read-only mode is the one worth building and the one teams skip. It
// keeps the system useful — people can still find documents — while
// removing generation, which is where most incidents live. The gap
// between "degraded" and "gone" is the difference between an incident
// and an outage.

Atlas's switches are flags with a 10-second propagation, and they are exercised quarterly. A switch that has never been pulled is a switch of unknown function, and the quarterly exercise has twice found one that no longer worked — once because a flag name changed in a refactor and once because a cached config meant propagation was 15 minutes rather than 10 seconds.

The first hour

const FIRST_HOUR = [
  '0-5   Contain. Pull the narrowest switch that covers the reported behaviour.',
  '5-15  Scope. Run the blast-radius queries. Do not theorise first.',
  '15-30 Preserve. Snapshot the audit rows, the config, the corpus id.',
  '30-45 Notify. Affected users, the owner, and whoever the contract requires.',
  '45-60 Decide. Restore, stay degraded, or keep it down.',
]

Containing before understanding is the right order and the hardest to hold, because the reported case is usually interesting and the instinct is to debug it. The reason to contain first is that the defect is still producing output while you read.

Blast radius

This is where chapter 124's design pays for itself, and the queries are worth having written down in advance rather than composed at 2am.

-- 1. Everyone who got an answer from the same defective path. Keyed on
--    the config that was wrong, not on the question.
select distinct a.user_id, count(*) as answers, min(a.at), max(a.at)
  from audit_run a
 where a.config->>'promptHash' = $1
   and a.at between $2 and $3
 group by 1 order by 2 desc;

-- 2. Who reached a document they should not have. This is the query that
--    is impossible without the per-document access index.
select aa.user_id, aa.document_id, aa.at, aa.used_in_answer
  from audit_access aa
  join documents d on d.id = aa.document_id
 where aa.at > $1
   and not (d.acl_group = any(current_groups(aa.user_id)))
 order by aa.at;

-- 3. What actually changed state. The irreversible subset, which decides
--    whether this is a correction or a remediation.
select t.run_id, t.name, t.args_hash, a.actor->>'userId' as user_id, a.at
  from audit_tool t join audit_run a using (run_id)
 where t.decision = 'allow' and t.name = any($1::text[])   -- the write tools
   and a.at > $2
 order by a.at;

Atlas's one real incident: a retrieval configuration change widened an ACL join, reported as "it told me something I don't think I should see". The report was one answer. Query two found 31 answers to 9 users over five days, all from one re-classified document set. The reporter was the only person who noticed.

One reported wrong answer is a sample. Assume the population is everyone who asked a similar question since the change, and let a query tell you otherwise.

Preserving evidence

// Before anything is fixed. A rollback destroys the thing an
// investigation needs, and the urge to fix first is strong.
const SNAPSHOT = [
  'The audit rows for the window, exported and hashed',
  'The exact config: prompt hash, resolved model id, corpus snapshot id',
  'The retrieval index state, or its snapshot id',
  'The flag state at the time',
  'The deploy history for the window',
]

// And the reproduction, built from the snapshot rather than from memory.
// It is also the eval case, which is how chapter 102's set grows.
const repro = await run(reportedQuestion, {
  config: snapshot.config,
  corpus: snapshot.corpusSnapshotId,
  as: snapshot.affectedUser,
})

Notification

// Who, and what they need. The second column is the part that gets
// written badly under pressure.
const NOTIFY = {
  affectedUsers: 'what was wrong, which of their answers, what to do now',
  dataSubjects:  'only if personal data crossed a boundary — chapter 17 has the obligation',
  theOwner:      'the business owner of the data involved, not only the engineering manager',
  leadership:    'if it touches a customer commitment or a contract',
}

The first row has a specific requirement that distinguishes this from a normal outage notice: name the answers. A user who is told "some answers between Tuesday and Friday may have been wrong" cannot act. A user who is given the four specific questions they asked can check what they did with each one.

Atlas's notice listed each affected answer with its date and question, which was possible only because the audit log stores the question in full — the chapter 124 decision that looked like a privacy risk and turned out to be the thing that made the notification useful.

The review

// The questions that produce changes, as opposed to the ones that
// produce a document.
const REVIEW = [
  'What control should have caught this, and why did it not exist?',
  'Which detection told us, and how long did it take?',
  'What would have made the blast radius smaller?',
  'What did we need and not have in the audit log?',
  'Which eval case now exists that did not before?',
]

The second question is the one that changed the most at Atlas. The incident was found by a user after five days. No alert fired, because nothing watched the rate of answers citing documents outside the caller's normal ACL groups — an obvious signal in hindsight and one nobody had thought to want.

-- The alert that exists now, running hourly. It would have fired in
-- about forty minutes rather than five days.
select count(*) as out_of_profile
  from audit_access aa
 where aa.at > now() - interval '1 hour'
   and aa.acl_group <> all (select unnest(usual_groups) from user_profile up where up.user_id = aa.user_id)
having count(*) > 5;

Two things only a real incident teaches

// 1. The switches were not as fast as believed. Documented at 10 seconds,
//    measured at 10 seconds, and the actual time from decision to effect
//    was 6 minutes — because the decision needed someone who knew which
//    switch, and that person had to be found. The runbook now names the
//    switch for each symptom.

// 2. "Unwind it" is usually not available. Chapter 71 made actions
//    reversible and chapter 73 made them idempotent, and neither helps
//    with an answer someone already acted on. The remediation for a
//    disclosed document is a conversation, not a rollback.

The second is worth sitting with. Most of this section prevents disclosure because disclosure cannot be undone — and that asymmetry is the reason the controls are placed before the model rather than after it.

Key points

The blast radius is unknown at the start and larger than the report. One reported answer meant 31 answers to 9 users over five days.
Build four switches before you need them: disable a tool, disable writes, read-only mode, full stop. Each one action, one person, no deploy.
Read-only mode is the one teams skip and the one that keeps an incident from becoming an outage.
Exercise the switches quarterly. Atlas's exercise twice found one that no longer worked.
Contain, then scope, then preserve, then notify, then decide. Containing before understanding is the hardest order to hold and the right one.
Write the blast-radius queries in advance. The per-document access index is what makes the important one possible.
Snapshot the config, corpus id, index state, flags and deploy history before fixing anything.
Name the specific answers in the user notification. Storing questions in full is what makes that possible.
Ask which detection found it and how long it took. Atlas's was a user, after five days; the alert that now exists would have fired in forty minutes.
Switch latency in practice includes finding the person who knows which switch. Put the symptom-to-switch mapping in the runbook.
Disclosure cannot be unwound. That asymmetry is why the controls sit before the model.

Next: the security review before you let anyone else use it — the checklist that closes the section.