Chapter 124 · Guardrails and security

Audit logs: what to record for every run

The record that answers 'what did this user reach' three months later. Five questions it must answer, what to store and what to reference, retention, and the log that is itself a leak.

8 min read

Every control in this section assumed a record existed. Chapter 89 built a trace for debugging; this is the different artifact that sits beside it — written for someone who was not there, reading months later, often under pressure.

The two differ in almost every property, which is why one cannot do both jobs.

	Trace	Audit log
Reader	The engineer, today	An investigator, in six months
Retention	Days	Years
Mutable	Rewritten freely	Append-only
Content	Everything	What a question needs, and no more
Sampled	Often	Never
Access	The team	Restricted, and itself audited

Five questions it has to answer

// Written as the design requirement, because "log everything" produces an
// artifact that answers none of them.
const QUESTIONS = [
  'What did this user reach, across all their sessions?',
  'Who reached this document, and when?',
  'What actions were taken, by whom, with what authority?',
  'Why did the system answer the way it did on this run?',
  'What changed in the system between these two dates?',
] as const

Question two is the one most logs cannot answer, because they are keyed on runs rather than on data. An investigator asking "has anyone seen the Northline contract" needs an index by document, and adding it afterwards is impossible — the data was never recorded.

The record

// One row per run, plus child rows for retrieval and actions. Content is
// referenced rather than copied wherever a reference is stable.
interface AuditRun {
  runId: string
  at: Date
  actor: { userId: string; groups: string[]; sessionId: string; via: 'ui' | 'api' | 'scheduled' }
  question: string                // stored: it is the user's own input
  questionHash: string            // for grouping without reading

  retrieved: { chunkId: string; documentId: string; score: number; aclGroup: string }[]
  recalledMemories: string[]      // ids only — chapter 94's content stays in its own store
  tools: { name: string; argsHash: string; decision: 'allow' | 'deny'; reason?: string; resultRef?: string }[]

  answerRef: string               // pointer to the answer store
  answerHash: string
  citations: number[]
  refusal?: { reason: RefusalReason }

  config: { promptHash: string; modelId: string; corpusSnapshotId: string }
  flags: string[]                 // 'injection.suspected', 'unsourced_specific', ...
  cost: { tokens: number; usd: number; ms: number }
}

// Two deliberate choices.
//
// 1. Retrieved chunk ids with their ACL group, not their content. The
//    content is in the corpus and the corpus is versioned; copying it
//    into the audit log duplicates restricted data into a store with
//    different access rules. The ACL group is copied because it is the
//    fact being audited and it changes over time.
//
// 2. The question is stored in full. It is the user's own text, it is the
//    single most useful field in any investigation, and hashing it makes
//    the log useless for the question it most often has to answer.

The document index

-- The table that makes question two answerable. One row per chunk per
-- run, which is large and worth it.
create table audit_access (
  run_id      uuid not null,
  at          timestamptz not null,
  user_id     text not null,
  document_id text not null,
  chunk_id    text not null,
  acl_group   text not null,
  used_in_answer boolean not null    -- retrieved vs actually cited
);

create index on audit_access (document_id, at desc);
create index on audit_access (user_id, at desc);

used_in_answer separates two very different findings. Retrieval touched a document; a citation means the content reached the user. An investigation into a leak cares about the second and an investigation into a misconfigured ACL cares about the first.

Atlas's audit_access grows by about 15 million rows a year at 1,300 questions a week with 12 chunks each. Partitioned monthly, it is a few gigabytes — unremarkable, and the reason to decide on it early is that it cannot be backfilled.

Append-only, provably

-- Revoking update and delete is the baseline. It stops an accident and a
-- careless script, not an administrator.
revoke update, delete on audit_run, audit_access from app_role;

-- A hash chain makes tampering detectable rather than impossible, which
-- is the achievable property without separate infrastructure.
alter table audit_run add column prev_hash text, add column row_hash text;

// Each row commits to its predecessor. Verifying the chain is a scan, run
// nightly, and a break is an incident regardless of its cause.
const rowHash = sha256(`${prevHash}|${runId}|${at.toISOString()}|${actor.userId}|${answerHash}`)

For a higher bar the chain head gets written to a store the application cannot reach. Atlas ships the daily head to the same place its backups go, which is enough to detect a rewrite without building anything.

The log is itself restricted

// An audit log holds every question every person asked, which is a
// detailed record of what everyone is working on. It is one of the more
// sensitive stores in the company.
const AUDIT_ACCESS_POLICY = {
  read: ['security', 'compliance'],
  selfRead: 'any user may read their own rows',
  queriesAudited: true,            // reading the audit log is an audited event
  exportRequiresApproval: true,
}

The second line is worth having for a reason beyond fairness: a user who can see their own record can check it, and Atlas's only discovered gap in the log — a tool-call row missing its deny reason — was reported by a user looking at their own history.

The third is the one that turns an audit log into something a security team trusts. A log nobody can read unobserved is a log an insider can mine.

Retention

// Three tiers, because the questions have different time horizons.
const RETENTION = {
  fullRuns:     '90 days',    // debugging-adjacent; the heaviest rows
  accessIndex:  '7 years',    // "who saw this document" — the long question
  actions:      '7 years',    // anything that changed state
  answers:      '90 days',    // referenced by answerRef; expires with the run
}

// Which means a run older than 90 days can still answer "who reached this
// document and did an action follow", and cannot reproduce the answer
// text. That is the right trade: the access and action records are what
// an investigation needs, and the answer text is the bulk and the risk.

Chapter 17 set the contractual constraints; the point here is that retention differs per table and a single policy over "logs" gets both ends wrong.

What not to log

const NEVER = [
  'Retrieved chunk content — it lives in the corpus, referenced by id',
  'Memory statements — chapter 94, referenced by id',
  'Tool result bodies — hash and reference; they contain customer data',
  'Full prompts — the hash identifies the version; the text is in the repo',
  'Anything the chapter 16 exclusion rules kept out of the corpus',
]

The last line closes a loop that is easy to leave open. Data deliberately excluded from the corpus can still arrive in a question a user types, and a log that stores questions in full will capture it. Atlas runs the same exclusion patterns over the question field at write time and replaces a match with a marker plus the pattern name — which keeps the log useful for investigation and stops it becoming the one place a national identifier is retained for seven years.

Key points

An audit log and a debug trace are different artifacts: different reader, retention, mutability, sampling and access.
Design it against five questions. "Who reached this document" is the one most logs cannot answer, because they are keyed on runs.
Reference content by id rather than copying it. Copy the ACL group, because that is the fact being audited and it changes.
Store the question in full. It is the user's own text and the most useful field in any investigation.
Build the per-document access index from day one. It cannot be backfilled, and 15 million rows a year is a few partitioned gigabytes.
Distinguish retrieved from cited. A leak investigation cares about one; an ACL investigation cares about the other.
Revoke update and delete, hash-chain the rows, and ship the daily head somewhere the application cannot reach.
The audit log is itself sensitive. Restrict reads, audit the reads, and let users see their own rows — Atlas's only discovered gap was found that way.
Retain per table: 90 days for full runs and answers, seven years for access and actions.
Never log chunk content, memory statements, tool result bodies or full prompts — and run the corpus exclusion patterns over the question field.

Next: incident response when an agent does something wrong — using all of this under pressure.