Chapter 145 · Interface and adoption

Feedback: capturing it and actually using it

A thumbs-down tells you nothing. Where to put the control so it carries a cause, the 4% response rate problem, implicit signals that beat explicit ones, and the loop that closes.

8 min read

Every section of this series has quietly depended on this chapter. Chapter 93's memory rules fire on corrections. Chapter 102's case set grows from complaints. Chapter 143 put a report control at the citation. None of it works if feedback is a thumbs-down button under the answer.

Why the thumbs-down fails

// Atlas, three months of a single thumbs up/down control.
const PLAIN = {
  responseRate: 0.04,
  negativeShare: 0.71,        // only the annoyed respond
  actionable: 0.09,           // could a cause be determined? almost never
  casesProduced: 3,
}

Four percent response rate, and of the responses, 9% could be turned into anything. The problem is structural: a thumbs-down is a single bit attached to a whole answer, and an answer has a dozen things that could be wrong with it.

// The information you need and the control cannot carry:
//   - which claim was wrong
//   - whether it was wrong, incomplete, or just badly worded
//   - what the right answer is
//   - whether the source was wrong or the reading of it was

Put the control where the cause is

// Three controls at three granularities, each capturing something the
// others cannot.
<Answer>
  <Claim>
    Damaged goods may be returned within 30 days [1]
    <ClaimFeedback>  {/* ⚠️ this one */}
      <Option>This isn't right</Option>
      <Option>The source doesn't say this</Option>
    </ClaimFeedback>
  </Claim>
  <SourcePanel>
    <Report>This doesn't support the claim</Report>     {/* chapter 143 */}
  </SourcePanel>
  <AnswerFeedback>
    <Option>Didn't answer my question</Option>
    <Option>Too long / too short</Option>
    <Option>Helpful</Option>
  </AnswerFeedback>
</Answer>

// Measured against the plain control.
const STRUCTURED = {
  responseRate: 0.11,         // 4% -> 11%
  actionable: 0.78,           // 9% -> 78%
  casesProduced: 94,          // 3 -> 94 over the same window
}

The response rate not quite tripling is good and not the point. Actionable feedback went from 9% to 78%, because a claim-level control arrives with the claim, the citation and the run attached — which is everything chapter 129 needs to start.

Implicit signals beat explicit ones

// Nobody fills these in, and everybody produces them. Each is already
// recorded for another reason.
const IMPLICIT = {
  rephrased: 'the same question, reworded, within 2 minutes — the answer failed',
  citationOpened: 'they checked. Chapter 62 found falling rates are healthy',
  cancelled: 'chapter 142 — not worth the wait',
  escalated: 'chapter 113 — they emailed a human about it afterwards',
  copied: 'they used it. The strongest positive signal available',
  actedOnSuggestion: 'the failure screen worked',
}

-- The rephrase detector, which is the single most useful implicit
-- signal. It needs no user action and fires on ~14% of answers.
select q2.id as failed_answer, q1.question as original, q2.question as retry
  from question_log q1
  join question_log q2
    on q2.user_id = q1.user_id
   and q2.asked_at between q1.asked_at and q1.asked_at + interval '2 minutes'
   and q2.id <> q1.id
 where similarity(q1.question, q2.question) > 0.5
   and q1.asked_at > now() - interval '7 days';

Atlas's rephrase rate is 14% against an 11% explicit feedback rate — more signal, from nobody doing anything, and it identifies the specific answer that failed.

// The copy signal is worth instrumenting deliberately. A copied answer
// was used, which is a stronger claim than "rated helpful". Atlas's
// copy rate is 23%, and copied answers correlate with low escalation
// far better than thumbs-up does (r = 0.71 against 0.34).

Closing the loop

The part that decides whether feedback keeps arriving.

// Every piece of actionable feedback gets one of four outcomes, and
// the user is told which.
type Outcome =
  | { kind: 'fixed'; what: string }            // a document, a prompt, a config
  | { kind: 'case-added'; caseId: string }     // it is now a regression test
  | { kind: 'document-owner-notified'; owner: string }
  | { kind: 'working-as-intended'; why: string }

// And the notification, which is one line and is the reason the rate
// does not decay.
<Notice>
  You flagged an answer about Northline's terms on 14 October. The 2024
  agreement was superseded and has been removed from the index.
  <Link>See the corrected answer ›</Link>
</Notice>

Atlas measured this directly. Among users who gave feedback and received no response, the rate of giving feedback again within a month was 19%. Among those who received a closing notice, 74%.

Feedback is a loop or it is a form. A form stops being filled in within two months, and the data you need most arrives from the people who have already given up.

Routing it

// Feedback arrives about four different things and only one is an
// engineering problem. Routing it by cause is what keeps the engineering
// queue meaningful.
const ROUTE: Record<FeedbackCause, Destination> = {
  'document-wrong':      'the document owner — chapter 91 refused to store this as memory',
  'document-stale':      'the document owner, with the freshness stamp attached',
  'retrieval-missed':    'engineering — becomes a chapter 49 retrieval case',
  'generation-wrong':    'engineering — becomes an eval case',
  'out-of-scope':        'product — chapter 149 decides whether to expand',
  'interface':           'product',
}

Atlas's distribution over a quarter: 41% document-wrong or stale. The single largest category of complaints about the assistant was the company's own documents being wrong, which the assistant was faithfully reporting.

That number changed how the system was described internally. An assistant that surfaces 41% document problems is partly a documentation audit, and saying so turned a source of complaints into a reason the operations team valued it.

What to do with the people who stop

// The hardest feedback to collect and the most valuable: from users who
// used it three times and never came back. Chapter 148 measures them;
// here is how to ask.
const LAPSED = {
  when: 'four weeks after last use',
  how: 'one question, from a person, not a form',
  ask: '"You tried Atlas in September and stopped. What were you trying to do?"',
}

Atlas asked 40 lapsed users and got 22 replies. The answers were not about quality: 14 of 22 said it could not do the thing they wanted, and nine of those fourteen named something it could do. They had asked once, phrased it in a way that failed, and concluded.

That is an interface finding and an expectations finding, and no amount of in-product feedback would have surfaced it, because these people never gave any.

Key points

A plain thumbs-down got a 4% response rate and 9% actionable. It is one bit attached to an answer with a dozen things that could be wrong.
Put controls at three granularities: the claim, the source, and the answer. Actionable feedback went 9% to 78%.
A claim-level control arrives with the claim, the citation and the run attached — everything an investigation needs.
Implicit signals beat explicit ones. The rephrase detector fires on 14% of answers with no user action and names the failed answer.
Instrument copying. Copy rate correlates with low escalation far better than thumbs-up does.
Close the loop. Feedback-again rates were 19% with no response and 74% with a closing notice.
Route by cause. 41% of Atlas's feedback was the company's own documents being wrong, faithfully reported.
That reframing turned a complaint source into a reason the operations team valued the system.
Ask lapsed users directly. 14 of 22 said it could not do something, and nine of those named something it could.

Next: onboarding people who have never used one — reaching those nine before they leave.