Chapter 148 · Interface and adoption

Measuring adoption against usage

Question volume goes up and nothing improved. The cohort view that cannot be gamed, the four metrics worth reporting, and the honest way to count a system that replaces email.

7 min read

Six months in, somebody asks whether it is working. The available number is usually questions per week, it is going up, and it answers nothing.

// Atlas's month six, as first reported.
const VANITY = {
  questionsPerWeek: 1_300,     // up from 400
  usersThisMonth: 142,         // of 200
  answersGenerated: 5_600,
}
// Every number rising. Three of them are compatible with the system
// being useless.

Rising volume is compatible with people asking the same question repeatedly because the answer is bad, with a rollout to more people who will each try it twice, and with the system being genuinely useful. The three are indistinguishable from the top line.

The cohort view

-- Retention by signup cohort. It cannot be inflated by a rollout, and
-- it answers the only question that matters: do people keep coming back?
select cohort_month,
       count(*) as users,
       round(100.0 * count(*) filter (where weeks_active >= 2) / count(*)) as w2,
       round(100.0 * count(*) filter (where weeks_active >= 4) / count(*)) as w4,
       round(100.0 * count(*) filter (where weeks_active >= 12) / count(*)) as w12
  from user_cohorts group by 1 order by 1;

Cohort	Users	Week 2	Week 4	Week 12
Pilot (wave 1)	6	100%	100%	83%
Wave 2	24	79%	71%	67%
Wave 3 (ops)	18	89%	83%	78%
Wave 4 (everyone)	152	68%	54%	41%

Forty-one percent of the company still using it at twelve weeks. That is the honest adoption number, and it is both much lower than the 142-monthly-users figure and much more useful.

// The fall from 54% to 41% between weeks 4 and 12 is the number to
// work on, and chapter 145's lapsed-user interviews are how you find
// out why. Atlas's answer: 14 of 22 said it could not do the thing
// they wanted, nine of those naming something it could.

Four metrics worth reporting

// One of each kind: breadth, depth, substitution, and quality-in-use.
const REPORT = {
  breadth:      'share of the target population active in the last 4 weeks',
  depth:        'median questions per active user per week',
  substitution: 'escalation rate — did it replace the thing it was meant to',
  qualityInUse: 'share of answers where a citation was opened',
}

// Atlas, month twelve.
const MONTH_12 = {
  breadth: 0.47,           // 94 of 200
  depth: 3.2,              // per active user per week
  escalationRate: 0.11,    // from 0.31 at launch
  citationOpenRate: 0.19,  // from 0.48 at launch — chapter 62's healthy shape
}

The escalation rate is the one that answers the business question. It fell from 31% to 11%, which means two thirds of the questions that used to reach the operations team by email now do not — and that is measured on the operations side rather than claimed on the assistant's side.

// ⚠️ With the caveat that keeps it honest: escalation is a RATE over
// assistant sessions, and a system nobody uses has a wonderful one.
// It is only meaningful read beside breadth.
const meaningful = breadth > 0.3 && escalationRate < 0.15

Segment it, or it hides the finding

-- The aggregate hides the distribution, the same way chapter 108's
-- aggregate hid the slice table.
select team, count(distinct user_id) as users,
       round(count(distinct user_id)::numeric / max(team_size), 2) as penetration,
       round(avg(questions_per_week), 1) as depth
  from adoption_by_team group by 1 order by 2 desc;

Team	Size	Penetration	Depth
Inside sales	34	0.91	6.8
Operations	18	0.83	4.1
Field sales	41	0.44	2.2
Warehouse	62	0.11	1.3
Finance	12	0.33	1.8

Forty-seven percent overall is three very different situations. Inside sales has adopted it completely; the warehouse has not, and the warehouse is the largest team.

// The warehouse finding, from chapter 145's interviews: their questions
// are about stock locations and pick sequences, which live in a system
// Atlas does not index, and they ask them on a phone in a noisy
// building. Neither is a quality problem and neither was visible in
// the aggregate.

That is a scope decision, not an adoption problem, and the segmented table is what turns one into the other.

What not to measure

const AVOID = {
  'time saved':  'computed from an assumed baseline nobody measured. It is a ' +
                 'number invented to justify a budget, and a sceptical reader ' +
                 'knows it. Measure escalation instead; it is real.',
  'satisfaction scores': '4% response rate, extremes only — chapter 145.',
  'answers generated': 'a measure of supply, not demand.',
  'accuracy, as a headline': 'it needs a judge version to be meaningful ' +
                 '(chapter 112) and it gets quoted without one.',
}

The first is the one that appears in every internal tool review. Atlas's team was asked for it repeatedly and gave the escalation number instead, with the arithmetic shown: 20 percentage points of 1,300 weekly sessions at a measured 6.5 median minutes is about 28 hours a week, stated as a derived estimate with its inputs visible rather than as a headline.

A number whose inputs you can show survives a sceptical question. A number computed from an assumption does not, and offering one costs credibility that the real numbers would have had.

Reporting it

// One page, monthly, same four metrics plus the cohort table. The
// consistency matters more than the completeness: a report that
// changes shape every month cannot show a trend, which is the only
// thing anyone reads it for.
const MONTHLY = [
  'cohort retention table, all cohorts',
  'the four metrics, against last month and launch',
  'penetration by team',
  'escalation rate, with the arithmetic',
  'top 3 reasons from lapsed-user interviews this month',
  'what changed, and what is next',
]

The lapsed-user line is the one that makes the report useful to a reader who is not the team. Everything else says how it is going; that line says what is being done about the part that is not.

Key points

Rising question volume is compatible with the system being useless. It cannot distinguish repeated failed attempts from genuine use.
Use cohort retention. Atlas's twelve-week retention for the full rollout was 41%, against a 142-monthly-users headline.
The fall between week 4 and week 12 is the number to work on, and lapsed-user interviews are the only way to explain it.
Report four metrics: breadth, depth, substitution, and quality-in-use.
Escalation rate answers the business question — 31% to 11% — and it is measured on the side of the team it was meant to relieve.
Escalation is only meaningful beside breadth. A system nobody uses has an excellent escalation rate.
Segment by team. Atlas's 47% was 91% in inside sales and 11% in the warehouse, the largest team.
The warehouse gap was a scope and device problem, not a quality one, and invisible in the aggregate.
Never report computed time saved. Show escalation with its arithmetic instead; a number whose inputs you can show survives a sceptical question.
Keep the monthly report the same shape. Consistency is what lets anyone read a trend.

Next: knowing when to expand scope, and when not to — what to do about the warehouse.