Empirical audit · Claude Code transcripts
I measured it against my own logs instead of arguing about it. Some things moved, one didn't, one cannot be measured at all — and every figure on this page is regenerated from the transcripts, none typed in by hand.
Requests ending in a concession
1.87×
12.9% → 24.2% per request
Fisher exact p = 0.00051
Blind-labelled friction
2.67×
12% → 32% per request
Fisher exact p = 0.028
Output tokens in conceded requests
26.9%
up from 19.4%
share of all generated tokens
Claude Code writes every session to disk as JSONL, timestamped, with the model recorded per message. That makes a personal, longitudinal dataset: not benchmark scores, but what actually happened across months of real work — homelab automation, PCB design, firmware bring-up, media infrastructure.
The comparison is opus-4-8 against opus-5. Everything is normalised per request — one thing I asked for — rather than per message, because a chattier model emits more messages for the same work. Dividing by messages hides the effect behind the model's own verbosity.
The unit is a request that ends with Claude conceding it was wrong. Most concessions credit the human — phrasing Claude cannot use unless it was just contradicted — so they largely track errors the human caught; the remainder are Claude flagging its own error unprompted, and the verdicts split the two. What a rising concession rate cannot do by itself is name the cause: more errors, a readier concession reflex, better error-recognition, and a harder mix of work all move it the same way. The blind labelling in §4 checks the first of those; the limitations in §5 are the honest budget for the rest.
Change from opus-4-8 to opus-5, per request
multiplier · baseline 1.0 = no change
| Measure | opus-4-8 | opus-5 | × | p |
|---|
Share of requests that ended with Claude admitting it was wrong
by week · bar colour = dominant model that week
| Week | Requests | Conceded | Token burn | Model |
|---|
Automated counting gives ×1.87 (p = 0.00051), taking the share of requests that end with Claude conceding from 12.9% to 24.2%. Blind hand-labelling of 100 sampled exchanges — model identity withheld until every label was fixed — agrees: ×2.67 (p = 0.028).
A concession is an admission, not a verified error. More actual mistakes move this number — but so would a model that concedes more readily, one that recognises its own errors better, or a harder mix of work arriving in the same weeks. The strictest blind label — a caught, corrected error — went 3→8 of 50: same direction, but a sample too small to stand alone (p = 0.20).
Decomposed: my explicit pushback rate did not move (3.9% → 3.0% of requests, p = 0.66), so “the human simply objected more” is not what the data shows. Concessions that credit me rose from 9.7% to 14.4% (p = 0.081); admissions that credit no one — Claude flagging its own error unprompted — rose from 3.2% to 9.7% (p = 0.001) and are the largest mover. Whether those self-flagged corrections are real errors caught mid-task or a more self-correcting narration style, the transcripts cannot say.
Out-of-sample: restricted to opus-5 requests from weeks beginning 2026-08-10 or later — data that did not exist when this analysis was first run — the rate is 22.4% (n = 107) against the same baseline, p = 0.029. The effect is not an artifact of the window that produced it.
Median output tokens per request rose 109% (p = <0.0001) and tool calls per request rose 120% (p = <0.0001). The share of all generated tokens spent inside requests that end in a concession went from 19.4% to 26.9%. Rising tool calls and tokens are as consistent with a deliberately more thorough agentic style as with churn — the churn proxies in the final verdict do not separate the two — but the cost per request is real either way.
Meanwhile the characters I type per request rose 24% (p = 0.077). Front-loading more context is not preventing the friction; both moved together.
Hand-labelled restatement went from 6% to 8% of requests — p = 1.00.
One real blind spot: the labeller saw only two prior turns within a single session, so re-teaching the same lesson in a new session is invisible to this method. That is a different measurement, and it has not been built yet.
Thinking content is stored as an empty string plus an encrypted signature for every model except sonnet-4-6. There is no reasoning text to measure. Any claim that “thinking got shorter” would be an artifact of a storage policy change, not a finding.
Behavioural proxies for flailing: self-reversals 4→5 per 100 requests, p = 0.44; failed edits 4→6 per 100 requests, p = 0.26; duplicate commands 4→5 per 100 requests, p = 0.85; repeated file reads 35→14 per 100 requests, p = <0.0001. More tokens and more tool calls per request are equally consistent with thoroughness and with churn; nothing here separates them.
Everything is computed from local transcripts by scripts that re-read them live. Three guards matter:
Limitations — read before citing this
Model and date are the same variable. Model assignment was never randomised, so anything that changed over calendar time is indistinguishable from anything that changed with the model.
Task mix shifted. The opus-5 sessions skew toward harder debugging, where errors are likelier regardless of which model is running.
The labeller was Claude. Blinding removes the model identity, but writing style may still leak the answer. An independent human labeller on the same sample would settle it.
The headline metric measures admission, not error. A future model that concedes less readily without actually improving will look like progress on this chart. Periodic blind labelling — last run on 100 exchanges — is the only thing that catches that.
One machine's transcripts cannot separate “this model got worse”
from “this user's work got harder.” More datasets can. The
tooling is a few Python files with no dependencies beyond the standard
library, and it never transmits anything — it reads
~/.claude/projects and writes locally.
python3 snapshot.py archives your transcripts, appends weekly
per-model metrics, and rebuilds this page. It takes about two seconds and
is safe to re-run. What is worth sharing back is
metrics_history.jsonl: counts only, no message content.
Before you share anything
Raw transcripts contain plaintext secrets. Mine held a wifi password, personal email addresses, and vault variable names — none of which I put there deliberately. The archive directory is gitignored for that reason. Never push it, and check what you are sending before you send it.
One setting is worth changing immediately whether or not you run any of
this: cleanupPeriodDays defaults to 30, so
your transcripts are on a rolling window that quietly deletes its own
baseline. Mine was about three weeks from being unrecoverable when I
checked.
Generated 2026-08-30 from 1,085 requests · 2026-06-01 to 2026-08-31
· models: fable-5, opus-4-8, opus-5, sonnet-4-6, sonnet-5 · rebuild with
python3 report.py