Benchmark Auto-Analysis – nightly-2.1.220

Generated 2026-07-25 21:39 UTC by scripts/benchmarks/analyze-digest.py: claude-opus-4-8 narrative over docs/benchmarks/digest-nightly-2.1.220.md, adversarially verified. The digest is the ground truth; on any conflict trust the digest.

Verdict

Blended cost — the real per-task spend a user actually pays, cache luck and all — moved only modestly this cycle: haiku fell 6.6% and sonnet fell 6.4% (the largest moves), fable slipped 2.0%, and opus-4-8 edged up 1.6%. But the warm-only control — which strips out random cache-hit luck to test whether a change is real — shows no significant move for any family (all warm Mann-Whitney p-values are far above 0.05). So these blended shifts are likely cache luck, not real cost changes from 2.1.219 to 2.1.220.

Pooled blended comparison

“Pooled” means all sessions for a family are combined into one number; “blended” means every session counts, including cold-start cache misses — this is the headline that matches the dashboard and cost feed.

family	prev $/sess	cur $/sess	delta %	blended p
fable-5	0.1208	0.1184	-2.0%	0.649
haiku-4-5	0.0158	0.0147	-6.6%	0.613
opus-4-8	0.0495	0.0503	+1.6%	0.937
sonnet-5	0.0493	0.0461	-6.4%	0.992

The two biggest headline moves are haiku (-6.6%) and sonnet (-6.4%); opus-4-8 is the only family that rose (+1.6%).

Pooled warm comparison (significance control)

“Warm-only” restricts to sessions where the cache was already warm, removing the random luck of a cold-start cache miss. It answers the separate question: did cost actually change, or did the blended number just get lucky? For that question, trust these p-values, not the blended deltas above.

family	prev $/sess	cur $/sess	delta %	warm p
fable-5	0.1117	0.1047	-6.3%	0.413
haiku-4-5	0.0148	0.0135	-8.8%	0.529
opus-4-8	0.0446	0.0450	+0.9%	0.888
sonnet-5	0.0431	0.0430	-0.2%	0.990

Every warm p-value is well above 0.05, so none of the blended moves clears the bar for a real change. Notably, sonnet’s 6.4% blended drop shrinks to just -0.2% once cache luck is removed — a clear sign the headline dip was cache-driven, not a genuine cost reduction.

Notable cells (secondary)

Per-cell movers are a much weaker lens: these are single family-and-prompt combinations, each with small samples, drawn from 36 simultaneous comparisons. With that many tests you’d expect roughly one false positive at p<0.05 for every 20 cells, so no single cell is a finding on its own.

cell	warm delta %	n (prev/cur)	cost p	blended delta %
haiku · fix-off-by-one	-33.5%	6/6	0.076	-33.6%
haiku · add-dry-run-flag	+17.3%	6/6	0.117	+8.7%
sonnet · rename-refactor	+16.4%	6/6	0.917	+15.8%

None of these reaches significance (all p ≥ 0.076), so treat them as noise to watch, not results.

Cross-model (within cycle; blended)

These compare cost per task on identical prompts within this cycle — not per-token rates. A cheaper-rated model can still out-cost a pricier one by taking more agentic steps.

Blended cost per session, cheapest to most expensive: haiku-4-5 ($0.0147) « sonnet-5 ($0.0461) < opus-4-8 ($0.0503) < opus-5 ($0.0665) « fable-5 ($0.1184). Fable is by far the most expensive family; every pairing against it is significant (p=0.000), and it is costlier on all 9 shared prompts against each of the others. Haiku is cheapest across the board, costlier on 0 of 9 prompts against every rival, all significant.

• Near-parity pair: opus-4-8 vs sonnet-5 is the one pairing that is not significant (delta -8.3%, p=0.420) — these two are effectively tied on pooled cost.
• Cheaper-pooled-but-costlier-on-prompts pair: in that same opus-4-8 vs sonnet-5 pair, sonnet is cheaper pooled, yet opus-4-8 comes out cheaper on 3 of the 9 shared prompts while sonnet is cheaper on 6 — a reminder that a pooled edge doesn’t hold on every prompt.
• Within the opus family, opus-5 is significantly costlier than opus-4-8 (+32.3%, p=0.019), costing more on 8 of 9 prompts.

The per-prompt median matrix (e.g. rename-refactor at fable 0.1878 vs haiku 0.0258) uses single cells of about 5 sessions each — use those numbers only as illustration, not as standalone findings.

Behavior

These warm medians (current vs prev 2.1.219) carry no significance test — they only help explain a cost move, and are never findings on their own.

• fable: 3 (3) API calls, 2 (2) tool calls, output tokens 491 (523), cache-read 62862 (63273). Slightly fewer output tokens, consistent with its small -2.0% blended dip.
• haiku: 3 (3) API calls, 2 (2) tool calls, output tokens 568 (558), cache-read 71712 (69225). Step counts unchanged; nothing here points to a real cost change behind the -6.6% blended move.
• opus-4-8: 3 (3) API calls, 2 (2) tool calls, output tokens 458.5 (459), cache-read 59857.5 (58908). Essentially flat behavior, matching its tiny +1.6% blended change.
• sonnet: 4 (4) API calls, 3 (3) tool calls, output tokens 366 (367), cache-read 127517 (127517). Behavior is unchanged down to identical cache-read — reinforcing that sonnet’s headline -6.4% is cache luck, not a behavior shift.

In short: no family changed its step count or tool usage, so nothing in behavior supports treating the blended moves as real.

Trend

Blended mean $/session has been stable across recent cycles. Fable has hovered between 0.1129 and 0.1218 (now 0.1184 at n=54); haiku around 0.0147–0.0167 (now 0.0147); opus-4-8 near 0.0495–0.0567 (now 0.0503); sonnet 0.0444–0.0493 (now 0.0461). This cycle (nightly-2.1.220) is the first with opus-5 in the series, entering at 0.0665 (n=53). No family shows a sustained departure from its recent band.

Caveats

• Two lenses, one control. Blended is the real per-task spend including cache luck (the headline); warm-only strips that luck and is the test for whether cost really changed. Judge real change by the warm p-value, not the blended delta — a blended move with no significant warm move may just be cache luck.
• The CLI version changed (2.1.219 → 2.1.220), so any pooled delta measures the effect of the CLI version — that’s what this benchmark exists to test. Do not attribute a delta to the model itself without a same-CLI backtest.
• Per-cell movers span 36 simultaneous comparisons, where about one false positive per 20 cells at p<0.05 is expected by chance; a single significant cell is not a finding.
• Cross-model rows compare cost per task, not per-token rates: a lower-rated model can cost more by taking more agentic steps, and the per-prompt medians are single small cells (about 5 sessions) — illustrative only.
• Behavior medians are descriptive with no significance test — use them to explain a cost move, never as findings on their own.
• Haiku token capture is known to be incomplete, so treat all Haiku cost and token figures as lower bounds.
• Latency is wall-clock (it includes model/thinking time), and cost is API-equivalent from the hook calculation — not dollars actually billed.

---

Verification appendix

• Verifier verdict: PASS after 2 round(s).
• A first draft was rejected and revised; the objections were resolved.