Claude Benchmarks

Models
Claude Code CLI version range

Showing 30 of 93 — the table below each chart always shows every published version.

Effort

11 lines shown

Suite cost (USD)

  • sonnet-5 (high)
  • fable-5 (high)
  • haiku-4-5
  • opus-4-8 (high)
  • opus-5 (high)
  • fable-5-1 (high)
  • opus-5-5 (high)
  • sonnet-5-5 (high)
  • haiku-5-5
  • sonnet-4-6 (high)
  • fable-5-2 (high)
Suite cost (USD)
Data table
Claude Code CLI versionsonnet-5 (high)fable-5 (high)haiku-4-5opus-4-8 (high)sonnet-5 (medium)opus-5 (high)opus-5 (low)opus-5 (max)opus-5 (medium)opus-5 (xhigh)fable-5 (medium)opus-4-8 (medium)fable-5 (max)fable-5 (xhigh)opus-4-8 (max)opus-4-8 (xhigh)sonnet-5 (max)sonnet-5 (xhigh)fable-5-1 (high)fable-5-1 (medium)fable-5-1 (max)fable-5-1 (xhigh)opus-5-5 (high)opus-5-5 (medium)sonnet-5-5 (high)sonnet-5-5 (medium)haiku-5-5sonnet-4-6 (high)sonnet-4-6 (medium)fable-5-2 (high)fable-5-2 (medium)
2.1.1960.39959——————————————————————————————
2.1.2020.3074411.5854730.1386590.7877060.292861——————————————————————————
2.1.2040.435911.5893450.1369380.74730.423905——————————————————————————
2.1.2050.4019361.4129470.1370070.684710.399699——————————————————————————
2.1.2060.4158581.4523280.1340950.7118290.411794——————————————————————————
2.1.2070.3131611.1691420.1517350.4944310.292131——————————————————————————
2.1.2080.3015291.0528040.1372170.4733210.271329——————————————————————————
2.1.2100.2823381.0525050.1358790.4788240.261952——————————————————————————
2.1.2110.3043121.0425010.1351690.4413840.257755——————————————————————————
2.1.2120.3049671.0375620.1318740.4490240.266073——————————————————————————
2.1.2140.2982631.003270.1409860.4706040.270236——————————————————————————
2.1.2150.2896740.989780.1292310.4033630.265379——————————————————————————
2.1.2160.2698610.9324450.1435140.4199730.250677——————————————————————————
2.1.2170.293840.9706180.1380430.4054170.256946——————————————————————————
2.1.2180.2842980.9935260.1284580.4029940.265344——————————————————————————
2.1.2190.2864481.0402090.1328450.4081530.261634——————————————————————————
2.1.2200.2768640.9874170.1255540.401210.2644210.6578460.4326750.8150550.5661230.762884—————————————————————
2.1.2210.2860490.9773720.1319990.401626—0.65512—0.873209———————————————————————
2.1.2220.290281.0981880.1319290.4436370.2685450.646637——0.56267—0.8922970.380577———————————————————
2.1.2230.2960990.9979150.1360630.4195630.263880.654669——0.563948—1.0194590.417344———————————————————
2.1.2240.2921221.0852810.1318630.4418560.2649690.662316——0.574012—0.9313410.397382———————————————————
2.1.2250.294781.0847270.1367270.4247060.262170.68277——0.582747—0.9453280.392409———————————————————
2.1.2260.2989911.0595510.1430380.4353420.2621720.688851——0.593155—0.9245630.423615———————————————————
2.1.2270.2964321.0745690.1320820.4746790.2600690.72558——0.567964—0.9923870.43579———————————————————
2.1.2280.3094031.069730.1336590.4221490.2533760.679035——0.573721—0.9122150.413112———————————————————
2.1.2290.302241.0693660.1361530.4526090.2597790.683558——0.564781—0.9091370.424274———————————————————
2.1.2310.3016371.0460280.1349290.4428980.2613130.709022——0.574353—0.9463310.407458———————————————————
2.1.2320.2967411.0461690.1269880.4481570.2633670.636067——0.593776—0.9346240.429926———————————————————
2.1.2330.2994941.0393480.1310540.3970760.2635680.660987——0.608725—0.8930370.423469———————————————————
2.1.2340.2838711.0497890.1236640.4156980.2677990.627984——0.55549—0.8841930.415903———————————————————
2.1.2350.289561.0271510.1282070.4508770.2511890.621802——0.533325—0.8820080.436997———————————————————
2.1.2360.2875150.9571250.1329530.4545120.2732480.631873——0.516582—0.9239760.445696———————————————————
2.1.2370.2959281.0621910.1316690.4293560.2542270.624628——0.512943—0.9230610.42662———————————————————
2.1.2380.2739720.9590720.1351260.4184910.266260.620326——0.505544—0.9027590.421965———————————————————
2.1.2390.2770271.0058880.1322680.440070.2569250.668432——0.521478—0.8630740.412674———————————————————
2.1.2400.2966051.0500340.1294660.464848—0.627648—————————————————————————
2.1.2410.2848861.0196150.1347390.4329110.2644020.634396——0.495076—0.891140.440323———————————————————
2.1.2430.2813471.0657040.1287770.4390550.2587680.639663——0.510795—0.8950020.418189———————————————————
2.1.2450.2776181.0786320.1276990.4374610.2751640.62811——0.506187—0.9388240.4392———————————————————
2.1.2460.2929351.0263440.1283290.4304040.2625550.598752——0.512927—0.901360.445008———————————————————
2.1.2470.2746491.0693610.128530.4528960.2620640.612011—0.8818190.5096410.7594010.9444050.4315131.4688292.0450820.7101670.5495910.4272450.319708—————————————
2.1.2480.2419690.8507890.1160720.3722710.2210590.537779—0.8072330.4456080.6747770.7865640.3799492.8427551.9037060.6623010.4578670.3874190.280323—————————————
2.1.2500.2433620.9459360.1199910.3729470.226320.55282—0.8285570.4480030.6961820.7608130.3543012.8294691.8714190.6509260.457870.3540360.281948—————————————
2.1.2510.2505430.8707220.1180750.3855710.2351050.53371—0.7870980.4154210.723680.7844720.3599073.0176971.8570150.6708610.4339310.368420.282411—————————————
2.1.2520.250620.8914680.121860.3729190.2360970.527568——0.405826—0.7754540.361293———————————————————
2.1.2570.256590.871960.1149440.3804570.2252490.525918——0.439004—0.7387260.363476———————————————————
2.1.2580.2471040.8190510.118790.383242—0.535461—————————————————————————
2.1.2590.2482020.8399040.1180030.3863570.2331960.526803——0.430426—0.7473310.360691———————————————————
2.1.2600.2480780.8362410.1174410.3671970.2238460.554445——0.415243—0.7873310.365427———————————————————
2.1.2610.2424460.8676810.12130.370430.2223790.554465——0.44107—0.746020.368507——————0.561440.466341———————————
2.1.2630.2465180.9285230.120430.3830420.2290810.580929——0.412345—0.7176380.366587——————0.5733910.454946———————————
2.1.2650.2518230.8802320.1219110.3864370.2277460.601206——0.429159—0.8067860.345241——————0.5429450.468057———————————
2.1.2660.2515990.8250640.1182250.3800740.2265310.582992——0.439395—0.7902820.358274——————0.5311190.47197———————————
2.1.2670.2601240.9137930.1238040.3804950.2235340.584541——0.463294—0.7566740.359009——————0.5362310.471379———————————
2.1.2680.2475740.8471960.1206010.4025120.2333320.572696——0.467918—0.748910.371418——————0.5718520.479094———————————
2.1.2690.2475320.8831610.1254960.376190.2258840.595832——0.46909—0.7923190.367078——————0.5494460.480273———————————
2.1.2700.2480770.8722740.1206720.363450.2177740.606586—0.7883270.4498360.693980.7790550.3839921.3029870.9709250.6123350.4617920.3590380.2803610.5811840.4766682.0297291.025853—————————
2.1.2710.2487050.8750750.1170430.3797880.2287410.607335——0.468559—0.781330.364044——————0.6026640.471878———————————
2.1.2720.2542960.8853350.125930.378320.2281850.650927——0.472291—0.7278330.367279——————0.5648560.485264———————————
2.1.2780.2557920.8764150.1149510.4164640.2288450.63328——0.48095—0.7359230.364167——————0.5924770.488103———————————
2.1.2800.2522070.8822130.1163820.3653320.2308380.602693——0.439992—0.7506660.373698——————0.5720170.47293——0.2623470.215071———————
2.1.2810.24830.8668070.1209140.3752790.2231120.622124——0.478544—0.71620.374442——————0.5839110.470826——0.2708790.2318———————
2.1.2820.241220.8856780.119630.3816610.223760.622266——0.483949—0.7233190.361705——————0.6094780.481824——0.2763170.225929———————
2.1.2830.2460440.859390.124120.3927250.2297950.6315——0.479399—0.7407950.368393——————0.5835290.487228——0.2655120.234032———————
2.1.2840.2478210.8338540.1195820.4006040.23540.608328——0.469789—0.7267280.368811——————0.565290.486625——0.2622670.2256970.1186750.101985—————
2.1.2850.2460520.86160.1200870.3815050.2289270.671058——0.478074—0.7288240.371018——————0.5726130.473703——0.2595440.2311850.116930.102549—————
2.1.2860.2459590.8616610.1242460.3948390.232360.603531——0.474349—0.7449790.367086——————0.5587590.488417——0.2682580.2315450.1139050.10114—————
2.1.2870.2556940.8609430.1305120.3982070.223980.580606——0.486412—0.7322170.372231——————0.5705210.472712——0.2603960.2299050.1124080.097659—————
2.1.2880.2522450.8509350.1146650.3928570.2288340.653173——0.48468—0.7332340.380491——————0.60010.482764——0.2655220.2267150.1170250.101566—————
2.1.2890.253340.9097010.1183780.3974660.2305860.613609——0.476427—0.7582540.38755——————0.587070.498698——0.2708110.2332090.1163110.095377—————
2.1.2900.2643980.9136710.1270230.3766290.2409150.653922——0.528847—0.7432750.371305——————0.595590.493815——0.2670040.2486930.1149140.106292—————
2.1.2930.27512—0.1289760.4111070.2499530.70118——0.539082——0.377414——————0.5480370.468933——0.2657030.240930.1147350.1008520.0124070.3174120.286383——
2.1.3010.274533—0.1291010.4034380.2402840.67537——0.517589——0.402355——————0.6129230.494517——0.2801730.2572480.1172120.1028520.013514————
2.1.3020.27476—0.129450.408780.2453850.740242——0.530708——0.380468——————0.5740910.504586——0.2733630.2391830.1158890.103310.013046————
2.1.3030.276517—0.1377230.3890240.2469570.737249——0.523192——0.364898——————0.5964190.49159——0.2876140.2390360.1265070.1016580.012874————
2.1.3040.285516—0.125910.4131520.2418880.704236——0.581775——0.368391——————0.5882240.501371——0.2721620.2436860.1180360.0997020.013014————
2.1.3050.269313—0.1289560.406630.2396640.687509——0.540101——0.368287——————0.5527770.499523——0.2753690.234560.1151570.1002870.01288————
2.1.3060.292285—0.1283550.4159160.2406750.696166——0.536437——0.35114——————0.5943120.48451——0.26567—0.114957—0.013645————
2.1.3090.271531—0.1315350.4047780.2467780.718432——0.544257——0.38016——————0.618490.500838——0.2722960.2461570.1181410.0980340.012776————
2.1.3100.2726—0.1266660.4107760.2436380.662596——0.534537——0.39158——————0.6121380.493487——0.2838740.2604080.1181330.1039020.013107————
2.1.3110.25525—0.1218140.3873150.2410230.712077——0.568956——0.36072——————0.5995460.50011——0.2885960.2386030.1223420.1004480.012357————
2.1.3120.272963—0.1293090.3999720.2392280.720583——0.54346——0.370936——————0.648775———0.276535—0.118213—0.013254————
2.1.3180.285755—0.1204860.3941220.2414990.709363——0.527107——0.384219——————0.5616250.486796——0.2811360.2270970.115950.0976650.012456————
2.1.3210.272182—0.1252450.4387310.2419370.651376——0.548398——0.366116——————0.596970.504014——0.2646780.2388460.1168890.0973950.012379——0.5273940.348641
2.1.3220.259945—0.126160.4398880.2392710.672562——0.529529——0.369822——————0.5955990.481277——0.2791340.2434560.1190070.0996490.01294——0.4720550.353539
2.1.3230.278131—0.1234470.407840.2385850.699652——0.523971——0.376634——————0.5906150.492405——0.2853880.2349160.1209370.1010510.013744——0.5843630.357154
2.1.3240.264695—0.126850.4209630.2445830.713214——0.49357——0.372694——————0.6221370.514318——0.2782030.2491880.1185460.1003070.013532——0.4428380.336239
2.1.3250.274011—0.1304890.4364780.2417920.719755——0.482108——0.374326——————0.632440.504899——0.2952320.2336590.118750.1003230.013501——0.4313510.34793
2.1.3260.267158—0.124790.4147250.2487150.694019——0.538314——0.384619——————0.5949710.49——0.2915060.2312340.113610.1077110.013273——0.5047570.355548
2.1.3270.266702—0.1317390.4277320.2420770.721012——0.531878——0.413209——————0.6082950.495829——0.276010.2371030.1188830.102060.013062——0.4434850.362947
2.1.3280.27141—0.1259190.4074490.2568950.69162——0.531173——0.392702——————0.6145740.518693——0.2791670.2420420.1132040.0982690.013588——0.5579310.395189
2.1.3290.279635—0.1279010.4018720.2386960.699724——0.551297——0.366651——————0.5925040.499057——0.2871360.2422590.1261650.101620.014354——0.5170040.385566
2.1.3300.26751—0.1308540.4071890.2407570.686058——0.555319——0.394668——————0.6407670.476908——0.2724170.2396440.118540.0989030.013456——0.5343760.356969
highnightly-2.1.330

Data table
Metricfable-5-1 (high)fable-5-2 (high)haiku-4-5haiku-5-5opus-4-8 (high)opus-5 (high)opus-5-5 (high)sonnet-5 (high)sonnet-5-5 (high)
Cost per session (USD)mean0.0902610.0670330.0166410.0015990.0561640.086850.0353990.0342140.015943
Cost per session (USD)median0.0959910.0429080.0163360.0015230.0596620.0818070.0302810.0367270.014024
Output tokens600489789984.5454778435390398
Input tokens66626868686
Cache-read tokens57994567196702969971.548728746055268911469552587
Cache-write tokens117262712891386743157363310410
Input + cache tokens59038576846774474835.552829759795337511941852930
API calls333434343
Tool calls22.523.523233
Thinking blocks003211010
Latency (s, wall-clock)15.8115.82158.3436.4857.48814.7885.12258.6466.877
Suite cost (USD)No published column carried Suite cost (USD) for this cycle.
mediumnightly-medium-2.1.330

Data table
Metricfable-5-1 (medium)fable-5-2 (medium)opus-4-8 (medium)opus-5 (medium)opus-5-5 (medium)sonnet-5 (medium)sonnet-5-5 (medium)
Cost per session (USD)mean0.0649460.0485590.0501270.0668840.0309190.0295560.013662
Cost per session (USD)median0.0642060.0409790.0518460.0626710.0261530.0323920.010789
Output tokens473400444573367368344
Input tokens66666686
Cache-read tokens546145323552388562214918211872552215
Cache-write tokens630017111224547490
Input + cache tokens586965688552829573455305411934352731
API calls3333343
Tool calls2222232
Thinking blocks0000010
Latency (s, wall-clock)15.2674.517.238.0236.7058.2175.418
Suite cost (USD)No published column carried Suite cost (USD) for this cycle.

Claude Code CLI 2.1.330 Analysis

Measured 2026-11-21 02:22 UTC from this cycle’s benchmark runs, written up from those measurements alone, then checked by an independent reviewer. Where the wording and the tables disagree, the tables are correct.

Verdict

In the high reasoning setting, one task costs this per session, cheapest to most expensive:

  • haiku-5-5 $0.0016
  • sonnet-5-5 $0.0159
  • haiku-4-5 $0.0166
  • sonnet-5 $0.0342
  • opus-5-5 $0.0354
  • opus-4-8 $0.0562
  • fable-5-2 $0.0670
  • opus-5 $0.0868
  • fable-5-1 $0.0903

In the medium reasoning setting, the same task costs:

  • sonnet-5-5 $0.0137
  • sonnet-5 $0.0296
  • opus-5-5 $0.0309
  • fable-5-2 $0.0486
  • opus-4-8 $0.0501
  • fable-5-1 $0.0649
  • opus-5 $0.0669

Across the last 10 cycles in the high setting, each family moved like this, with the dollar band it stayed inside:

  • fable-5-1 drifted up, $0.0771 to $0.0903
  • fable-5-2 no clear direction, $0.0593 to $0.0755
  • haiku-4-5 no clear direction, $0.0147 to $0.0167
  • haiku-5-5 no clear direction, $0.0015 to $0.0018
  • opus-4-8 no clear direction, $0.0527 to $0.0602
  • opus-5 no clear direction, $0.0827 to $0.0901
  • opus-5-5 no clear direction, $0.0351 to $0.0402
  • sonnet-5 no clear direction, $0.0312 to $0.0357
  • sonnet-5-5 no clear direction, $0.0154 to $0.0180

Across the last 10 cycles in the medium setting:

  • fable-5-1 no clear direction, $0.0649 to $0.0781
  • fable-5-2 no clear direction, $0.0473 to $0.0529
  • opus-4-8 no clear direction, $0.0460 to $0.0527
  • opus-5 no clear direction, $0.0607 to $0.0686
  • opus-5-5 no clear direction, $0.0305 to $0.0318
  • sonnet-5 no clear direction, $0.0288 to $0.0308
  • sonnet-5-5 no clear direction, $0.0134 to $0.0146

This cycle has no change figures against the previous CLI version. In the high setting and in the medium setting alike, no family has runs on both sides of the 2.1.329 to 2.1.330 comparison, so no percentage move could be measured and nothing could be checked against normal run-to-run variation. Costs now and the bands above are what this cycle supports.

One other thing: in the high setting the two ends of the list are far apart. Measured against fable-5-1, haiku-5-5 comes in 98.2% lower, and comparing the individual run costs of the two families gives p=0.000.

Which cost figure is which – setting high

No family has sessions on both sides, so there is nothing to reconcile.

Which cost figure is which – setting medium

No family has sessions on both sides, so there is nothing to reconcile.

What these terms mean

New to this report? Here is the vocabulary used below, in plain words. Skip ahead if you already know it.

  • Session (also “task”): one complete run of one prompt by one model, start to finish. Every cost in this report is per session.
  • Family: a model line, such as opus or sonnet. One family can cover more than one version, for example opus-4-8 and opus-5.
  • Cache, warm and cold: the tool reuses recent context instead of sending it again, which is far cheaper. A session that gets to reuse it is “warm”; one that has to send everything fresh is “cold”. Whether a given session lands warm or cold is partly luck, and that luck moves the cost.
  • Cache luck (or cache-hit luck): the cost swing caused by how many sessions happened to land warm rather than cold, rather than by any real change in the tool or the model.
  • Blended cost: cost measured across every session, warm and cold alike. This is the real money figure, and it is what the cost feed shows. “Blended” says which sessions are counted, not how they are averaged; see “Which cost figure is which” above for the averaging.
  • Warm-only cost: cost measured across warm sessions only. Removing the cold ones removes most of the luck, so this is the fair test of whether cost actually changed.
  • Pooled: all sessions for a family added together into a single number, rather than split out per prompt.
  • Mean cost per session: the ordinary average across every session. It multiplies out to real spend, so it is the budgeting figure and the one this report leads with.
  • Median cost per session: the middle session once they are lined up by cost. It ignores an unusually cheap or expensive run, but it does not add up to real spend.
  • Suite cost: the price of one full pass over every prompt in the catalog, found by taking each prompt’s median and adding those up. It is a total for the whole catalog, not an average per session, so it is a larger number than either figure above and is not comparable to them.
  • Cell: one family and one prompt combination, for example haiku on the rename-refactor prompt. Cells hold few sessions, so a single cell is noisy.
  • n: how many sessions sit behind a number. A larger n is more trustworthy.
  • p-value, and “significant”: the same prompt run twice costs different amounts, so a p-value asks whether two sets of runs can be told apart at all. Line up every run from both cycles: if one cycle’s runs sit consistently higher, p is near 0 and the difference is called “significant”; if the runs are mixed together, p is near 1 and the two sets look like the same thing measured twice. Precisely, it is the chance of seeing a gap at least this lopsided if the two cycles were truly identical. A high p is not weak evidence of a change, it is no evidence either way.
  • API-equivalent cost: dollars calculated at published API rates. It is not necessarily dollars billed, because a subscription plan may already cover the usage.

Pooled blended comparison

This is the headline lens: real per-task spend in dollars per session, counting every session of the cycle, which is what a user actually pays.

Setting high (nightly-2.1.329 to nightly-2.1.330):

No family has sessions on both sides – no comparison possible.

Setting medium (nightly-medium-2.1.329 to nightly-medium-2.1.330):

No family has sessions on both sides – no comparison possible.

So for both settings there is no pooled blended change to report this cycle. The current per-session levels for each setting are the ones given above.

Pooled warm comparison (significance control)

This lens answers the separate question of whether a change is real, by setting aside the luck of which runs started with a warm cache and comparing only the warm runs from each cycle.

Setting high:

No family has sessions on both sides – no comparison possible.

Setting medium:

No family has sessions on both sides – no comparison possible.

With no family measured on both sides at either setting, there is no significance test to run this cycle. That does not mean costs held still: it means this measurement cannot see a move either way.

What the reasoning setting costs

Blended mean $/session per model, medium measured against high. A negative percentage means medium is the cheaper setting. The p-value is Mann-Whitney over the two pooled session sets; this test is available here because both sides carry per-session rows.

model high mean $ medium mean $ change p n
fable-5-1 0.0903 0.0649 -28.0% 0.010 61/53
fable-5-2 0.0670 0.0486 -27.6% 0.337 54/53
opus-4-8 0.0562 0.0501 -10.7% 0.416 61/53
opus-5 0.0868 0.0669 -23.0% 0.014 61/59
opus-5-5 0.0354 0.0309 -12.7% 0.414 54/53
sonnet-5 0.0342 0.0296 -13.6% 0.042 61/53
sonnet-5-5 0.0159 0.0137 -14.3% 0.193 54/53

Medium is the cheaper setting in every row of this table. Comparing the individual run costs at each setting, three of those differences clear the 0.05 line:

  • fable-5-1 costs 28.0% less at medium, p=0.010
  • opus-5 costs 23.0% less at medium, p=0.014
  • sonnet-5 costs 13.6% less at medium, p=0.042

For the other four, the difference cannot be told apart from normal run-to-run variation:

  • fable-5-2 costs 27.6% less at medium, p=0.337
  • opus-4-8 costs 10.7% less at medium, p=0.416
  • opus-5-5 costs 12.7% less at medium, p=0.414
  • sonnet-5-5 costs 14.3% less at medium, p=0.193

Two models were measured at one setting only and so have no cross-setting figure: haiku-4-5 and haiku-5-5.

Notable cells (secondary)

This lens looks for individual prompts that moved more than the rest. There is nothing to report: in the high setting and in the medium setting, no cell moved beyond the noise threshold, and with no family measured on both sides there were zero cells available to compare in either setting. Single prompts are a watch-next-cycle signal anyway, never a result on their own: each cell holds only about five runs, and scanning many prompts at once means some will look like movers by chance, so any such cell is quoted with its run count and its p-value and read as something to confirm later.

Caveats

  • Two cost lenses are reported. The headline is the real per-task spend, counting every session, including the luck of whether a run started with a warm cache; that is the figure that matches the dashboard. The warm-only lens strips that cache luck out and is the control for whether a change is real. Judge “did cost change” by the warm comparison, not by the headline move. A headline move with no matching warm result may just be cache luck rather than a real change.
  • The compared cycles ran on different CLI versions (2.1.329 then 2.1.330). Any pooled difference therefore measures the effect of the CLI version, which is what this benchmark exists to track. Do not read it as a change in the model itself unless a run that holds the CLI version fixed says so.
  • Token capture for the Haiku models is known to be incomplete, so their cost and token figures should be treated as lower bounds: the true numbers are at least that much.
  • Timing is wall-clock, so it includes the time the model spends thinking. Cost is calculated as an API-equivalent figure, not read off a bill, so it is an estimate of spend rather than dollars actually charged.

Verification appendix

  • Verifier verdict: PASS after 1 round(s).