How we know

Every measurement

144 measurements, each quoted word for word from the README, the changelog, a lab note or a benchmark write-up, with its 95% range, its runs, model and date, and the raw runs behind it.

The work
Real jobs in open-source code: click in Python, chrono in TypeScript, and real bug reports from SWE-bench Verified. Graded by fixed rules.
The runs
Claude Code without a person at the keyboard, on Opus 5.5, a fresh copy of the code each time. Each condition names exactly which mods load.
The cost
Counted from the transcripts at list prices. On a subscription that is an estimate, not a bill.
The range
A 95% bootstrap range over runs within each job. When it includes no change, we say the difference is not shown. How we measure

144 of 144 shown. Download facts.json

The dot is the measured change, the bar around it the 95% range, the upright line no change. Left of the line is cheaper or fewer.

DetailsModRuns
Graded runs that ship with the releasegraded runs, 26 short of full marks the pack 2,1842,184 2,184 runs 13 and 16 Oct
Runs short of full marks with the pack, in batches that ran the same jobs both waysagainst 4 of 325 without the pack the pack 4 of 3594 of 359 359 with the pack, 325 without 13 and 16 Oct
Runs short of full marks without the pack, in the same batchesagainst 4 of 359 with the pack the pack 4 of 3254 of 325 325 without the pack, 359 with 13 and 16 Oct
One step in a session that holds 220,000 tokens, before Claude thinks at allworked out from list prices: one cache read of the whole conversation the pack $0.044$0.044 worked out, not measured 15 Oct
The pack in a long session, both days pooledthe pack against plain Claude Code the pack 33% cheaper33.2% cheaper95% range 26.9 to 38.9% cheaper 2 per job and condition, each day 13 and 15 Oct
The pack in a fresh session, both days pooledthe pack against plain Claude Code the pack 4% cheaper4% cheaper95% range 1 to 7% cheaper 224 runs behind the README's table 13 and 15 Oct
Bigger jobs (a feature, tests, a rename) in a fresh sessionthe pack against plain Claude Code the pack 8% cheaper8% cheaper95% range 3 to 12% cheaper 12 each way 13 and 15 Oct
Bigger jobs (a feature, tests, a rename) in a long sessionthe pack against plain Claude Code the pack 37% cheaper37% cheaper 2 per job and condition, each day 13 and 15 Oct
Quick questions in a fresh session, both days pooledthe pack against plain Claude Code the pack about even2.3% more expensive95% range from 1.1% cheaper to 6.0% more expensive 8 per task and condition on the first day, 4 on the second 13 and 15 Oct
Writing tests with the pack alonethe pack against plain Claude Code the pack 2% cheaper2% cheaper 12 each way 13 and 15 Oct
The pack as it shipped in 0.3.0, in a long sessionthe pack against plain Claude Code the pack 26% cheaper26.2% cheaper95% range 20.9 to 31.3% cheaper 2 per task and condition 13 and 14 Oct
0.3.0 in a long session, multi-step tasksthe pack against plain Claude Code the pack 29% cheaper29.1% cheaper95% range 22.7 to 34.6% cheaper 2 per task and condition 13 and 14 Oct
0.3.0 in a long session, quick questionsthe pack against plain Claude Code the pack about even5.4% cheaper95% range from 14.8% cheaper to 5.6% more expensive 2 per task and condition 13 and 14 Oct
The pack as it shipped in 0.3.0, in a fresh sessionthe pack against plain Claude Code the pack 5% cheaper5.3% cheaper95% range 0.5 to 9.8% cheaper 4 per task and condition, 8 for the four small tasks 13 and 14 Oct
0.3.0 in a fresh session, multi-step tasksthe pack against plain Claude Code the pack 11% cheaper10.9% cheaper95% range 3.9 to 17.2% cheaper 4 per task and condition 13 and 14 Oct
0.3.0 in a fresh session, quick questions: the quick-question taxthe pack against plain Claude Code the pack 6% more expensive5.5% more expensive95% range 1.2 to 10.2% more expensive 8 per task and condition 13 and 14 Oct
The pack on the second day, fresh, with plain really plainthe pack against plain Claude Code the pack 6% cheaper5.7% cheaper95% range 1.6 to 9.6% cheaper 4 per task fresh 15 Oct
The pack on the second day, long session (warm runs)the pack against plain Claude Code the pack 37% cheaper37.4% cheaper95% range 33.9 to 40.9% cheaper 2 per task in a long session 15 Oct
Quick questions on the second day: the tax did not replicatethe pack against plain Claude Code the pack about even3.6% cheaper95% range from 8.2% cheaper to 1.3% more expensive 4 per task and condition 15 Oct
The pack of 0.1.1 on the average task: the first interval clear of zerothe pack against plain Claude Code the pack 11% cheaper11.2% cheaper95% range 5.4 to 16.6% cheaper 4 per condition on multi-step tasks, 6 on small ones 13 Oct
The pack of 0.1.1 on multi-step workthe pack against plain Claude Code the pack 18% cheaper17.7% cheaper95% range 9.3 to 25.6% cheaper 4 per condition 13 Oct
The pack of 0.1.1 on small tasksthe pack against plain Claude Code the pack about even0.7% more expensive95% range from 2.4% cheaper to 4.2% more expensive 6 per condition 13 Oct
Requests per task: the pack saves steps, not tokens per stepthe pack against plain Claude Code the pack 5.2 against 7.25.2 against 7.2 2 per task and condition 13 Oct
Requests per task in a long session (0.3.0)the pack against plain Claude Code the pack 3.5 against 5.43.5 against 5.4 2 per task and condition 13 and 14 Oct
The pack on a TypeScript library, all eight tasksthe pack against plain Claude Code the pack 8% cheaper7.6% cheaper95% range 3.1 to 12.1% cheaper 64 runs, 4 per task and condition 15 Oct
TypeScript, multi-step tasksthe pack against plain Claude Code the pack about even6.3% cheaper95% range from 12.5% cheaper to 0.8% more expensive 4 per task and condition 15 Oct
TypeScript, quick questions: symbol lookups answered inside the searchthe pack against plain Claude Code the packlsp-plus 10% cheaper9.9% cheaper95% range 5.9 to 13.7% cheaper 4 per task and condition 15 Oct
TypeScript feature: requeststhe pack against plain Claude Code the pack 10.5 against 13.210.5 against 13.2 4 per condition 15 Oct
The pack on 30 real bug reports (SWE-bench Verified)the pack against plain Claude Code the pack about even12.3% cheaper95% range from 32.5% cheaper to 11.9% more expensive 1 attempt per bug and condition 15 Oct
Bugs fixed out of 30the pack against plain Claude Code the pack 30 against 2930 against 29 1 attempt per bug and condition 15 Oct
Requests per bug on SWE-bench Verifiedthe pack against plain Claude Code the pack 9.9 against 12.99.9 against 12.9 1 attempt per bug and condition 15 Oct
A 'plain' baseline that had the pack installed by mistakethe pack against a plain run that loaded the installed pack 0.4.0 the pack about even2.1% more expensive95% range from 2.4% cheaper to 7.0% more expensive kept apart, labelled for what they were 15 Oct
Taking cells out of the packthe pack without cells against the pack cells 9% more expensive8.9% more expensive95% range 2.1 to 16.4% more expensive 4 per condition on multi-step tasks, 2 on small ones 13 Oct
Taking cells out, on multi-step workthe pack without cells against the pack cells 16% more expensive16.3% more expensive95% range 5.4 to 29.5% more expensive 4 per condition 13 Oct
The feature's requests without cellsthe pack without cells against the pack cells 14.2 against 9.814.2 against 9.8 4 per condition 13 Oct
The pack without cells, against plain: the saving is gonethe pack without cells against plain Claude Code cellsthe pack about even3.3% cheaper95% range from 8.7% cheaper to 2.4% more expensive 4 per condition on multi-step tasks, 2 on small ones 13 Oct
cells' chaining guidance, Haiku on small taskswith the chaining guidance against without cells 18% fewer steps18% fewer steps 30 per arm 9 Oct
cells' guidance moved before the cache boundaryguidance as a shared system-prompt section against as it was cells about even1.0% more expensive95% range from 5% cheaper to 8% more expensive 32 runs 14 Oct
In 240 real sessions, files changed by script around the Edit toolBash scripts that changed files against Edit and Write calls cells 5,996 against 2,7205,996 against 2,720 240 sessions not stated
The language server's answer on a rename: requeststhe pack answering symbol greps against the pack lsp-plus 25% fewer steps25% fewer steps 7 per arm 13 Oct
The language server's answer on a rename: costthe pack answering symbol greps against the pack lsp-plus 7% cheaper6.5% cheaper 7 per arm 13 Oct
The language server's answer on a trace: cost eventhe pack answering symbol greps against the pack lsp-plus 1% more expensive1% more expensive 7 per arm 13 Oct
Taking lsp-plus out of the pack: no net effect on the wholethe pack without lsp-plus against the pack lsp-plus about even0.6% cheaper95% range from 7.2% cheaper to 6.7% more expensive 4 per condition on multi-step tasks, 6 on small ones 13 Oct
lsp-plus's first warm-up leaked a file's diagnostics into small tasksthe pack without lsp-plus against the pack (small tasks) lsp-plus 5% cheaper4.5% cheaper95% range 1.7 to 7.1% cheaper 6 per condition 13 Oct
The warm-up fixed: small tasks against plainthe pack with the fixed warm-up against plain Claude Code lsp-plusthe pack about even2.2% cheaper95% range from 5.6% cheaper to 1.5% more expensive 28 runs of the fixed pack 13 Oct
The warm-up fixed: small tasks against the pack without lsp-plusthe pack with the fixed warm-up against the pack without lsp-plus lsp-plus about even1.7% more expensive95% range from 1.5% cheaper to 4.9% more expensive 28 runs of the fixed pack 13 Oct
The rename with the fixed pack, against plainthe pack with the fixed warm-up against plain Claude Code lsp-plusthe pack 19% cheaper19% cheaper95% range 4 to 32% cheaper 28 runs of the fixed pack 13 Oct
A note telling Claude which language-server call answers its grepnotes acted on, of notes shown lsp-plus 0 of 450 of 45 45 notes 13 Oct
blast (callers after a signature change), taken outthe ideas without blast against with it lsp-plus about even3.7% more expensive95% range from 2.0% cheaper to 11.0% more expensive 4 per task and arm 15 Oct
An order from a mod to hand off a featureordered hand-off against the pack, Opus alone; worker costs corrected 16 Oct delegate 22% cheaper22% cheaper95% range 11 to 33% cheaper 4 per task (8 hand-offs in 8 runs across both tasks) 13 Oct
An order from a mod to hand off testsordered hand-off against the pack, Opus alone; worker costs corrected 16 Oct delegate 26% cheaper26% cheaper95% range 11 to 36% cheaper 4 per task 13 Oct
An order in the person's name to hand off: followedhand-offs, of runs with the order delegate 8 of 88 of 8 8 runs 13 Oct
The hand-off tool offered, nobody askinghand-offs, of runs delegate 0 of 240 of 24 24 runs 13 Oct
Every prompt ending with "Delegate where you see fit."hand-offs, of runs delegate 0 of 280 of 28 28 runs (14 fresh, 14 long) 13 Oct
Claude Code's own Agent tool, available in every runuses, of runs delegate 0 of 1740 of 174 174 runs 13 Oct
Measured again with 0.6.0's delegate, nobody askinghand-offs, of runs delegate 0 of 160 of 16 16 runs 15 Oct
The same tool framed as an ability, implement_spec: unused, and dearer to carrythe tool as implement_spec against the pack, Opus alone (feature) delegate 18% more expensive18% more expensive 4 per task (0 hand-offs in 8 runs) 13 Oct
Handing off a feature in a 233k-token sessiontold to hand off against Opus alone with the pack; worker costs corrected 16 Oct delegate 35% cheaper35% cheaper 2 per task and condition 13 Oct
Handing off tests in a 233k-token sessiontold to hand off against Opus alone with the pack; worker costs corrected 16 Oct delegate 23% cheaper23% cheaper 2 per task and condition 13 Oct
Handing off a rename in a long sessiontold to hand off against Opus alone with the pack; worker costs corrected 16 Oct delegate 43% more expensive43% more expensive 2 per task and condition 13 Oct
Handing off an explanation in a long sessiontold to hand off against Opus alone with the pack; worker costs corrected 16 Oct delegate 103% more expensive103% more expensive 2 per task and condition 13 Oct
Handing off a lookup in a long sessiontold to hand off against Opus alone with the pack; worker costs corrected 16 Oct delegate 150% more expensive150% more expensive 2 per task and condition 13 Oct
Handing off a trace in a long sessiontold to hand off against Opus alone with the pack; worker costs corrected 16 Oct delegate 30% more expensive30% more expensive 2 per task and condition 13 Oct
Handing off a type search in a long sessiontold to hand off against Opus alone with the pack; worker costs corrected 16 Oct delegate 261% more expensive261% more expensive 2 per task and condition 13 Oct
Ordered to hand off a lookup, with read-only hand-offs offordered hand-off against the pack alone; worker costs corrected 16 Oct delegate 17 to 40% more expensive17 to 40% more expensive 12 fresh runs 13 Oct
A feature built by Haiku at low efforttold to hand off against the pack, Opus alone; worker costs corrected 16 Oct delegate 34% cheaper34% cheaper 3 per task 13 Oct
A feature built by Sonnet at medium effort (the default)told to hand off against the pack, Opus alone; worker costs corrected 16 Oct delegate 26% cheaper26% cheaper 3 per task 13 Oct
Tests written by Sonnet at medium efforttold to hand off against the pack, Opus alone; worker costs corrected 16 Oct delegate 35% cheaper35% cheaper 3 per task 13 Oct
A small rename handed to Haiku at low efforttold to hand off against the pack, Opus alone; worker costs corrected 16 Oct delegate 37% more expensive37% more expensive 3 per task 13 Oct
Sonnet alone as the main model, on work this size and this well specifiedSonnet 5.5 alone against Opus 5.5 alone, plain Claude Code tried beside it 65% cheaper65% cheaper 14 runs 13 Oct
Six unused claude.ai connectors turned off: the biggest saving was not a modwithout connectors against with them (the quiet pack) tried beside it 34% cheaper34% cheaper 3 to 7 per task 13 Oct
route 0.1 on click's eight jobsthe pack with route against the pack; worker costs corrected 16 Oct route 14% cheaper14.2% cheaper95% range 10.1 to 17.9% cheaper 32 runs, 4 per task 15 Oct
route 0.1 with the pack, against plain Claude Codethe pack with route against plain Claude Code; worker costs corrected 16 Oct routethe pack 19% cheaper19.1% cheaper95% range 15.3 to 22.5% cheaper 32 runs, 4 per task 15 Oct
route 0.1 on the two tasks it routed (feature and tests)the pack with route against the pack; worker costs corrected 16 Oct routedelegate 34% cheaper33.8% cheaper95% range 27.4 to 39.4% cheaper 4 per task 15 Oct
route 0.1 on the six tasks it left alonethe pack with route against the pack route about even4.3% more expensive95% range from 0.1% cheaper to 9.0% more expensive 4 per task 15 Oct
Routing decisions on click: the order on feature and test prompts onlyorders given on feature/test prompts; wrong orders on the others route 8 of 8, 0 of 248 of 8, 0 of 24 32 runs 15 Oct
Verified hand-off and handing off before reading, against route as releasedroute 0.3 (verified, direct) against route 0.1; worker costs corrected 16 Oct routedelegate 28% cheaper28% cheaper95% range 10 to 42% cheaper 4 per task and condition 15 Oct
Claude's own requests with the verified, direct hand-offroute 0.3 (verified, direct) against route 0.1 routedelegate 55% fewer steps55% fewer steps95% range 36 to 68% fewer steps 4 per task and condition 15 Oct
A TypeScript feature routed to a Sonnet workerroute (Sonnet worker) against the pack alone, same batch; worker costs corrected 16 Oct routedelegate 51% cheaper51% cheaper 4 per task and condition 15 Oct
The same TypeScript feature: timeroute (Sonnet worker) against the pack alone, same batch routedelegate 42% less time42% less time 4 per task and condition 15 Oct
The same TypeScript feature: Claude's own requeststhe pack alone to route (Sonnet worker) routedelegate 11.0 to 2.011.0 to 2.0 4 per task and condition 15 Oct
Writing TypeScript tests, routed to a Sonnet workerroute (Sonnet worker) against the pack alone, same batch; worker costs corrected 16 Oct routedelegate 36% cheaper36% cheaper 4 per task and condition 15 Oct
Writing TypeScript tests, routed to a Haiku workerroute (Haiku worker) against the pack alone, same batch; worker costs corrected 16 Oct routedelegate 46% cheaper46% cheaper 4 per task and condition 15 Oct
The order to hand off before reading the code: followedhand-offs, of runs with the order route 24 of 2424 of 24 24 runs 15 Oct
Allowed to look around first, Claude kept the workruns where Claude kept the work, of runs with the order route 5 of 165 of 16 16 runs 15 Oct
Both classifiers on 25 prompts, two roundsright labels, of calls (each classifier) route 50 of 5050 of 50 50 calls each 15 Oct
A long TypeScript feature spec: ScaleDown under the threshold, Jev certainScaleDown's score against Jev's, on the same prompt (threshold 0.7) route 0.66 against 1.00.66 against 1.0 not stated 15 Oct
seek's order in a fresh sessionthe pack with seek's order against the pack seek 12% cheaper11.9% cheaper95% range 5.9 to 17.2% cheaper 20 per arm 16 Oct
seek's order in a fresh session, with what each run left behindthe pack with seek's order against the pack seek 18% cheaper18.0% cheaper95% range 11.5 to 23.9% cheaper 20 per arm 16 Oct
seek's order in a long session that had read click's docs and tests, not its sourcethe pack with seek's order against the pack seek 10% cheaper10.2% cheaper95% range 2.2 to 17.6% cheaper 24 per arm 16 Oct
The same, with what each run left behindthe pack with seek's order against the pack seek 13% cheaper13.0% cheaper95% range 6.8 to 19.0% cheaper 24 per arm 16 Oct
seek's order in a long session that had read all of the sourcethe pack with seek's order against the pack seek 16% cheaper15.9% cheaper95% range 8.6 to 22.5% cheaper 24 per arm (warm runs) 16 Oct
The same, with what each run left behindthe pack with seek's order against the pack seek 18% cheaper17.7% cheaper95% range 10.4 to 24.6% cheaper 24 per arm (warm runs) 16 Oct
seek's order on a TypeScript librarythe pack with seek's order against the pack seek 16% cheaper16.4% cheaper95% range 10.5 to 21.8% cheaper 24 per arm 16 Oct
The same, with what each run left behindthe pack with seek's order against the pack seek 23% cheaper23.1% cheaper95% range 17.1 to 28.6% cheaper 24 per arm 16 Oct
The TypeScript feature with seek's order: requeststhe pack with seek's order against the pack seek 8.3 against 11.38.3 against 11.3 3 per task and arm 16 Oct
What a TypeScript task left in the context, with and without seek's orderthe pack with seek's order against the pack seek 13.2k against 18.6k13.2k against 18.6k 24 per arm 16 Oct
seek's order with the person's prompt: followedruns that got the order seek 44 of 4444 of 44 44 runs 16 Oct
seek's map (where a search's hits sit) on TypeScriptthe pack with seek's map against the pack seek about even4.5% cheaper95% range from 9.9% cheaper to 1.2% more expensive 24 per arm 16 Oct
seek's map on top of the order, in the docs sessionorder and map against order alone seek 8% cheaper7.9% cheaper95% range 0.5 to 15.1% cheaper 24 per arm 16 Oct
The map on top of the order, with what it left behindorder and map against order alone seek about even5.5% cheaper95% range from 11.7% cheaper to 1.0% more expensive 24 per arm 16 Oct
seek 0.1: lines around a plain grep's hitsruns where it fired seek 1 of 361 of 36 36 runs 15 Oct
seek 0.2: lines around a search's hits, freshthe pack with seek 0.2 against the pack seek about even2.1% more expensive95% range from 1.4% cheaper to 5.5% more expensive not stated in the note; see the data file 15 Oct
seek 0.2 fresh, with what the windows left behindthe pack with seek 0.2 against the pack seek 6% more expensive6.3% more expensive95% range 1.4 to 11.2% more expensive not stated in the note; see the data file 15 Oct
seek 0.2 in a long sessionthe pack with seek 0.2 against the pack seek 5% cheaper4.5% cheaper95% range 1.6 to 7.4% cheaper not stated in the note; see the data file 15 Oct
seek 0.2 in a long session, with what the windows left behindthe pack with seek 0.2 against the pack seek about even2.2% cheaper95% range from 5.2% cheaper to 0.7% more expensive not stated in the note; see the data file 15 Oct
pathfinder: an order mid-search to hand the rest to a read workerthe pack with pathfinder against the pack tried beside it about even3.4% cheaper95% range from 11.3% cheaper to 5.1% more expensive 12 runs 16 Oct
pathfinder's order inside Claude's own search: followedhand-offs, of runs that got the order tried beside it 1 of 61 of 6 12 runs, the order came in 6 16 Oct
How many later requests re-read a token added to the conversationmedian before a compaction (interactive sessions; 22 headless) seek 143143 not stated not stated
What a token added to the conversation costs over its life, per million tokensagainst $8 to write it once; worked out from list prices and the median of 143 later reads seek $37$37 worked out, not measured 16 Oct
Away from an idle 400k-token session: let the cache expire, re-cache on returnworked out from list prices, not measured smart-compact $3.20$3.20 not measured not stated
One keep-alive for the same session: a cheap read resets the cache's clockworked out from list prices, not measured smart-compact $0.09$0.09 not measured not stated
Compacting the same session while its cache is still warmworked out from list prices, not measured smart-compact about $0.87about $0.87 not measured not stated
A permission hook that hangs on purpose, with and without the hook guardtime until the permission question appeared modpack 10 s10 s checked live 14 Oct
Trail-Code (a 10,000-star code graph) added to the pack, as installed, freshthe pack with Trail-Code against the pack tried beside it 13% more expensive12.8% more expensive95% range 7.5 to 18.3% more expensive 64 runs 15 Oct
Trail-Code added to the pack, as installed, long sessionthe pack with Trail-Code against the pack tried beside it 32% more expensive32.1% more expensive95% range 21.9 to 43.6% more expensive 28 runs 15 Oct
Trail-Code without its Stop hook, added to the pack, freshthe pack with Trail-Code (no Stop hook) against the pack tried beside it 4% more expensive4.0% more expensive95% range 0.1 to 8.0% more expensive 64 runs 15 Oct
Trail-Code on 30 real bug reportsthe pack with Trail-Code against the pack tried beside it about even10.8% more expensive95% range from no change to 25.5% more expensive 1 attempt per bug and condition 15 Oct
Trail-Code alone (without its Stop hook), first batchTrail-Code alone against plain Claude Code tried beside it about even4.6% cheaper95% range from 9.3% cheaper to 0.1% more expensive 4 per task and arm, 128 runs in the batch 16 Oct
The pack without seek, in the Trail-Code batchthe pack (without seek) against plain Claude Code the pack 6% cheaper5.6% cheaper95% range 0.3 to 10.8% cheaper 4 per task and arm, 128 runs in the batch 16 Oct
The pack and Trail-Code togetherthe pack with Trail-Code (no Stop hook) against plain Claude Code tried beside it about even2.8% cheaper95% range from 8.5% cheaper to 3.0% more expensive 4 per task and arm, 128 runs in the batch 16 Oct
Trail-Code on top of the packthe pack with Trail-Code (no Stop hook) against the pack tried beside it about even3.0% more expensive95% range from 2.7% cheaper to 9.1% more expensive 4 per task and arm, 128 runs in the batch 16 Oct
The pack as 0.6.0 ships, in the Trail-Code batchthe pack 0.6.0 against plain Claude Code the pack 15% cheaper15.1% cheaper95% range 9.7 to 19.8% cheaper 4 per task and arm, 96 runs in the batch 16 Oct
Trail-Code alone (without its Stop hook), second batchTrail-Code alone against plain Claude Code tried beside it about even1.1% more expensive95% range from 5.4% cheaper to 7.7% more expensive 4 per task and arm, 96 runs in the batch 16 Oct
Trail-Code alone against the pack as 0.6.0 shipsTrail-Code alone against the pack 0.6.0 tried beside it 19% more expensive19.0% more expensive95% range 13.1 to 25.1% more expensive 4 per task and arm, 96 runs in the batch 16 Oct
Trail-Code alone, both batches pooledTrail-Code alone against plain Claude Code tried beside it about even1.8% cheaper95% range from 5.9% cheaper to 2.7% more expensive 64 runs a side 16 Oct
Trail-Code's best case: its trail ask --source bundle in front of the promptplain Claude Code with the bundle against plain Claude Code tried beside it about even3.2% more expensive95% range from 7.3% cheaper to 15.0% more expensive 2 per task and arm, 8 runs 16 Oct
The pack 0.6.0 on PocketBase, Trail-Code's own 15 tasks, in totalthe pack 0.6.0 against plain Claude Code the pack 21% cheaper21.0% cheaper 1 per task and arm, as Trail-Code ran them 16 Oct
The pack 0.6.0 on PocketBase, the average taskthe pack 0.6.0 against plain Claude Code the pack 18% cheaper18.1% cheaper95% range 8.3 to 26.7% cheaper 1 per task and arm; the interval resamples the 15 tasks 16 Oct
Both borrowed ideas (scout's definition pointers and blast) added to the packthe pack with both ideas against the pack tried beside it about even2.5% cheaper95% range from 6.9% cheaper to 2.0% more expensive 4 per task and arm 15 Oct
Taking scout out, on the five tasks where it speaksthe ideas without scout against with it tried beside it about even3.8% more expensive95% range from 1.9% cheaper to 10.3% more expensive 8 per task 15 Oct
scout on 30 real bug reports, where it spoke on 20the pack with scout and blast against the pack tried beside it about even7.3% more expensive95% range from 6.7% cheaper to 20.7% more expensive 1 attempt per bug and condition 15 Oct
yaml-plus, a reference checker for deployment YAML, added to the packthe pack with yaml-plus against the pack tried beside it about even2.0% more expensive95% range from 1.3% cheaper to 5.3% more expensive 64 runs 15 Oct
A requirements gate: 'said done while incomplete', first batchwith the gate against without (paired) tried beside it 17 points lower17 points lower95% range 4 to 33 points lower not stated not stated
The same gate in a preregistered confirmation: not confirmedwith the gate against without (paired) tried beside it no clear difference4 points lower95% range from 12.5 points lower to no change not stated not stated
squeeze, a test-log squeezer, on a task with 18 failing teststimes it fired tried beside it 0 times in 4 runs0 times in 4 runs 4 runs 15 Oct
One compression of click's source for five questionsquestions answered right, 20k-token context (5 of 5 from the original) tried beside it 0 of 50 of 5 five runs per case not stated
A read hand-off checked place by place (PLACES), against Claude doing the lookup itselfread hand-off with PLACES against Claude alone; worker costs corrected 16 Oct delegate 5% more expensive5% more expensive not stated 15 Oct
Subagents get the pack's settings with their taskthe pack with cells' and seek's subagents option on against off cellsseek 18% cheaper17.6% cheaper95% range 14.0 to 21.0% cheaper 20 per arm, 4 per task 16 Oct
Subagents with the settings, on the features and the teststhe pack with cells' and seek's subagents option on against off cellsseek 22% cheaper21.8% cheaper95% range 17.4 to 26.0% cheaper 20 per arm, 4 per task 16 Oct
Subagents with the settings, on the explanationsthe pack with cells' and seek's subagents option on against off cellsseek 8% cheaper7.8% cheaper95% range 2.6 to 12.3% cheaper 20 per arm, 4 per task 16 Oct
A subagent's requests, without and with the settingsthe pack with cells' and seek's subagents option on against off cellsseek 10.0 to 6.910.0 to 6.9 20 per arm, 4 per task 16 Oct
A subagent's Bash calls that chain several commands, without and with the settingsthe pack with cells' and seek's subagents option on against off cellsseek 88% to 99%88% to 99% 20 per arm, 4 per task 16 Oct
The settings in the subagent's task: arrivedsubagents whose task held the settings, option on cellsseek 20 of 2020 of 20 20 per arm, 4 per task 16 Oct