Lab notes
Lab notes
What we tried while building this modpack, how we measured it, and why most of it did not ship.
The modpack is eight mods for Claude Code. Behind them is a lab where every idea has to show, in graded runs, that it saves something without making the work worse. Most ideas don't make it: a nudge Claude never followed, a gate that reassured falsely, a 10,000-star tool that, installed, saved nothing we could measure. These notes are about those decisions. The data behind every number is in benchmarks/, with the harness to run it again.
Who wrote this. Claude (Opus 5.5) wrote these notes from this repository's results files and commit history; HarzerHeribert reviewed them. We build a competing pack, so read what we say about other tools with that in mind: every claim about them links to their own words, and every number of ours to the runs behind it. If we got something wrong about your project, open an issue and we will correct it here.
The notes
- 1The cheapest step is the one Claude never takesAsk most people what makes a Claude Code session expensive and they will say tokens: big files, long answers, a chatty model. That is true, but it points at the…15 Oct 2026
4 min read - 2Notes nobody readsThe most natural way to improve an AI agent with a plugin is to give it advice. Notice it doing something slow, tell it the faster way. We built two mods on tha…15 Oct 2026
4 min read - 3A 10,000-star code graph on our benchA friend asked a fair question: why build our own code tools when projects like Trail-Code (about 10,000 GitHub stars) put a whole codebase into a graph that Cl…15 Oct 2026
12 min read - 4Borrowing the good ideasThe code graph in the last note saved nothing we could measure on our bench, alone or beside the pack. That does not make their ideas wrong. Two of them are goo…15 Oct 2026
4 min read - 5The graveyardEight mods ship in this pack. Many more ideas were built, measured and dropped. A project's README usually shows the survivors; this note shows the rest, becaus…15 Oct 2026
7 min read - 6One server per extensionlsp-plus answers Claude's code searches from a language server. That only works where a server answers. The obvious next question, asked by someone who works wi…15 Oct 2026
6 min read - 7Small models as plumbingIf every token in Claude's context is paid for again on every step, an obvious trick is to shrink what enters it. A long test log, a big diff, a sprawling grep …15 Oct 2026
6 min read - 8Our plain runs were not plainEvery comparison in these notes rests on one thing: knowing exactly what loaded in each run. On 15 October we found out that for a day and a half, in one condit…15 Oct 2026
4 min read - 9Beyond clickEvery number in the first eight notes comes mostly from one library: click, 79 files of Python, tasks we wrote ourselves. A fair reader's first objection is the…15 Oct 2026
4 min read - 10Two requestsA hand-off to a cheaper worker saves money because the worker does the building. But the main model still pays for every request it makes around the hand-off, a…15 Oct 2026
5 min read - 11What looking around leaves behindBefore Claude changes code, it looks around: a search, a read, another search. Each step is a model request, and each request re-reads the whole conversation (n…16 Oct 2026
9 min read
How we measure
The same setup runs through every note unless it says otherwise.
- The work. Seven tasks on click 8.5.0 (79 files of Python), from a one-line lookup to a feature, later eight with a signature change, plus four tasks on a generated 54-file deployment repository (Kubernetes, Compose, GitHub Actions, Ansible). Every task is graded by a fixed rule: hidden tests for code, a parser for YAML, a term list for explanations. A run that saves money and gets the answer wrong counts as a failure, not a saving.
- The runs. Headless Claude Code (
claude -p), Claude Opus 5.5, one fresh copy of the repository per run. Each condition names exactly which mods load, so nobody's personal settings decide anything. A long session forks a prepared session that has already read all of click's source,src/click(about 220k tokens of context), which is where real work spends most of its time. - The cost. Read request by request from the transcripts and priced at list prices: Opus 5.5 at $4 per million input tokens, $20 output, $0.20 for a cache read, $8 for a cache write (Claude Code writes with the one-hour lifetime; a five-minute write is $5). On a subscription these are API-equivalent estimates, not a bill.
- The uncertainty. Two to eight runs per task and condition. Differences are reported for the average task with 95% bootstrap intervals, resampling runs within each task (Efron & Tibshirani 1993). An interval that crosses zero means not shown, and we say so. Before a confirmation run we wrote down what would count as confirmed (an example).
- The limits. Mostly one small Python repository and tasks every condition solves (note 9 repeats the main comparisons on TypeScript and on real bug reports): these benchmarks can show cost and step differences at equal quality. They cannot show where a weaker setup fails on harder work. Each note says what its data can and cannot carry.
References
- Efron, B., & Tibshirani, R. J. (1993). An Introduction to the Bootstrap. Chapman & Hall.
- Robertson, S., & Zaragoza, H. (2009). The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval, 3(4), 333–389.
- Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. PNAS, 115(11), 2600–2606.