How we know

What we tried and dropped

Eight mods ship. Far more ideas were built, measured and dropped. The reasons they failed taught us more than the features that worked, so here they are, grouped by what they taught.

Claude ignored it

A hint is easy to add and easy to ignore. Each of these spoke to Claude from the wrong place, and Claude carried on with what it was doing.

The grep hint

lsp-plus noticed a grep for a code symbol and attached a note: this LSP call lists exactly its uses. Correct, specific, one call away.

It fired 45 times across every benchmark run with the pack, and Claude acted on it 0 times0 of 45. Following it meant two more steps for an answer Claude believed it already had.

Off by default. Replaced by the answer itself, attached to the grep.

Four ways to make Opus delegate

A routing rule in the system prompt; the person's own "Delegate where you see fit"; the tool presented as an ability, implement_spec; telling Opus what each of its requests costs. Handing a feature to a cheaper model saves a third. If Opus did it by itself, everyone would save.

No hand-offs at all: none in 18 runs with the rule, none in 280 of 28 with the person’s own words, none in 8 as an ability, where the unused tool still cost up to 18% more18% more expensive to carry, and none in 4 with the cost spelled out. Only an order works.

Dropped. delegate gives an order when you approve a plan; route gives one for feature and test prompts.

squeeze: the log squeezer

Replace a long test log with its failures, each with its file, line and message, and keep the full log on disk. Locally excellent: all 18 failures kept, nearly all of the rest of the log gone, in about a millisecond.

On a task with 18 failing tests it fired 0 times in 4 runs0 times in 4 runs: Opus piped every test run through tail or grep itself. Haiku did the same.

Dropped. A tool for a habit neither model has.

An order in the middle of a search

pathfinder waited for the third consecutive search in a large context and then told Claude to hand the rest of the search to a cheaper read worker. Long chains of looking around, mostly in contexts over 400k tokens, held a tenth to a sixth of all the cost.

The order came in 6 of 12 runs, and Claude handed off once. The cost: no clear difference3.4% cheaper, 95% range from 11.3% cheaper to 5.1% more expensive.

In the graveyard. The same kind of order beside the person's prompt is followed every time; inside Claude's own search it is not.

It cost more than it saved

A step saved by adding text is paid for again by every later step. Each of these saved something and still came out dearer.

Handing off the small stuff

Let delegate take read-only work too: lookups, traces. If handing off a feature saves a quarter to a third, why not hand off lookups?

Told to hand off a lookup, Opus turned it into an edit task, and the runs cost 17 to 40% more17 to 40% more expensive, leaving stray files in the repository. A hand-off takes two to five steps of its own; a lookup takes one.

Reverted.

A 10,000-star code graph (Trail-Code)

Not ours: a graph of the codebase Claude can query, with hooks and tools around it, measured the way a person installs it. Its own benchmark reports it about a third cheaper, and someone had cut their usage sharply with a tool like it.

Installed into Claude Code the way its README says, alone it came out about even1.8% cheaper, 95% range from 5.9% cheaper to 2.7% more expensive against plain Claude Code over two batches. Added to the pack it came out 13% more expensive12.8% more expensive, 95% range 7.5 to 18.3% more expensive in a fresh session and 32% more expensive32.1% more expensive, 95% range 21.9 to 43.6% more expensive in a long one; alone, against the pack as 0.6.0 ships, 19% more expensive19.0% more expensive, 95% range 13.1 to 25.1% more expensive. Its own best case, the bundle its benchmark puts in front of every task, rebuilt in Claude Code: about even3.2% more expensive, 95% range from 7.3% cheaper to 15.0% more expensive. Claude never called its MCP tools and used its command line as a second grep.

Not adopted. Two of its ideas, scout and blast, were borrowed and measured on their own.

Windows after a search

The first seek attached the lines around a search's hits to the search's result, so Claude would not need a request of its own to read them. In real sessions the most common predictable step after a search was opening a file it had just listed.

seek 0.1 fired in 1 run of 361 of 36: Opus’s searches sit behind a cd, in chains and in pipes. seek 0.2 fired, saved steps, and still cost about the same2.1% more expensive, 95% range from 1.4% cheaper to 5.5% more expensive; priced with what the windows left in the conversation, 6% more6.3% more expensive, 95% range 1.4 to 11.2% more expensive.

Retired. Replaced by an order at the prompt. Count what stays: a step saved by adding text is paid for again by every later request.

It never left the noise

When the range of a measurement still includes no change at all, we cannot say a mod helps, so it does not ship. Most ideas end here.

lastenheft

A gate that turned an order into checkable requirements and would not let Claude stop until each was proven. Agents that say "done" on half-finished work are a real problem; a gate against it looked like obvious value.

A first batch looked promising: the share of runs that said done while unfinished fell by 17 points17 points lower, 95% range 4 to 33 points lower. A confirmation, planned in advance and run on fresh repeats, gave 4 points4 points lower, 95% range from 12.5 points lower to no change: not confirmed. And in 38 of 43 unfinished runs, every point on the checklist showed passed.

Kept in the lab, switched off. A checklist that turns green on incomplete work reassures falsely.

Moving text before the cache boundary

cells' guidance is the same text in every session, so put it in a system-prompt section shared across sessions. Written to the cache once, read ever after.

No saving1.0% more expensive, 95% range from 5% cheaper to 8% more expensive: the first request wrote no less.

Reverted.

scout: definition pointers

When a prompt spells out a code name, attach where it is defined, from one git grep per name. It skips Claude's first search, and it can only be wrong if the code changed in the last second.

A small saving on click that never left the noise, no clear difference2.5% cheaper, 95% range from 6.9% cheaper to 2.0% more expensive with blast beside it, and on 30 real bug reports, where it spoke on 20, no clear difference either7.3% more expensive, 95% range from 6.7% cheaper to 20.7% more expensive.

Stays out of the pack.

blast: callers after a signature change

After an edit changes a function's parameters, list the places that call it, from the language server. Trail-Code prints a blast radius after edits; callers break when parameters change.

On a task built for it, taking it out made no clear difference3.7% more expensive, 95% range from 2.0% cheaper to 11.0% more expensive. Before editing a function, Claude greps for it, and lsp-plus already answers that grep with the same callers.

In lsp-plus, off by default.

yaml-plus

One language server for every .yaml file that checks the names connecting Kubernetes, Compose, GitHub Actions and Ansible files. No language server resolves the names that actually connect a deployment.

No clear difference2.0% more expensive, 95% range from 1.3% cheaper to 5.3% more expensive on 54 files, where models read everything; on 954 files Opus wrote its own 50-line reference checker in one step and found all ten planted errors anyway.

Kept out of the pack: nothing here says it helps.

seek's map, left off

A code search's result names the definitions that hold its hits, with their lines, in no more tokens than a saved request is worth. The window idea at its leanest: a few lines, only where a request is dear.

Fresh on click it changed nothing; on TypeScript no clear difference4.5% cheaper, 95% range from 9.9% cheaper to 1.2% more expensive; on top of the order in a long session, counting what it left behind, no clear difference either5.5% cheaper, 95% range from 11.7% cheaper to 1.0% more expensive.

It ships switched off, as the option map.

Wrong about the important part

These worked on the easy part and failed on the part that mattered.

The projection that was wrong

An early analysis projected that hand-offs save more the fuller the session: about half of a feature’s cost at 200k tokens. Each step re-reads more in a full session, so removing steps should save more.

Measured in a 233k-token session, the feature saved 35%35% cheaper, about what it saved fresh, and a small question handed off cost up to 3.6 times as much. A session that knows the code needs far fewer steps.

Replaced by the measurement. Rule of thumb: hand off features and tests only.

Compression that forgets

A small, cheap model keeps only the passages of Claude's context that look relevant to a question. Every token in the context is paid for again on every step; the price of the small model is almost nothing.

4 of 5 answers right when each question had its own compression, 0 to 1 of 50 of 5 when one compression had to serve all five; one answer was confidently wrong. In a coding session the next question is never known in advance.

Not used.

Summaries of logs and worker reports

Let a small model summarize a test log or a worker's report before Claude reads it. Shorter text for every later step to re-read.

It named the failing tests but never kept a file and line; on an 18-failure log, four of five summaries were just the counts. A report that falsely said "the new tests pass" was repeated 3 of 3 times.

Not used. Extracting with a schema was right, but a local parser is as right in a millisecond.

It never reached the bench

One failed its own calibration; one was never measured at all. Neither ships.

Keyword pointers

BM25 keyword ranking over whole files, to point Claude at the right file before its first search. The strongest method in an independent evaluation of code-graph tools, with no index to keep fresh.

Calibrated on 13 prompts before a single benchmark run: it pointed "commit and push this" at an example program, then went quiet on 11 of 13 and was wrong on the other 2.

Dropped. Its deterministic half, where the name you typed is defined, became scout, measured below.

turn-savers

Lighter thinking on routine steps; the code at a failure attached to the failing run. Both sound like obvious savings.

Built, never measured.

Does not ship. A mod without a measurement is a guess with a version number.

What it taught us

The first good number is usually too good. Write down what would confirm it, then run it again.
A check that passes is not a check that checks. Agents’ own tests share their blind spots.
Models have habits. They grep, they tail logs, they keep work they can do. A feature that fights a habit loses; one that answers inside it wins.
Count what stays. A step saved by adding text is paid for again by every later step.
Removing is a result. Every idea here made the pack cheaper or simpler by staying out.

Lab note 5 tells each of these stories in full.