Three Claude Code add-ons made large efficiency claims. In a JetBrains series of paired A/B tests, two produced much smaller savings than advertised and one was more expensive in the tested low-effort setup. The useful lesson is not which add-on to install. It is how to test a claimed optimisation against the work your coding agent actually does.
What the three tests found
Ponytail is designed to make an agent write less code. Its published claims included 54% less code, 22% fewer tokens, 20% lower cost and 27% less time. JetBrains tested it across 80 paired SkillsBench tasks using Claude Code 2.1.201 and Claude Sonnet 5 at medium reasoning effort. The measured result was smaller: about 15% less code, 10.3% lower cost and 11% less time. The cost result was statistically strong in that test; the code reduction was less certain, and no quality difference was detected.
The setup also exposed a practical detail that matters when you test an agent skill: installation is not the same as activation. In ten sessions where Ponytail was merely available as a skill, Claude Code did not invoke it. The benchmark therefore tested the ruleset when it was actually injected. If you do not confirm that the treatment happened, you can end up measuring a feature that was never used.
rtk takes a different approach. It intercepts eligible shell commands and compresses their output before the agent sees it. Its README advertised token reductions of 60-90%. JetBrains' low-effort run instead measured a 7.6% increase in cost across 80 clean pairs, with no quality improvement; at high reasoning effort there was no cost difference.
The reason is useful beyond rtk itself. Before running the paid benchmark, JetBrains replayed existing agent transcripts and found that much of the information the model consumed never passed through the shell hook at all. File reads and searches bypassed it, and many shell commands were not covered. The add-on could compress its own slice of the workflow while having very little room to change the total.
Caveman promised a 65% reduction in output tokens by making the agent's prose much terser while leaving code and tool output alone. With the skill forced on, the full benchmark measured an 8.5% reduction in output tokens across 82 clean pairs, with no detectable quality degradation. JetBrains described the associated cost saving as roughly 10% in expectation, but fragile enough for a single long-context outlier to reverse the raw totals.
The Caveman tests also show why one quick run is weak evidence. An initial ten-task run appeared to show a 29.5% token saving. Repeating the same ten tasks reduced that result sharply, and the larger run settled at 8.5%. The first number was eye-catching; it was not stable.
The common mistake: measuring the part instead of the workflow
All three tools act on a narrower part of an agent's work than their headline percentage suggests. That is the pattern worth keeping.
A compression tool may reduce the text it touches without reducing the model's total context very much. A terseness instruction may cut narration while leaving code and tool calls unchanged. A minimal-code rule may help most on tasks where the baseline agent has room to over-build. The local mechanism can work while the end-to-end saving is much smaller.
That means the useful question is not simply, does this optimisation work? It is: does it materially improve the complete workflow I actually run, without making the result worse?
A practical test before you adopt an optimisation
- Write down the claim you care about. Tokens, cost, elapsed time and lines of code are different outcomes. Decide which one would actually improve your workflow.
- Choose representative tasks. Do not build the test around the one kind of task the optimisation is designed to win.
- Run paired attempts. Give the baseline and the changed workflow the same tasks, model, settings and budget so that the comparison is about the change rather than a different setup.
- Confirm the optimisation actually ran. Check hooks, injected instructions or logs rather than assuming installation means activation.
- Measure quality as well as efficiency. Saving tokens is not useful if the agent needs extra turns, reruns commands or produces worse software.
- Repeat before believing the first result. Coding-agent runs vary. A striking result from one small run can disappear when the same comparison is repeated.
- Judge the end-to-end effect. Measure the bill, elapsed work and accepted result, not only the narrow stream of tokens or code the add-on can directly alter.
What these tests do not prove
These are three tests from one JetBrains series, not three independent replications. They use particular versions of Claude Code, Claude Sonnet 5, SkillsBench tasks and specific activation methods. They do not establish a permanent result for every repository, model or future version of the tools.
They also do not show that agent optimisations are pointless. Ponytail produced a statistically strong cost reduction in its test. Caveman reduced output tokens when forced on. rtk's tested low-effort setup went the other way. The point is that none of the headline percentages can substitute for measuring the workflow you actually care about.
This is the same distinction behind "Production Ready" Is Not Evidence: a claim describes an expectation; a test tells you what happened. For coding-agent add-ons, the safest default is to make the optimisation earn its place with evidence before you build your workflow around it.