The question behind the token bill
If you've watched an AI agent grind through a large file — scrolling past line 4,000 to make a two-line change — you've probably had the instinct that this is wasteful. It's harder to say exactly how wasteful, or whether the time spent breaking that file apart would ever pay for itself. "Cleaner code" is a reasonable argument, but it's not a number, and it's hard to put in front of anyone who's asking why you're refactoring instead of shipping.
A recent experiment, reported by Giles Edwards-Alexander (Thoughtworks) on Martin Fowler's site, puts an actual figure on it. It's a single case study on one codebase, not a benchmark you can generalise from — but the method is worth understanding, because you can run something similar on your own project.
What was measured
Edwards-Alexander built an application — around 150,000 lines, mostly Rust — entirely through AI agents (mainly Claude Code, with some Cursor), without reading or reviewing most of the generated code. Over time, the data access layer grew into a single Rust file of 17,155 lines: every database query repeating the same HTTP setup and JSON encoding, with little function or class extraction.
That gave a clean natural experiment. Because an AI agent doesn't "learn" the codebase between sessions the way a human engineer would, the same representative change could be requested fresh, against the same file, before and after each refactoring step — with no risk of the result being skewed by an agent remembering the previous version. The process, as reported:
1. Write one representative change as a single prompt. 2. Run it in a sub-agent against the current state of the code, recording token usage, and then discard the change (baseline). 3. Apply one step of a planned refactoring. 4. Run the same prompt again in a fresh sub-agent, record token usage, discard the change. 5. Repeat for each refactoring step, tracking input tokens, output tokens, time taken, and lines of code at every stage.
One methodological caveat the source is explicit about: Claude doesn't currently offer a reliable live token count, so the reported figures use character counts divided by four as an approximation (via tiktoken), rather than exact billed tokens. Treat the numbers as indicative, not precise.
What the numbers reportedly showed
Across fifteen refactoring steps, the data access layer went from one 17,155-line file to nineteen files, with the largest single file eventually 3,695 lines (a test file, itself a candidate for further splitting).
The two metrics that moved differently are the interesting part:
- Input tokens per change stayed roughly flat for the first several steps (around 150,000–170,000) while the file was still large, then fell as the largest single file shrank — dropping to around 27,360 by the final step, once the largest file was down to 3,695 lines.
- Output tokens per change stayed comparatively flat throughout (roughly 1,700–2,500), regardless of file size.
- Time per change did not fall smoothly, and did not end lower than where it started — it spiked to 1,353 seconds at two intermediate steps, and the final step took 454 seconds against a 342-second baseline.
The author's own reading, quoting the agent's own description, is that input tokens didn't decline gradually — they stayed flat and then fell sharply once the largest file crossed some threshold. That's a claim about this one dataset, not a general law: the sample is one file, one codebase, one class of representative change, one team's refactoring plan.
What this does and doesn't tell you
What it supports, cautiously: the input-token cost of a change is driven more by the size of the file the agent has to load and reason about than by the size of the codebase overall. Splitting a 17,000-line file into many smaller ones didn't reduce total lines of code much — it reduced the largest file an agent had to read in full when making a targeted change.
What it doesn't support: a claim that refactoring always pays for itself, that this ratio holds for other languages or change types, or that output tokens (and therefore the "thinking" cost) will also fall. In this case they didn't move much at all — the cost reduction was specifically on the input side, i.e. what the agent had to read, not what it had to produce.
It's also one experiment run by the person doing the refactoring, on a codebase they built and shaped the refactoring plan for. There's no independent replication here, and the token figures are an approximation rather than a billed total.
The practical consequence for how you work
If you're following something like PDAID's Build and Test & Correct phases, this gives a concrete reason — not just a tidiness argument — to keep files small as part of ordinary increments, rather than treating refactoring as a separate, hard-to-justify project:
- Treat "does this increment leave a smaller largest-file?" as a real question, not an afterthought, when you're deciding how to size the next increment. A large, undifferentiated file is a cost that compounds on every future change requested against it, not just an aesthetic problem.
- If you suspect a file is expensive to work against, you can test the theory cheaply. Run the same representative prompt against a throwaway branch before and after splitting the file, and compare what the agent reports it consumed. You don't need Edwards-Alexander's full fifteen-step plan to get a useful signal — a single before/after comparison on your worst file will tell you whether the same pattern shows up in your project.
- Don't expect the saving to show up in every metric. In this experiment, output tokens and execution time didn't fall as cleanly as input tokens did. If you run your own comparison and only input cost improves, that's consistent with what's reported here, not a sign the exercise failed.
- This is a reason to refactor deliberately, inside a controlled increment, not an excuse to let an agent restructure a file as a side effect of an unrelated change. The discipline described in how AI projects drift still applies: decide the refactoring is happening, define what must not change in behaviour, and verify it afterwards using something like the site's test record before treating the result as a new stable baseline.
The headline claim worth taking away isn't "refactoring always saves tokens" — it's narrower and more useful than that: in at least one measured case, the size of the single largest file an agent had to read was a better predictor of per-change token cost than the size of the codebase as a whole. That's a testable claim about your own project, not a guarantee.