The problem with old code and a helpful model
Ask an AI assistant to help you get started with a codebase from 2005 and it will oblige. It will read the README, infer a modern project layout, suggest a clean build file, and hand you a tidy "Hello World". It sounds like exactly what you asked for.
The trouble, as developer Nik Malykhin describes in a detailed account published on Martin Fowler's site, is that in this 20-year-old system the README was not a reliable guide to the code as it actually stood. A plausible AI answer could therefore still be wrong in ways that only surfaced once he started relying on it.
Malykhin was restoring a 20-year-old Java codebase built with Ant on Java 1.5, one that no longer built on a modern machine. His first attempt was what he calls the "Tourist Prompt": he asked the AI to act as a senior developer, read the repo, and give him a summary and a starter example. It produced a modern build.gradle, assumed a standard Maven folder layout, and demonstrated a clean class that avoided the messiest parts of the code entirely.
Every part of it was plausible. Almost none of it matched the actual codebase:
- it specified
commons-pool2(a modern, API-incompatible library) when the code actually used the oldercommons-pool— a difference that would have caused the build to fail with class-not-found errors far removed from the real cause; - it assumed a standard
src/main/javalayout when the project actually used a non-standard Ant structure; - its example featured a pooled implementation class while quietly omitting the core one, which was not thread-safe, swallowed exceptions, and was tested only by integration tests that needed a live database — not the unit tests they were labelled as.
Malykhin's description for this pattern is useful: the AI defaults to optimism. Asked "how do I run this?", it assumes the answer exists and produces one. On a system this old, that assumption is often false, and following it would mean debugging modern-looking code that was never a true description of what was there.
Why this is a distinct problem, not just "AI makes mistakes"
The site's other pieces on testing AI-generated code and debugging without making things worse assume you can run the software you're checking. Legacy modernisation can start before that's true. In Malykhin's case, the code no longer built reliably and the README could mislead. That leaves the AI with less trustworthy ground truth to work from and, as Malykhin found, it can fill those gaps with plausible-sounding answers rather than accurate ones. That's a different failure mode from a wrong fix to a bug you can already reproduce.
Martin Fowler, summarising a Thoughtworks report from an industry retreat, notes that legacy modernisation is being flagged as one of the clearer near-term areas where AI adds value. Our interpretation is that this makes verification especially important: the model's account of an old system should be checked against the code and its actual behaviour rather than trusted on first read.
What changed the outcome: treat the code as evidence, not a source of answers
Once the tourist approach failed, Malykhin reset the exercise. Instead of asking the AI what the system did, he asked it to conduct a "forensic audit": explicitly told not to summarise the README (which he treats as an unreliable narrator in legacy projects), and assigned the persona of a sceptical reviewer rather than a helpful guide. The audit surfaced findings the friendly summary had hidden or missed — including that some of the project's existing "tests" didn't actually test the behaviour they claimed to.
This maps onto ground that PDAID already covers, applied to a harder starting condition. The Context & Constraints phase exists precisely so that what the AI is told about the existing system is accurate rather than assumed. On a legacy system, establishing that context isn't a formality — it's most of the work, and it has to come from the code and its actual behaviour, not from asking the model to describe it.
From there, Malykhin's process followed a recognisable shape:
1. Get the system running somewhere stable before changing anything. He used a fixed Docker environment to pin down exactly what "working" meant on this codebase, rather than trying to modernise the environment and the code at the same time. 2. Find out what the existing tests actually verify. Some of what looked like a safety net — "unit tests" that in fact required a live database, and results that didn't reflect real behaviour — had to be identified and treated as unreliable before they could be trusted for anything. 3. Refactor in small steps once a genuine baseline exists, testing after each one. This is the same discipline the site describes in always having something that works to go back to — but on a legacy system, establishing that first known-good point is itself a significant piece of work, not a given.
A separate account Fowler cites, of a developer restructuring an old Laravel and React application over several months, follows a similar order of operations: characterisation tests and static analysis first, then a shift from reviewing every AI-suggested change line by line ("in-the-loop") to supervising a harness that catches problems automatically ("on-the-loop"). The trust in the AI's output increased only after the checks around it existed — not before.
The practical consequence
If you're modernising an old or poorly-documented codebase with AI, the risk isn't that the AI will refuse to help. It's that it will help fluently, in a direction that isn't grounded in what the code actually does. Two habits reduce that risk:
- Ask what the code does before asking how to change it, and verify the answer against the code and its actual runtime behaviour — not against the README, comments, or the AI's own summary of them. Treat an AI's description of an unfamiliar legacy system as a hypothesis to check, not a fact to build on.
- Don't let modernisation begin until you have a stable, reproducible way to run the existing system and a truthful account of what its tests do and don't cover. Everything downstream — refactoring, dependency upgrades, rewriting — depends on that baseline being real rather than assumed.
It's also worth applying this scepticism to reports of AI-driven results generally. OpenAI, for instance, has published an account stating that Asana used its Codex tool to replace an outdated testing system in two weeks, work the company had estimated would otherwise take five years. That figure is reported by OpenAI in its own account of Asana, and we have not independently verified it. We hold only a one-line summary of that case study, not the study itself, so what it says about how the replacement was validated is simply unknown from here. It may well be accurate - but the same instinct that catches a hallucinated dependency in a legacy build should also be applied to a vendor's account of a striking result: ask what evidence supports the claim before treating it as a fact you can plan around.
For genuinely high-risk or safety-relevant legacy systems, the point at which technical judgement and specialist review become necessary — covered in when you need a developer to review AI-generated software — arrives earlier than it does for new code, precisely because there's less to check the AI's answers against and more scope for a plausible-sounding one to be wrong.