The pilot proved the AI worked once.
Every AI deployment has a proud moment. The pilot ran, the outputs looked good, the stakeholders nodded, and someone wrote "validated" next to the use case. That moment is real, and it is worth far less than the sign-off it usually earns. A pilot proves that a particular version of a model, fed a particular slice of data, produced acceptable results on a particular day. Production holds none of those things still. The model changes underneath you, the data shifts, and the way people use the tool drifts away from what you tested. The system you validated quietly stops being the system that is running, and unless you are measuring, you will not know.
This is not a hypothetical. It is the default state of any AI system left unmonitored, and the past month made the sharpest version of it impossible to ignore.
The shift: the ground moves without you touching it
Consider what happened when Claude Sonnet 5 shipped on 30 June 2026 and quietly became the default in Claude Code and on Claude.ai. Any team relying on the previous default had its underlying model swapped, without changing a line of its own code, without a project, without a decision. If that team validated a workflow on the old model, its validation is now describing a system that no longer exists. The output might be better, worse or simply different. The only way to know is to have been measuring.
Model version change is the most visible form of drift, but it is not the only one, and the others are quieter.
Three kinds of drift degrade an AI system that passed its pilot. Data drift is when the real inputs move away from what you tested, as customer language, document types or edge cases the pilot never saw start arriving. Behaviour change is when the provider updates the model, retunes its safety behaviour, or changes a default, altering how it responds to prompts you had dialled in. Quality regression is when the outputs simply get measurably worse, from a bad prompt change, a degraded retrieval source, or an update that helped one task and hurt yours. None of these requires you to touch your own system. All of them can turn a validated deployment into a failing one, silently, between one week and the next.
What it actually means: validation is a state, not an event
The deeper point is that AI quality is not a property you establish once. It is a state you maintain. Traditional software, tested and shipped, keeps behaving the way it did unless someone changes the code. AI does not offer that stability, because the model is probabilistic, the provider is iterating, and the world feeding it is moving. Treating the pilot as the end of the quality question is a category error imported from an older kind of software.
That reframes the work. The launch-day test tells you the system can meet the bar. Keeping it above the bar is a separate, ongoing discipline, and it has a name: continuous evaluation. It is the difference between checking that a bridge was built to spec and monitoring whether it is still safe to cross. The first is necessary. The second is what keeps people from falling in.
The failure is quiet by design, which is what makes it dangerous. A model that has drifted does not throw an error. It keeps producing fluent, confident output that looks exactly like the output that passed the pilot, only now some of it is wrong in ways the pilot never checked for. There is no crash, no red light, no exception in a log. The system reports success while degrading, and the only place the degradation is visible is in the quality of the answers, which is precisely the thing nobody is measuring once the pilot is signed off. A drifted AI is the most reassuring kind of broken, because everything about it still looks fine.
A short scenario: the summary tool that quietly regressed
Picture a team at [ORGANISATION] that deployed an AI tool to summarise incoming case documents, validated it in a careful pilot, and moved on. For four months it was excellent. Then, over a fortnight, two things happened at once. The provider updated the default model, and the mix of incoming documents shifted as a new document type started arriving in volume. Nobody on the team changed anything. The summaries kept coming, still clean, still confident, and quietly started dropping a category of detail the pilot had confirmed the tool captured.
No one noticed for six weeks, because there was nothing to notice. The failure surfaced only when a downstream decision went wrong and someone traced it back to a summary that had silently omitted a material fact. Had the team kept a fixed evaluation set running against production, the drop would have shown up as a falling score within days of the model update, with an alert, an owner and a paper trail. Instead it showed up as an incident. The difference between those two outcomes is not the quality of the pilot. It is whether the measuring ever stopped.
Who should care, and how to build it
Anyone running AI in production past the demo stage needs this, and the build is more approachable than it sounds. You do not need a research team or a vendor platform to start; you need a habit and a small amount of engineering discipline. A continuous evaluation harness has five parts, and they compound, each one making the next more useful.
- Baseline with a fixed evaluation set. Build a representative set of real task examples with known good outputs, and keep it stable. This is your ruler. When anything changes, the model, the prompt, the data source, you re-run the eval set and see the effect as a number, not a hunch. Without a fixed ruler, every measurement is a new opinion.
- Log production traces. Capture real inputs and outputs from the live system, with enough context to score them. You cannot monitor what you do not record, and the production stream is where drift actually shows up first, in the inputs your pilot never imagined.
- Alert on the joint signal. The useful trigger is not one metric but two together: the inputs have drifted from your baseline distribution, and the quality score has dropped. Watching both catches the case that matters, where the world moved and the system stopped keeping up, and it filters out the noise of one moving without the other.
- Turn every failure into a regression test. When you find a bad output, add it to the eval set as a case that must pass. This is how a fixed problem stays fixed. Over time the harness accumulates the exact failure modes your system has hit, and it becomes progressively harder for an update to reintroduce an old one.
- Gate changes through the pipeline. Wire the eval set into your deployment process so a prompt change, a model swap or a config update has to clear the bar before it ships. The gate turns evaluation from a report you read into a control that acts, blocking a regression before your users meet it.
The domain angle: in regulated work, drift is a compliance event
For Australian regulated work, this stops being an engineering nicety and becomes an obligation. The governance frameworks already expect it. NIST's AI Risk Management Framework is built around measuring and managing AI risk on an ongoing basis, not certifying it once, and its measure and manage functions are about tracking system behaviour over time and responding when it changes. APRA's prudential standard CPS 230 treats a material change to a service, which a provider swapping your default model version plainly is, as a change event you have to identify and manage. The Privacy Act's accuracy expectations do not pause because a model drifted.
The consequence is concrete. If your AI supports a decision that affects a person or a regulated outcome, and it silently degrades, you have a compliance problem the moment it drifts, not the moment someone notices. And you cannot evidence that a system still meets the obligation it was deployed under if you stopped measuring it at the pilot. Continuous evaluation is how you turn "it passed once" into "we can show it still holds", which is the standard a regulator, a board or an auditor actually applies.
The hype check
Two cautions keep this honest. First, continuous evaluation is not a monitoring dashboard you buy and forget. The eval set has to reflect the real task, and it needs maintaining as the task evolves, or you end up precisely measuring the wrong thing. A stale ruler is worse than none, because it looks like assurance. Second, no harness removes the need for human judgement on the outputs that matter. It tells you the system has changed and where; deciding whether the change is acceptable, especially in a high-consequence use, still sits with a person. The harness narrows what you have to watch. It does not watch for you.
What to do this week
- Find one production AI system and ask when it was last measured. If the answer is "at the pilot", you have found your first case. A system running unmeasured since launch is running on faith.
- Build a small fixed eval set for it. Twenty to fifty real examples with known good answers is enough to start. Run it today to establish a baseline you can compare against later.
- Check whether you would even know if the model changed. Confirm how you would be told if your provider updated the default underneath you. If the answer is "we would not", that is the gap to close first.
- Put drift on the change register. For any AI supporting a regulated or high-consequence process, record model version change and quality drift as changes you monitor and manage, with an owner. That is the artefact that turns a silent failure into a governed one.
The pilot was never the finish line. It was the first measurement of a system that will keep moving whether you watch it or not. The teams that stay safe are not the ones that validated hardest at launch. They are the ones that never stopped measuring, because they understood that in AI, a passing grade is not a status you earn. It is a state you have to keep proving.
TheAICommand. Intelligence, At Your Command.
