CSV Assumes The AI System Sits Still — It Doesn't.

By Sachin Bhandari, TrustBridge Compliance

Every rule in the computer system validation playbook was written for software that holds its shape. AI doesn't, and that's why validating it has to become an act of governance.

I once opened a validation file for an AI system that had passed every stage cleanly. The validation master plan was signed, the qualification package was complete, the signatures were current. Then I checked the model behind those signatures and found it had been retrained twice since the plan was approved. No change control, no error in any log, no downtime. The paperwork described one system and the site was running another, and nobody had done anything wrong by the rules they were taught.

That's the moment the problem becomes concrete. Computer system validation (CSV) is a discipline I've practiced for a working lifetime, and it's very good at what it was built for. But it was built for software that stays where you put it, and an AI model doesn't. Validating AI is a different game on the same field, and the sooner a team sees which assumptions have stopped holding, the sooner it can build something that works.

What CSV Never Had To Say Out Loud

CSV grew up around software that was deterministic and stable, and it inherited four assumptions it rarely states. You specify the system, you test it against the specification, you freeze it, and you control every change through a documented event. Each of those steps rests on the same unspoken belief: the system will sit still. AI breaks that belief in four specific places.

Figure 1: The four assumptions computer system validation makes, how a learning model breaks each, and what governance adds on top.

Same Input, Same Output

The first assumption lives in every test script ever written: an expected-result column. You give the system an input, you know the answer it should return, and the actual result either matches or it doesn't.

A probabilistic model has no single expected result. Its output is a distribution, a confidence score, a most-likely class out of several, and two correctly functioning models given the same input can legitimately return different answers. The expected-result column collapses. Teams that paper over this by writing the expected result after watching the model perform on the test set are recording what the model does rather than testing it. The honest replacement is to make the expected result a metric against a predefined acceptance criterion, sensitivity, specificity, calibration error, agreed before the test runs, supplemented by structured exploratory testing for the boundary cases a metric won't reach.

The System Stays Put

The second assumption is that once validated, the system holds its state until a human decides to change it, at which point change control catches the change and revalidation follows.

An AI model can change with no human decision at all. It retrains on production data, or its vendor updates the hosted model, and inference behavior shifts without a single change request at the customer site. Both pass through the audit trail invisibly unless the configuration baseline is capturing the model artifact, its version, and its dependencies. The CSV instinct is to forbid the change, and for a critical GMP process that instinct is right, because draft Annex 22 doesn't permit dynamic models there. For lower-risk use, forbidding change throws away the point of the model. The principle that survives is that no change to the validated state happens without a governance step. What changes is the mechanism: retraining becomes that governance step, with its own trigger, its data-quality gate, its delta check, and its approval.

Behavior Depends On The Configuration You Wrote Down

The third assumption is that behavior is a function of configuration, and your configuration baseline lists everything that matters: the version, the patch level, the parameters, the roles, the integrations. It doesn't list training data, because traditionally the system wasn't trained.

For a model, the training data is part of the configuration. A model trained on a different data set is a different model, and a model trained on a contaminated data set is a contaminated one, whatever the version number says. This is where most quality systems are not yet structured. Training data is often pulled from a validated source system through an extraction nobody validated, labeled after the fact by a handful of experts with no inter-rater evidence, its class balance never checked. None of that was in scope for the original Annex 11. All of it is in scope now. ALCOA+ applies to the training set with the same force it applies to a batch record.

Failure Announces Itself

The fourth assumption is the most dangerous, because it's the one operations relies on without noticing. A system that fails produces an error, a downtime, or a complaint. The audit trail catches it, a deviation opens, and the CAPA loop closes it.

A drifting model produces none of that. It doesn't produce an error, it doesn't go down, and the service log shows a healthy system. Outputs keep flowing at the expected rate, with the expected confidence, into the expected workflows, and the reviewer signs the same form they signed yesterday. Underneath, the relationship between input and correct answer has moved, because the world changed, or the input mix changed, or the prevalence of each outcome changed, and none of those announce themselves. The only signal is monitoring data: performance measured against a labeled sample of real output, drift statistics on the inputs, override rates and their direction. Failure is silent until that monitoring exposes it.

A Validation That Passed, And A System That Did Not

Put the four together in one system and you get a failure mode I've seen more than once. A team validates a model that triages incoming complaints into critical, major, and minor, using a traditional approach. The operational qualification scripts carry an expected result for each test case, matched to the most likely class. The performance qualification runs for two weeks against rehearsed scenarios. The model passes, and the system goes live.

A few months later someone compares the model's calls against the manual decisions over the same period and finds them disagreeing on roughly one in seven. About a quarter of those are fair judgment calls. The rest fall into two patterns: the model under-classifies a product family that was thin in its training data and over-classifies complaints from one site whose records are short and noisy. The validation had passed because the model did exactly what a deterministic system does under a deterministic test. The trouble was that the system was never deterministic, and the framework that validated it needed to be the AI framework, with subgroup acceptance criteria stratified by product family and site at qualification and override-rate analysis with direction and reason codes once it was live. Neither is exotic. Both sat outside the frame the team brought.

Where More Documentation Stops Helping

Put the four together and the shape of the problem is clear. You can't out-document a system that moves after you've documented it. A thicker qualification binder proves the state of the model on the Tuesday it was signed and says nothing about the Friday it quietly drifted. The lever most teams reach for, more testing up front, pushes on the part of the life cycle where a model is most stable and leaves the volatile part untouched.

The center of gravity has to move. Validation, as we've practiced it, is a proof you complete at a point in time. Governing an AI system is a state you hold over time. The distinction I keep coming back to is that governance decides whether the system may exist at all, and validation earns and then keeps its validated state. For deterministic software, those two things collapsed into one signed package. For a model, they come apart, and the governance half is the half that keeps the system safe after go-live.

What Governance Actually Adds

None of this is abstract, and most of it reuses machinery a mature quality organization already owns. Moving toward governance means a handful of concrete additions on top of the validation you already do.

Risk tiering by consequence comes first, because it decides how much of everything else you need. A model whose error could reach a patient earns depth and a senior signer; a model drafting a first pass of a document a human fully checks does not. In practice that becomes a depth schedule: a low-risk model can lean on vendor evidence and light checks, while a high-risk one earns full IQ, OQ, and PQ with exploratory testing for the boundaries. Intended use gets sharper, naming the inputs the model was built for and the cases it must not touch, so you enforce the boundary rather than assume it. Monitoring becomes the living successor to performance qualification, running from the day of go-live rather than ending the validation. Retraining gets its own controlled path, because the change is coming whether or not your change control system has a box for it. A named human stays accountable for the decision the model shaped, with enough information and enough authority to actually say no. And above all of it sits an operating model, a governance forum that owns the tier decisions, the monitoring signals, and the retraining approvals across the portfolio. The FDA's Computer Software Assurance guidance, finalized in 2025, already pushes effort toward the highest-consequence decisions, and draft Annex 22 sets out where AI in GMP may and may not sit. Both point the same way.

There's more to each of these than a section can hold, which is roughly how I ended up writing a book on it. The short version is the one that matters here: the work moves from proving a state once to holding it steady while the model keeps moving.

What Still Carries Over

It would be a mistake to read any of this as CSV being finished. It isn't. The system around the model, the database, the interface, the integrations, still gets validated the way it always did. Traceability still holds: every decision has to be able to be reconstructed from the records. Intended-use discipline, qualified environments, named approvers, audit-trail thinking, all of it carries over, extended rather than replaced. The right mental model is CSV with a governance layer on top. It helps to remember who actually shows up to inspect: a CSV-trained inspector who has read Annex 22, not a data scientist. The structure they expect is still recognizable. What they now also expect is the monitoring evidence that shows the model is still inside the envelope you validated it in.

Where To Start

Here are a few moves to put this into practice today:

• Tier every AI use by consequence first, and settle on how much rigor it earns and who signs, before you write a single test.
• Replace the expected-result column with metric-based acceptance criteria agreed up front, and add exploratory testing for the boundary cases and the groups that are thin in your data.
• Put training data on the configuration baseline: the data set version, its lineage, and its label quality, with ALCOA+ applied to it.
• Stand up monitoring before go-live, not after, covering performance against a labeled sample, input drift, and override rate and direction, with named owners and a review cadence.
• Give retraining a governed path: a trigger, a data quality gate, a delta check against the original acceptance criteria, and a documented approval.
• Name a human who owns each decision the model shapes, with the information and the authority to overrule it, and make sure the governance forum actually meets.

So, the ballgame really is different, but it's played on ground you already know. The teams that come through this well are the ones who treat the validated state as something they govern, watch, and defend for as long as the model is live, rather than a finish line they cross once. The binder that proves the model was right on the day you signed it was never the hard part. Keeping it right, and being able to show you did, is the game now.

About The Author:

Sachin Bhandari is the founder of TrustBridge Compliance, an advisory practice in digital quality and validation. He has spent 25 years in pharma across CSV and CSA, data integrity, and enterprise quality systems, including an eQMS deployment for around 15,000 users and paperless validation across 40-plus sites. He has taken AI-supported quality systems through health authority inspection in live operation. He can be reached by email at sachin.bhandari@trustbridge-compliance.com.