MLOps at Enterprise Scale: Why Your Models Decay and What To Do About It

A deployed model is not a finished artefact. It is a claim about a relationship between inputs and outcomes, and that relationship has a half-life set by how fast the world it describes changes. MLOps is the engineering practice that makes the decay measurable, the correction routine, and the whole thing auditable — which is now a regulatory requirement as well as an operational one.

Reproducibility is the prerequisite for everything else

You cannot debug, audit, retrain or roll back a model you cannot reconstruct. Reproducibility is not a maturity-model nicety; it is the property every other capability depends on, and it requires versioning four things rather than the one most teams version.

The environment is the one most often skipped and the one that bites hardest. A minor version bump in a numerical library changes floating-point results, which changes a tree split, which changes a prediction. Pin the digest, not the tag.

The feature store exists to eliminate one specific class of bug

Training-serving skew — features computed differently in training than in production — is the most common cause of a model underperforming its evaluation. It arises structurally: training reads from a warehouse in batch with full history available, serving computes from a stream with only the present.

A feature store resolves this by defining each feature once and serving both paths from that definition, with an offline store for training and a low-latency online store for inference. Its most important capability is point-in-time correctness: assembling a training set using only feature values that were actually available at each historical decision point.

CI for machine learning tests things unit tests do not

Conventional software CI verifies that code behaves as specified. A model has no line-by-line specification, so the pipeline must test the data, the training process and the resulting artefact's behaviour.

• Data validation: schema conformance, range and null checks, and distributional comparison against the previous training set. A silently changed upstream unit will otherwise be learned as signal.
• Training reproducibility: the same inputs and seed produce the same artefact, verified by hash. Non-determinism here makes every later investigation guesswork.
• Performance gate: the candidate must beat the incumbent on a held-out set, and the comparison must be on the same slices, not just in aggregate.
• Slice testing: aggregate improvement can hide serious regression on a subgroup. Test every slice that matters commercially or legally, individually.
• Behavioural tests: invariances the model must respect and directional expectations it must satisfy — a credit model whose score improves when income falls has a defect that no aggregate metric will reveal.
• Serving parity: run the candidate through the actual inference path and confirm outputs match the training-time evaluation exactly.

Deployment, and the sample size nobody calculates

Replacing a model wholesale on the strength of offline metrics is unnecessary risk when staged rollout is available. The progression is shadow — the candidate scores live traffic with its output discarded, confirming the serving path and latency — then canary on a small share of real traffic, then full.

The step that gets skipped is deciding how much traffic and for how long. A canary is a hypothesis test, and one that lacks the power to detect the regression you care about will pass regardless of whether the model is better.

For outcomes with delayed labels the canary cannot measure what matters within any reasonable window, and the honest response is champion-challenger: run both continuously, route a fixed minority share to the challenger, and promote on accumulated evidence rather than on a rollout schedule.

Whatever the strategy, rollback must be a configuration change with the previous artefact already loaded and warm — not a redeploy. If reverting takes twenty minutes, the incident lasts twenty minutes.

Retraining: on a trigger, not on a calendar

Fixed-schedule retraining is simultaneously too frequent for stable domains, where it burns compute and adds risk for no gain, and too slow for volatile ones, where the model degrades between runs. Trigger on evidence instead: measured performance falling below the business floor, aggregate input drift crossing threshold, a material volume of new labels, or a known upstream change such as a new product line.

Automate the pipeline fully and keep a human at the promotion gate. Fully automated retraining and deployment can propagate a data quality incident into production in one cycle — the corrupted data trains a model that passes aggregate gates because the corruption is consistent, and the failure is not visible until it reaches customers.

Governance is now a compliance obligation

The EU AI Act introduces obligations that map directly onto engineering artefacts, and they are the artefacts a well-run MLOps practice already produces. For high-risk systems the requirements include risk management across the lifecycle, data governance and documented dataset characteristics, technical documentation, automatic logging of operation, human oversight provisions, and demonstrated accuracy and robustness.

Read as an engineering specification rather than a legal one, that is: a model registry with lineage from data to deployment; documented training data provenance and known limitations; inference logging sufficient to reconstruct any individual decision; an operational human review path; and evidence of performance across relevant subgroups. Organisations that built these for their own operational reasons find compliance largely a documentation exercise. Organisations that did not find it a rebuild.

The order that works

Reproducibility first, because nothing else is meaningful without it. Then the feature definition layer, because training-serving skew is the highest-frequency defect. Then automated validation, then staged deployment with real rollback, then trigger-based retraining. Governance artefacts fall out of the first four if the registry and lineage were built in rather than bolted on — and the practical difference between teams that run twenty models comfortably and teams that struggle with three is almost entirely this sequence, not the modelling.