12.9 — MLOps: Running Models in Production

A fraud model scores 0.94 in the notebook and 0.71 in production.

Nothing is broken. In training, one feature was transactions_last_30_days, computed from a table containing every transaction. In production it is computed from a stream that lags by two hours, so recent transactions are missing. The model sees a different feature than the one it learned, and no test catches it because the code is correct in both places.

Training-serving skew is the defining failure of machine learning in production, and it exists because a model system has three moving parts where ordinary software has one.

1. What is different from normal software

Code, data and model are three artefacts and all three must be versioned. Reproducing a result needs the code, the exact dataset, the hyperparameters, the environment and the random seed. Missing any one and you cannot answer "why did this prediction happen".

Correctness is statistical. There is no assertion that fails. A model that has quietly degraded from 94% to 71% raises no exception and serves no error page.

It decays without being touched. The world changes; the model does not. Software that has not been deployed for six months behaves identically; a model that has not been retrained for six months may not.

The feedback loop is slow and sometimes absent. You learn whether a loan prediction was right in two years. You learn whether a fraud prediction was right when a chargeback arrives — or never, if you blocked the transaction, which means your training data is biased by your own past decisions.

2. Experiment tracking

A hundred training runs later, "which one produced the model we deployed" must be answerable.

Log per run: hyperparameters, metrics over time, the code commit, the dataset version, the environment, the hardware, and the resulting model artefact.

Weights & Biases and MLflow are the common tools. W&B is a hosted service with strong visualisation, sweep management and collaboration; MLflow is open source, self-hosted, and includes a model registry. The choice matters less than the discipline, which is that every run is logged automatically rather than when someone remembers.

wandb.init(project="fraud", config={"lr": 2e-4, "r": 16, "epochs": 3})
...
wandb.log({"epoch": e, "train_loss": tl, "val_loss": vl, "val_pr_auc": pr})
wandb.log_artifact(model_path, type="model")     # the trained weights, tied to this run

The value shows up months later, when a model behaves oddly and you need to know exactly what produced it. A model registry is the other half: named, versioned models with stages (staging, production, archived), lineage back to the run, and an approval record.

3. Data and features

Version the data. A model is only reproducible if the dataset is. Content-hash the files and record the hash, or use a tool (DVC, LakeFS, or the table formats from Chapter 7.8.2, whose snapshots give this for free).

A feature store exists to solve the opening problem directly. Features are defined once, and both the training pipeline and the serving path read from that definition — offline in bulk for training, online with low latency for inference.

The critical property is point-in-time correctness. When building a training row for a transaction at 14:32 on 3 March, every feature must have the value it had at that moment, not the value it has now. Computing "transactions in the last 30 days" with today's table leaks the future into the past — the leakage of Chapter 12.3, in its most expensive form.

You do not always need a feature store. For a model with a handful of features computed from one request, the machinery costs more than it saves. Adopt one when several models share features, or when point-in-time correctness has already bitten you. But the discipline — one definition, used by both paths — applies at any size.

4. Labelling

Supervised learning needs labels, and getting them is usually the largest cost in a project.

Label Studio is the common open-source tool, covering text, image, audio and video, with support for pre-labelling from a model, review workflows and multiple annotators.

What decides label quality:

Written guidelines with examples, especially edge cases. Ambiguity is resolved differently by different people, and inconsistently by the same person on different days.

Measure inter-annotator agreement — Cohen's or Fleiss' kappa. If two humans agree only 70% of the time, no model will exceed that, and the fix is the guidelines, not the model. This single measurement prevents a large amount of wasted effort.

Overlap a sample — have several people label the same items — to detect drift in one annotator's interpretation.

Active learning — label the examples the model is least certain about, rather than a random sample. Often reaches the same accuracy with a fraction of the labels.

Model-assisted labelling — a model pre-labels, a human corrects. Far faster, and it introduces an automation bias: reviewers accept plausible wrong labels. Measure the correction rate; if it is near zero, people are rubber-stamping.

Weak supervision — write labelling functions (heuristics, keyword rules, existing systems), and combine their noisy votes into probabilistic labels. Useful when hand-labelling is impossible at the required scale.

5. Serving

Three shapes, and choosing wrongly is expensive:

Batch — score everything overnight and store the results. Simplest, cheapest, and the answer is stale by up to a day. Frequently sufficient, and frequently skipped in favour of real-time serving that nobody needed.

Online — score on request, in milliseconds. Needed when the input only exists at request time.

Streaming — score events as they arrive from a queue.

For classical models, serving is an ordinary service: load the model, expose an endpoint, and the usual concerns apply. Keep preprocessing inside the served artefact — a pipeline object containing the transformations and the model — so training and serving cannot diverge.

For large models, serving is its own discipline and the constraints come from Chapter 12.5.3:

• Continuous batching is the largest throughput win. vLLM, TensorRT-LLM and similar servers implement it.
• Paged attention manages the KV cache in fixed blocks like virtual memory (Chapter 2.5), removing the fragmentation that otherwise wastes most of the memory.
• Quantise to fit more, and to go faster, since decode is memory-bandwidth-bound.
• Cache prefixes for shared system prompts.

Autoscaling is genuinely hard for accelerators. Loading a model into memory takes tens of seconds to minutes, so scaling up does not help a spike that has already arrived. Keep a warm floor, scale on queue depth rather than on utilisation, and accept that some over-provisioning is the cost of predictable latency.

6. Hardware

Why GPUs win: a neural network is mostly matrix multiplication, and every element of the result is independent (Chapter 12.2). A CPU has a few dozen fast general cores; a GPU has thousands of simple ones plus much higher memory bandwidth. For training and for large-model inference the difference is one to two orders of magnitude.

VRAM is the binding constraint, not compute. A model that does not fit does not run at any speed. The arithmetic from Chapter 12.5.3 — parameters × bytes per parameter, plus the KV cache — is what decides which instance you need.

The tiers, and what each is for:

Class	Typical use
T4 (16 GB)	Cheap inference for small models; widely available
L4 (24 GB)	Modern efficient inference; video and mid-size models
A10G (24 GB)	Inference and small fine-tunes
A100 (40/80 GB)	Training, large-model inference
H100 / newer (80 GB+)	Frontier training, high-throughput serving

Choose by memory first, then by bandwidth, then by cost per throughput. A model that fits on a T4 should not be served on an A100; utilisation is the number that matters, and a barely-used large accelerator is the most common source of waste in a machine learning budget.

Spot and preemptible instances are much cheaper and can be reclaimed. Use them for training with frequent checkpointing — losing ten minutes is acceptable. Do not use them for serving unless you have capacity elsewhere.

Buy or host, for large models. A hosted API costs per token, needs no operations, and scales instantly. Self-hosting costs per GPU-hour whether or not you use it, and becomes cheaper only above a genuinely high and steady volume. Compute the crossover with your own numbers: measured tokens per second per instance against the API's per-token price. The other reasons to self-host — data residency, latency, and a fine-tuned model you control — are often more decisive than cost.

7. Deploying a new model

Shadow mode. Run the new model on live traffic, log its predictions, serve the old one. No user impact, real data, and it catches training-serving skew before anyone is affected. This is the highest-value step and the most often skipped.

Canary. 1% of traffic, then 5, 25, 100, watching metrics at each stage.

A/B test. Split traffic and compare business outcomes, not model metrics. Run it long enough for statistical significance, and beware novelty effects in the first days.

Champion-challenger. The current model keeps serving while challengers are evaluated continuously in shadow, and promotion happens on evidence.

And the fact that surprises people: offline and online metrics disagree regularly. A model with better AUC can perform worse in production — because the offline evaluation used a different distribution, because the metric does not capture what users do, or because the model's own predictions change future behaviour. The online result is the truth, and offline metrics are a filter for what is worth testing.

8. Monitoring

The ordinary service metrics apply — latency percentiles, error rate, throughput, cost (Chapter 10.10).

And four that are specific:

Data drift — the input distribution has moved. Compare current feature distributions with the training distribution using population stability index or a KL divergence, per feature. This is your earliest warning, because it needs no labels and arrives before performance degrades visibly.

Concept drift — the relationship between inputs and outputs has changed, even though inputs look the same. Fraud patterns adapt; user behaviour shifts. Only detectable with labels.

Prediction drift — the output distribution has moved. Cheap, and a good proxy when labels are slow.

Performance, when labels arrive. For fraud, labels come with chargebacks weeks later; for churn, months. Build for that lag explicitly — store the prediction with the input so it can be joined to the outcome whenever it appears.

Two practical rules. Alert on inputs, not only outputs: a feature that has become null for 30% of requests is a broken upstream pipeline, and it is visible immediately where the accuracy drop is not. And segment your metrics — overall accuracy hides a model that has failed for one customer type or one region, which is both an engineering and a fairness issue (Chapter 12.10).

For large language models the monitored quantities differ: token cost per request, latency percentiles, refusal rate, the rate of outputs failing schema validation, evaluation scores on the regression set (Chapter 12.6.3), and user signals such as thumbs-down and rephrasing.

9. Retraining

Decide the trigger deliberately:

• Scheduled — weekly or monthly. Simple, predictable, and sometimes unnecessary or too late.
• Performance-triggered — retrain when a metric crosses a threshold. Correct, and it requires labels.
• Drift-triggered — retrain when the input distribution moves. Earlier, and it can fire on harmless changes.

Whichever you choose, the pipeline must be automated and the result gated. A retrained model that is worse must not deploy: compare against the current one on a fixed evaluation set, and require it to win before promotion.

Beware the feedback loop. A recommender trained on clicks it generated learns from its own past behaviour and narrows over time. Reserve a small fraction of traffic for randomised or exploratory serving so the training data keeps some independence from the model's own choices.

10. Governance

A model card documents what a model is for, what data it was trained on, how it performs overall and by segment, its known limitations and its intended and out-of-scope uses. For regulated use it is a requirement; everywhere else it is the document that stops a model built for one purpose being reused for another it was never validated on.

Lineage. From a prediction, you should be able to reach the model version, the training run, the dataset version and the code commit. When a decision is challenged — and under Chapter 8.7's rules it can be — this is what answers it.

Approval and audit. Who approved this model for production, on what evidence, and when. Record it in the registry alongside the artefact rather than in a chat thread.

Recall

• Training-serving skew is the defining production failure — the same feature computed differently in the two paths, with no error raised.
• Machine learning versions three artefacts: code, data and model. Correctness is statistical (nothing throws), models decay without being touched, and feedback is slow or absent — and blocking a prediction means never learning if it was right.
• Log every run automatically: parameters, metrics, code commit, data version, environment, artefact. A model registry adds stages, lineage and approval.
• A feature store's real contribution is point-in-time correctness — a training row must see the feature values that existed then. Adopt it when features are shared or leakage has bitten; adopt the discipline always.
• Labels: written guidelines with edge cases, and measure inter-annotator agreement — no model beats the humans' agreement rate. Active learning cuts label counts; model-assisted labelling is fast and invites rubber-stamping, so watch the correction rate.
• Serving: batch is often enough. For large models, continuous batching, paged attention, quantization and prefix caching are the levers. Autoscaling accelerators is slow — keep a warm floor and scale on queue depth.
• VRAM is the binding constraint, not compute. Choose by memory, then bandwidth, then cost per throughput; utilisation is the cost metric. Spot for training with checkpoints, never for serving.
• Shadow mode is the highest-value deployment step and the most skipped. Offline and online metrics disagree regularly, and online is the truth.
• Monitor data drift (no labels needed — the earliest warning), concept drift, prediction drift, and delayed performance. Alert on inputs, and segment every metric. Gate every retrain against the current model, and keep some randomised traffic so the model does not train on its own choices.

Self-test: What exactly is training-serving skew, and why do tests miss it? · What does point-in-time correctness prevent? · Why does inter-annotator agreement cap model accuracy? · Why is autoscaling GPUs harder than autoscaling web servers? · Which drift signal arrives before labels do? · Why reserve a slice of traffic for randomised serving?