12.12.4 — The Model Lifecycle

In March, Cartwise retrained its fraud model from 12.3.7 on three more months of data. The new version scored better on the test set: higher precision at the same recall. An engineer copied the new model file onto the scoring servers on a Tuesday afternoon.

By Wednesday morning, the payments team reported that legitimate orders from Lucknow were being declined at three times the usual rate. It took four hours to roll back, because nobody was certain which file was the previous version or which training data had produced it. The cause, found the next day, was a feature: the fraud rate in the customer's pincode over the last 30 days. The training data computed it from the warehouse's complete records. The live scoring service computed it from a stream that, for Lucknow, had been missing two weeks of data since a pipeline change. The new model relied on that feature more than the old one, so the gap that the old model had barely noticed now decided thousands of orders. This is the training-serving skew of 12.4.4, arriving in production.

Nothing in the model was wrong. What was missing was the process around it: a record of which model was which, a way to compare a new model with the old one on live traffic before trusting it, and a rollback that took a minute instead of four hours. That process is the model lifecycle, the subject of this page. The discipline of running it is usually called MLOps, machine-learning operations.

1. Why a model needs a lifecycle at all

Chapter 12.1.4 explained why a model is a living system rather than a finished artefact. Three facts make its lifecycle different from ordinary software's.

Three things change, not one. A program's behaviour is set by its code. A model's behaviour is set by its code, the data it was trained on, and the trained weights. A change to any one changes predictions, and reproducing a past prediction needs all three.

Correctness is statistical. A bug in ordinary code usually fails loudly. A worse model fails quietly: it still returns a prediction, with no error, for every request. Only measurement over many requests shows that it is worse.

Models decay untouched. The world moves away from the data a model was trained on, so a model that was right in March is gradually less right by September, with no change at all. 12.12.5 covers how to notice.

The lifecycle is the set of steps that handles these facts: every model is versioned with its full history, trained by a repeatable pipeline, compared against the current model before release, released gradually, easy to roll back, and retrained on purpose. The rest of this page takes them in order.

2. Versions and lineage: the model registry

Training runs are recorded by the experiment tracking of 12.5.7. The models worth keeping go into a model registry: a catalogue of model versions, each stored with everything needed to explain and reproduce it.

For each version, the registry holds:

• the model file itself, in a fixed location the serving system reads from;
• lineage, the chain that produced it: the code version, the exact training data version, the feature definitions and their versions, and the training settings;
• evaluation results on the fixed evaluation set and on its important slices (by city, payment method, order size);
• a model card: a short document saying what the model is for, what it must not be used for, how it was evaluated, and its known weaknesses;
• its status, through aliases such as champion for the version serving traffic and challenger for the version being tested against it.

The aliases are what turn rollback from a four-hour search into a one-line change. Serving systems never load "the file on the server"; they load "whichever version is champion". Moving the alias back to the previous version is the rollback. With MLflow, a widely used open-source registry, it looks like this:

import mlflow
from mlflow import MlflowClient

client = MlflowClient()

v = mlflow.register_model(f"runs:/{run_id}/model", "fraud-scorer")                 # (1)
client.set_model_version_tag("fraud-scorer", v.version, "train_data", "orders@2026-03-01")  # (2)
client.set_registered_model_alias("fraud-scorer", "challenger", v.version)          # (3)

# the scoring service, at start-up and whenever the alias changes:
model = mlflow.pyfunc.load_model("models:/fraud-scorer@champion")                  # (4)

# rollback: point champion back at the previous version
client.set_registered_model_alias("fraud-scorer", "champion", previous_version)     # (5)

Line (1) registers the model produced by a tracked training run, so the registry entry links back to that run's settings, metrics and code version. Line (2) records the data version it was trained on. Line (3) marks it as the challenger, a candidate, not yet serving customers. Line (4) is how the scoring service loads a model: by alias, never by file name. Line (5) is the whole rollback. If the alias had existed in March, Wednesday's problem would have been a one-minute fix, followed by a calm investigation.

3. The training pipeline and its gates

A model that matters is never trained by someone running a notebook. It is trained by a pipeline, code that runs the same steps in the same order every time, on a schedule or when triggered. This is the pipeline idea of 12.4.4 extended to the model. Running it automatically is called continuous training.

The pipeline's most important parts are its gates: checks that stop a new model from going further unless it is genuinely better and safe.

def training_pipeline():
    data = load_training_data(as_of=today(), point_in_time=True)                      # (1)
    report = validate(data, contract=FRAUD_DATA_CONTRACT)
    if not report.ok:
        return stop("data failed validation", report)                                 # (2)

    model = train(data.train, params=FRAUD_PARAMS)
    new = evaluate(model, EVAL_SET)                                                   # (3)
    old = evaluate(load_model("fraud-scorer@champion"), EVAL_SET)

    if new.recall_at_review_budget < old.recall_at_review_budget - 0.005:
        return stop("worse than champion overall", new, old)                          # (4)
    for slice_name in ["city", "payment_method", "order_value_band", "new_customer"]:
        worst = worst_slice_drop(new, old, slice_name)
        if worst.drop > 0.02:
            return stop(f"regression on {slice_name}={worst.value}", worst)           # (5)
    if new.p99_latency_ms > 20:
        return stop("too slow to serve", new)                                         # (6)

    register_as_challenger(model, data.version, new)                                  # (7)

Line (1) builds the training data point in time: every row uses the feature values as they were at the moment of that order, never later values, the leakage rule of 12.4.2. Line (2) stops if the data breaks its contract (missing columns, impossible values, a sudden change in volume), because training on broken data produces a confidently broken model. Line (3) scores the new model and the current champion on the same fixed evaluation set, so the comparison is fair.

Line (4) requires the new model to be at least as good overall, measured by the metric that matches the business decision: how many frauds are caught within the number of orders the review team can check, as 12.3.7 set out. Line (5) is the gate that would have caught part of March's problem: it compares the two models on every important slice and stops if any one slice, such as one city, gets more than slightly worse, even if the overall number improved. An average can rise while one group gets much worse. Line (6) checks the model is still fast enough to serve inside the fraud check's time budget. Line (7) registers the model only as a challenger: passing the gates earns it a trial on live traffic, not the job.

4. Releasing safely: from offline to all traffic

Offline evaluation, however careful, is done on past data with features computed in the training environment. Live traffic has today's data, features computed by the serving system, and customers who react to the model's decisions. The two regularly disagree. So a challenger is released in stages, each one testing something the previous stage could not.

Each stage exposes the challenger to more reality, and catches a kind of problem the stage before could not: a worse model offline, training-serving skew in shadow, harm to real customers in the canary, and a lack of real business benefit in the A/B test.

Shadow mode. The challenger scores every live request alongside the champion, but only the champion's decision is used; the challenger's is logged. Nothing a customer experiences changes, and yet the challenger is now running on real traffic with features computed by the real serving system. Comparing the two models' scores request by request finds skew directly: if they disagree far more in one city than elsewhere, the features for that city are suspect. Shadow mode would have found March's Lucknow problem in an afternoon, with no customer affected. It is the most valuable stage and the one most often skipped.

Canary. The challenger's decisions are used for a small share of traffic, say 5%, chosen at random. The service watches guard metrics, the numbers that must not get worse, such as the share of orders declined and the number of complaints, and rolls back automatically if they cross a threshold. A canary limits the damage of a problem that shadow mode could not see, because it only appears once decisions are acted on.

A/B test. For changes whose value must be proven in business terms, a larger randomised split runs for long enough to measure outcomes: fraud losses, legitimate orders wrongly declined, customer complaints. Three cautions apply to A/B tests of models. First, some outcomes arrive late: whether an approved order was fraud is known only weeks later, when the chargeback arrives, so the test must run that long. Second, behaviour changes in the first days, as customers react to anything new, which is called the novelty effect, so early results mislead. Third, in a marketplace, the two groups can affect each other. A pricing model that discounts in one group can draw sellers' stock away from the other, so the split is made by city or by time period rather than by customer. Statistical tests decide when a difference is real rather than noise, the reasoning of 12.2.5.

A variation called a multi-armed bandit shifts traffic towards whichever version is winning while the test runs, instead of keeping a fixed split. It loses less money to the worse version, at the cost of a less clean measurement, and suits frequent, low-stakes comparisons such as which product images to show.

All traffic. The challenger becomes champion by moving the alias, and the previous champion stays loaded and ready, so rolling back is instant.

Offline and online results disagree, and online wins. A model with better offline metrics can do worse live: the evaluation data was not like today's, the offline metric was not quite the business goal, or customers respond to its decisions in ways no past data could show. Offline evaluation decides which models are worth testing; live results decide which one serves.

5. When things go wrong

When a model misbehaves in production, the order of actions matters.

1. Roll back first. Move champion to the previous version. Investigate afterwards, with customers protected.
2. Check the data before the model. Most model incidents turn out to be data problems upstream: a pipeline that stopped, a column whose meaning changed, a feature computed differently in one place. March's incident was one. The data validation of 12.4.4 and the monitoring of 12.12.5 usually point to the cause.
3. Have a fallback. If no model can be trusted, fall back to simple rules: for fraud, the rules the company used before the model, with more orders sent for manual review. A degraded service beats a wrong one.
4. Turn the incident into a permanent check. After March, Cartwise added a city-level slice to the pipeline gate, a shadow stage to every fraud release, and a data check comparing each feature's live values with its training values. The same failure cannot pass unnoticed again.

6. Retraining

Models are retrained for one of three reasons, and choosing between them is a deliberate decision:

Trigger	Strength	Weakness
Schedule (e.g. monthly)	simple, predictable	may be too late, or unnecessary
Performance drop	retrains when it matters	needs labels, which arrive late
Input drift	acts early	can fire on harmless changes

Most teams combine a schedule with drift and performance alerts that trigger an early run. Every retrained model goes through the same gates and release stages as the first; retraining is not a shortcut around them.

One trap needs special care. A model's decisions shape the data it will later be trained on. The fraud model declines suspicious orders, and a declined order never becomes a chargeback, so the training data never shows whether it was really fraud. Over time, the model learns only from the orders it chose to approve, and its blind spots can grow without anyone seeing them. The standard remedy is a small, controlled amount of exploration: approving a tiny random sample of borderline orders, with limits on the amount at risk, and sending another sample for manual review. The outcomes of these orders show what the model's declines would have been. Recommendation systems face the same trap and use the same remedy, covered in 12.12.6.

7. LLM applications have a lifecycle too

An LLM feature rarely involves training a model, but it has the same three-way problem: its behaviour is set by the model, the prompt and the context it is given (retrieval settings, the document index, the tools), and a change to any of them changes answers. So these are versioned together, as one release:

support-answer v14
  model:      claude-sonnet-5-5
  prompt:     support-system v14 (git 3f2a91c)
  retrieval:  hybrid, top 20 → rerank → 5; index built 2026-10-12
  tools:      get_order_status v3, check_return_eligibility v2
  eval:       support set 412 cases, 94.6% pass (v13: 93.9%)

The lifecycle steps carry over directly. The evaluation sets of 12.8.6 and 12.10.9 are the gates. The gateway's routing table from 12.12.1 is the alias: sending 5% of support-answer traffic to v14 is a canary, and moving back to v13 is a rollback. Shadow mode works too: run v14 on a copy of live questions and compare its answers with v13's using a model judge, before any customer sees them.

LLM applications add one lifecycle event of their own. Providers retire models, typically with months of notice. A feature pinned to a model version must be migrated before that date: run the evaluation set on the replacement, adjust the prompt where the new model behaves differently, release through the same stages. Teams that treat model retirement as a routine scheduled release handle it calmly; teams that discover it in the final week do not.

8. Retiring a model

Every model is eventually replaced, and retiring one properly is part of the lifecycle. Its traffic is moved to the successor and confirmed at zero. Anything that still calls it is found, by its callers in the gateway logs, and moved. Its registry entry, lineage and evaluation results are archived rather than deleted, because a decision it made may be questioned months later, by a customer, an auditor or a regulator, and the question "why was this order declined?" must still be answerable. That requirement, being able to explain any past decision, is the real reason for lineage, and Chapter 12.14.3 returns to it.

What the interviewer will push on

"How do you deploy a new model safely?" They want the stages, and what each one catches:

1. Offline checks against the current model, including each slice of the data.
2. Shadow mode, which catches differences between training and live data.
3. A canary, with guard metrics and automatic rollback.
4. An A/B test, for business results.
5. Full rollout, with the old version kept ready.

The tell is putting shadow mode in and explaining why.

"What is training-serving skew, and how do you catch it?" The same feature computed differently in training and serving, producing different inputs with no error. Prevent it with one feature definition used by both (a feature store), check live feature distributions against training ones, and catch it in shadow mode by comparing challenger and champion request by request.

"What is a model registry for?" Versioned models with lineage (code, data, features, settings), evaluation results, model cards and aliases, so that serving loads by alias, rollback is moving an alias, and any past prediction can be traced to what produced it.

"Your new model is better offline but worse in the A/B test. Which do you trust?" The A/B test, and then investigate why they disagree: different data, a metric that does not match the goal, or customers reacting to the model's decisions.

"How do you version an LLM application?" Model, prompt, retrieval configuration, index and tools together as one release, gated by evaluation sets and released through the gateway as canaries; plan migrations before providers retire models.

Something to volunteer: a model learns only from the outcomes of the decisions it allows, so its blind spots can grow unseen. A small, limited amount of exploration, such as approving a random sample of borderline orders, is how you keep measuring what it would otherwise never see.

Recall

• MLOps handles three facts: behaviour depends on code, data and weights; correctness is statistical and failures are silent; models decay untouched.
• A model registry holds each version with lineage (code, data, features, settings), evaluation results, a model card and aliases (champion, challenger). Serve by alias; rollback is moving the alias.
• Train with a pipeline: point-in-time data, validation against a data contract, evaluation of new and champion on the same set, gates on overall metric, on every important slice, and on latency; pass registers a challenger only.
• Release in stages: offline, shadow (catches training-serving skew), canary with guard metrics and automatic rollback, A/B test for business outcomes (late labels, novelty effects, marketplace interference), full rollout with the old version ready. Online beats offline when they disagree.
• Incidents: roll back first, check the data, fall back to rules, turn the incident into a permanent check.
• Retrain on a schedule plus drift and performance triggers, through the same gates; use limited exploration to escape the feedback loop.
• LLM features version model, prompt, retrieval, index and tools together; evaluation sets are the gates; gateway routing gives canaries and rollback; plan for provider model retirement. Archive retired models with lineage.

Self-test: Which release stage would have caught March's Lucknow problem with no customer affected, and how? Why does the pipeline compare slices as well as the overall metric? Name the five parts of an LLM feature that should be versioned together.

Next: 12.12.5 Monitoring and Drift — a released model has to be watched; the next page covers how the world drifts away from a model's training data, how to measure it before labels arrive, and what to watch for LLM features whose outputs have no single right answer.