12.12.5 — Monitoring and Drift

Cartwise's engineering dashboards are green almost all the time: requests answered, errors near zero, latency normal. They were green throughout the two worst model problems of the year. In the first, fraudsters discovered that old, dormant accounts were trusted, and for six weeks the fraud model approved their orders, the aged-account pattern from 12.1.4. In the second, the Lucknow incident of 12.12.4, a broken data stream fed the model wrong values for two weeks. In both cases every request succeeded, quickly, with a confident prediction. The predictions were simply wrong.

Ordinary monitoring asks is the service working? Model monitoring asks a different question: is the model still right? That question is harder, because a model being wrong produces no error, and the true answer, such as whether an order was fraud, often arrives weeks later. This page is about answering it anyway: what to watch, how to measure change in data, what to do while labels are missing, and how all of this works for LLM features, whose answers have no single right value.

1. Four layers of signals, arriving at different speeds

A model in production produces four layers of signals. The useful way to arrange them is by how soon each one arrives, because the earliest signal of a problem is usually not the most direct one.

The four layers of signals, placed by when they arrive. Service health is fastest but blind to wrong answers; outcomes are the truth but arrive weeks late. Inputs and outputs, in between, are where most problems are first noticed.

① Service health, the latency, errors and throughput of Chapter 10.10, arrives in seconds and is essential, but it cannot detect a model that is wrong, because a wrong model is a perfectly healthy service.

② Inputs arrive with each request. Two kinds of check apply. Data quality: is a feature suddenly missing for a third of requests, outside its possible range, or of an unexpected type? Those are almost always broken pipelines, and they are the commonest cause of model incidents. Distribution: do this week's inputs, taken together, look like the training data? Section 3 shows how to measure it.

③ Outputs arrive with each prediction. If the fraud model usually declines 1.8% of orders and today declines 4%, something has changed, either in the world or in the system, and the change shows within hours. Watching the distribution of scores, not just the decision rate, catches subtler shifts.

④ Outcomes are the only direct measure: was the prediction right? For the ticket classifier, the answer arrives within hours, when staff move a misrouted ticket. For fraud, it arrives when a chargeback does, often four to eight weeks later.

The lesson of the figure is that you cannot wait for outcomes. Both of the year's incidents were visible in the inputs and outputs long before the outcomes confirmed them: the Lucknow feature changed overnight, and the aged accounts appeared as a growing share of approved orders from accounts dormant for over a year. Monitoring inputs and outputs is how you get a warning in days rather than a report in months.

2. The ways the world moves

When inputs or outputs change, the change can be one of several different things, and they need different responses. Probability notation, from 12.2.5, describes them precisely. Write X for the inputs (the order's features) and Y for the true answer (fraud or not).

Data drift, also called covariate shift, is a change in P(X), the distribution of the inputs, while the relationship between inputs and answers, P(Y \mid X) ("the probability of Y given X"), stays the same. When Cartwise added a "buy now, pay later" option, a new kind of order appeared, larger on average and paid in a new way. Fraud did not work differently, but the model was now seeing inputs unlike most of its training data, where its predictions are least reliable.

Concept drift is a change in P(Y \mid X): the same inputs now mean something different. The aged-account pattern is concept drift. An account opened years ago used to signal a trustworthy customer; once fraudsters began buying and reviving old accounts, the same input started to signal risk. The inputs look exactly as before, so input monitoring cannot see concept drift. Only outcomes, or the outputs moving in a suspicious direction, reveal it.

Label shift, or prior shift, is a change in P(Y), how common each answer is, while what each answer looks like stays the same. During a festival sale, the share of fraudulent orders may rise even though fraudulent orders look as they always did. A model's raw scores may still rank orders well, but thresholds chosen for the old fraud rate, the threshold choices of 12.3.7, may need adjusting.

And there is a fourth thing that looks like drift and is not: a broken pipeline, as at Lucknow. A feature whose values change because the code computing it broke is not the world moving. It is a bug, it needs fixing rather than retraining, and retraining on the broken data would make things worse. That is why every drift alert is first checked against the data pipeline.

3. Measuring how much the data has moved

To compare this week's inputs with the training data, you need a number that says how different two distributions are. The most widely used in industry, especially in banking and credit, is the Population Stability Index (PSI).

To compute it for one feature, divide its range into bins, typically five to ten, chosen from the training data. Let e_i be the fraction of training examples in bin i, the expected share, and a_i the fraction of this week's requests in bin i, the actual share. Then

\text{PSI} = \sum_i (a_i - e_i) \ln\!\left(\frac{a_i}{e_i}\right)

Read it as: for each bin, take how much its share changed, multiply by the logarithm of how many times larger or smaller it got, and add these up over all bins. Each term is zero when a bin's share is unchanged, and positive otherwise, whichever direction it moved, because a_i - e_i and \ln(a_i / e_i) always have the same sign. Bins that changed a lot relative to their size count most, because of the logarithm: going from 10% to 20% weighs more than going from 30% to 40%.

Here is the order-value feature, in five bins from smallest to largest orders, in the week after "buy now, pay later" launched:

Bin	Training share e_i	This week a_i	(a_i - e_i)\ln(a_i/e_i)
smallest	0.30	0.20	0.0405
small	0.25	0.22	0.0038
middle	0.20	0.20	0.0000
large	0.15	0.18	0.0055
largest	0.10	0.20	0.0693

Take the last row as an example: (0.20 - 0.10) \times \ln(0.20 / 0.10) = 0.10 \times \ln 2 = 0.10 \times 0.693 = 0.0693. Adding the column gives a PSI of about 0.119. The industry's rough guide reads that number as follows: below 0.1, no meaningful change; between 0.1 and 0.25, a moderate shift worth investigating; above 0.25, a major shift. Here, the largest orders doubled their share, and the PSI flagged it in the first week.

import numpy as np

def psi(expected: np.ndarray, actual: np.ndarray, eps: float = 1e-6) -> float:
    e = np.clip(expected, eps, None)                       # (1) avoid dividing by zero
    a = np.clip(actual, eps, None)
    return float(np.sum((a - e) * np.log(a / e)))

def shares(values, edges):                                 # (2) fraction in each bin
    counts, _ = np.histogram(values, bins=edges)
    return counts / counts.sum()

print(round(psi(np.array([0.30, 0.25, 0.20, 0.15, 0.10]),
                np.array([0.20, 0.22, 0.20, 0.18, 0.20])), 3))
# 0.119

Line (1) replaces any empty bin's share with a tiny number, because a bin with zero examples would make the logarithm undefined. Line (2) turns raw values into bin shares, using bin edges computed once from the training data and then kept fixed, so that every week is compared with the same yardstick. The printed result matches the table.

PSI is one of several measures. For a continuous feature, the Kolmogorov–Smirnov test compares two distributions by the largest gap between their cumulative curves, and gives a significance level. For categories, such as payment method, a chi-squared test does the same job. With very large traffic, though, statistical tests flag even tiny, harmless differences as "significant", so a measure of size, like PSI, is usually more useful for alerting than a measure of certainty.

Text and images have no bins

Models whose inputs are text, such as the ticket classifier or "Ask Cartwise", have no single feature to bin. Three approaches work.

Watch derived features. Length, language, the share of messages in Hindi or with an order number, the share of each predicted category: each of these can be binned and tracked with PSI like any number.

Watch embeddings. Embed incoming messages with the models of 12.6.2, group the training messages into clusters (the k-means of 12.3.6), and track what share of each week's messages falls in each cluster, and how many fall far from every cluster. A sudden rise in far-away messages means something new is being asked. When Cartwise launched groceries, the share of tickets far from every training cluster rose from 2% to 9% in a week, and almost all of them mentioned fresh produce, which the classifier had never seen.

Train a classifier to tell old from new. Label a sample of training inputs "old" and a sample of this week's "new", and train a simple classifier to tell them apart. If it cannot do better than chance, with an AUC near 0.5 (12.3.2), nothing meaningful has changed. If it can, the inputs it finds easiest to recognise as "new" show exactly what changed. This works for any kind of input.

4. While the labels are late

For the fraud model, the true outcome of an order is known only when a chargeback arrives or the window for one closes. Monitoring cannot wait two months, so it works on three fronts.

Proxies that arrive early. Signals that tend to move with the true outcome but arrive sooner: the share of orders flagged for manual review that reviewers overturn, customer complaints about wrongful declines, early fraud reports from payment providers, and the share of approved orders from unusual segments such as long-dormant accounts. The aged-account pattern was visible in that last proxy weeks before the chargebacks confirmed it.

Joining outcomes when they arrive. Every prediction is stored with its ID, inputs, score, decision and model version, so that when the outcome eventually appears, it can be joined to the prediction that made it. Performance is then reported by the week the prediction was made, not the week the label arrived, so a bad week stands out clearly once its labels are in.

Segments, always. Every metric is broken down by the slices that matter: city, payment method, order value, new versus returning customer, device. The overall number can look fine while one slice collapses, as at Lucknow. Slices also matter for fairness: a model that has become worse for one group of customers is a problem in its own right, as Chapter 12.14.3 discusses.

5. Monitoring LLM features

An answer from "Ask Cartwise" has no label that will arrive later and settle whether it was right. Monitoring an LLM feature therefore relies on a wider set of signals, grouped the same way.

Layer	Signal	What a change suggests
Inputs	mix of topics, languages, lengths	new kinds of question
Inputs	messages far from known clusters	something the system wasn't built for
Retrieval	searches with no good match	missing or outdated documents
Retrieval	top similarity score falling	questions drifting from the documents
Outputs	decline ("I don't know") rate	coverage gap, or over-caution
Outputs	answers with citations	grounding slipping
Outputs	sampled faithfulness score	invented content (12.9.4)
Outcomes	thumbs down, rephrased questions	answers not helping
Outcomes	hand-offs to a person, repeat contact	unresolved problems
Cost	tokens and cost per answer	longer prompts, loops, model change

A few of these deserve explanation. Retrieval signals are among the most useful early warnings, because when questions stop matching the documents, answers soon go wrong. When groceries launched, the share of searches with no passage above the similarity threshold doubled in a day, because no grocery policies had been written yet. The decline rate cuts both ways: a rise may mean a coverage gap, and a fall may mean the assistant has started answering questions it should not. Sampled quality scores run the faithfulness check of 12.9.4 and the judges of 12.8.6 on a random sample of live answers every day, which turns "is it still good?" into a trend line. And behavioural signals, such as a customer rephrasing the same question, asking for a person, or contacting support again within a week, are the closest thing an LLM feature has to an outcome label.

LLM features have one source of change that classical models do not: the model itself can change underneath you, when you upgrade it or when the version you call is retired. Every signal is split by the release version of 12.12.4, so that a change in behaviour can be traced to a change in release or ruled out.

6. Alerting without drowning

Monitoring that produces constant alerts gets ignored, and then it misses the real problem. A few practices keep alerts meaningful.

• Compare with the right baseline. Festival weeks, weekends and month-ends all look different. A drift measure computed against the same period last year, or against a seasonally adjusted baseline, avoids an alert every Diwali.
• Alert on sustained change. One unusual hour is often noise; three consecutive hours, or a day, is a pattern.
• Separate urgent from interesting. A feature missing for 30% of requests is urgent: page someone now. A PSI of 0.12 on order value is interesting: open a ticket for the model's owner to look at this week.
• Send each alert to a named owner, with a runbook: a short document listing what to check first (the data pipeline, a recent release, a known event such as a sale) and how to roll back.

7. From a signal to an action

A monitoring signal is the start of an investigation, which follows the same order every time.

1. Is it a bug? Check the data pipelines, recent releases of the model and of anything upstream, and the data-quality checks. Most alerts end here, and the fix is to repair the pipeline, not to touch the model.
2. Is it a real change in the world? If the inputs genuinely moved (a new product line, a new payment method, a new kind of fraud), decide whether the model still performs acceptably on the new inputs, using whatever outcomes and proxies exist.
3. Act on the right lever. For a classical model: adjust thresholds for a label shift, retrain on recent data for data or concept drift, through the gates and stages of 12.12.4. For an LLM feature: write the missing documents, update the prompt, add evaluation cases for the new kind of question, or change the model, again through the release process.
4. Make it permanent. Add the new pattern to the evaluation set, and if the monitoring was slow to notice, add the signal that would have caught it sooner. After the aged-account incident, "share of approved orders from accounts dormant for over a year" became a monitored output with its own alert.

What the interviewer will push on

"How do you monitor a model in production?" They want layers, not just accuracy:

1. Service health.
2. Input data: its quality, and how it compares with the training data.
3. Outputs: their spread, and how often each decision is made.
4. Real outcomes once the true answers arrive, joined to each prediction by its ID and reported by the week the prediction was made.

All of it is split by customer segment and by model version. The tell is explaining why inputs and outputs matter when labels are late.

"What is the difference between data drift and concept drift?" Data drift is a change in the inputs' distribution, P(X), visible immediately; concept drift is a change in what inputs mean, P(Y \mid X), visible only through outcomes or suspicious output trends. Mention label shift, P(Y), and that broken pipelines masquerade as drift.

"How would you detect drift?" PSI per feature with fixed bins from the training data (under 0.1 stable, 0.1–0.25 moderate, over 0.25 major), KS or chi-squared tests with the caution that large samples make everything significant, and for text, derived features, embedding clusters, or a classifier trained to tell old inputs from new.

"How do you monitor an LLM application with no labels?" Input mix, retrieval match quality, decline and citation rates, sampled faithfulness and judge scores, user behaviour (thumbs, rephrasing, hand-offs, repeat contact), cost and latency, all split by release version.

"Drift alert fires. What do you do?" Check for a bug first, then decide whether the world really changed and whether performance is still acceptable, then use the right lever, then add a permanent check.

Something to volunteer: an accurate model can still be learning from biased data. A fraud model only sees outcomes for the orders it approves, so its monitoring numbers can look healthy while its blind spot grows, which is why a small exploration sample, from 12.12.4, is part of monitoring and not only of training.

Recall

• Model monitoring asks whether the model is still right, which produces no errors when it fails. Four layers by arrival time: service health (seconds, blind to wrong answers), inputs (quality and distribution, earliest warning), outputs (scores, decision rates), outcomes (labels, often weeks late).
• Kinds of change: data drift, a change in P(X); concept drift, a change in P(Y \mid X), invisible in inputs; label shift, a change in P(Y); and broken pipelines, which look like drift but need fixing, not retraining.
• PSI = \sum_i (a_i - e_i)\ln(a_i/e_i) with fixed bins from training: under 0.1 stable, 0.1–0.25 moderate, over 0.25 major. KS and chi-squared tests flag tiny differences at large scale. For text: derived features, embedding clusters and far-away share, or an old-versus-new classifier.
• With late labels: early proxies, predictions stored and joined to outcomes later, performance reported by prediction week, every metric by segment.
• LLM features: input mix, retrieval match quality, decline and citation rates, sampled faithfulness, user behaviour, cost, all by release version.
• Alerts: seasonal baselines, sustained changes, urgent versus interesting, named owners with runbooks. Investigate: bug first, then real change, then the right lever, then a permanent check.

Self-test: A bin's share falls from 0.25 to 0.10; what does it contribute to the PSI? Why could input monitoring not detect the aged-account fraud, and which signal could? Name two signals that would show "Ask Cartwise" is being asked about something its documents do not cover.

Next: 12.12.6 Recommenders and Ranking — the last production system in this chapter is the one most e-commerce revenue flows through: choosing, from 400,000 products, the twenty each customer sees, within a fraction of a second.