12.1.4 — The Living System

Cartwise's fraud model went live in January. In testing it caught 82 percent of fraud while wrongly blocking only one genuine order in two hundred, which was far better than the old hand-written rules. For two months, fraud losses fell. By June they were almost back to where they started.

Nobody had touched the code. The model file was byte-for-byte identical to the one that shipped. So what broke?

Three things had happened, and none of them were bugs in the usual sense:

1. The fraudsters changed what they did. Once large orders from new accounts started getting blocked, they switched to "aged" accounts: real accounts bought cheaply from people who had stopped using them. The pattern the model had learned was simply no longer what fraud looked like.
2. The inputs changed shape. In April Cartwise launched a "buy now, pay later" option. Orders paid that way had no card country at all, so one of the model's most useful inputs was suddenly empty for twelve percent of orders. The model had never seen that during training and handled it badly.
3. The model was starving itself of lessons. A blocked order never ships, so it never produces a dispute, so it never gets a label saying whether it really was fraud. Every week, the orders the model was most suspicious of were exactly the ones that disappeared from the next round of training data.

This is the most important difference between machine learning systems and ordinary software, so it is worth saying slowly. In ordinary software, behaviour is defined by code, and code does not change unless somebody changes it. In a machine learning system, behaviour is defined by data, and the world keeps changing the data whether you touch anything or not. A model is a snapshot of how the world looked during training. The moment it ships, the world starts moving away from the snapshot.

So an ML system is not a thing you build once. It is a loop that keeps running, and an engineer's job is to build the whole loop, not just the model in the middle.

1. The loop

The machine learning loop. The model in the centre is the smallest part. The six stages around it keep running for as long as the model is in use, and the purple arrows never stop.

Let us walk around the loop once, using the fraud model, and for each stage say what happens and which of January's three failures it would have caught.

① Collect data. Every order, with everything known about it at the moment of checkout, gets recorded. Later, when a dispute arrives or the delivery is confirmed, the outcome gets recorded too and joined back to the original order. The quality of everything downstream is capped by what gets recorded here. If Cartwise does not log the device a customer used, no future model can ever learn from it.

② Features and labels. The raw records are cleaned and turned into the inputs the model sees, which are its features from 12.1.1, and the label is computed exactly as the problem card from 12.1.3 defined it. This is also the stage where the "buy now, pay later" failure should have been caught: a check that says "the card country is missing in 12 percent of rows this week, up from 0.1 percent" would have raised the alarm in April. Chapter 12.4 covers this stage.

③ Train and evaluate. A model is trained on the prepared data and then judged on data it has not seen, and it only moves forward if it beats both the baseline and the model currently in production. Chapters 12.3 and 12.5 teach the training, and Chapters 12.3.2 and 12.3.3 teach the honest evaluation.

④ Serve. The approved model is put behind an interface the checkout can call, and it has to answer within the time budget, here about 50 milliseconds, for every order, all day. Chapter 12.12 is about doing this reliably and cheaply.

⑤ Monitor. Once the model is live, you watch three things. First, do its inputs still look like the training data? Second, do its outputs look normal? For example, is it suddenly blocking twice as many orders as last week? Third, once the true answers arrive, is it still right? This is the stage that would have caught the aged-account fraud: the fraud rate among orders the model approved would have crept up week by week. Chapter 12.12.5 builds this monitoring.

⑥ Feedback. New outcomes flow back into stage ①, and alerts from stage ⑤ trigger retraining. The loop closes, and a new model, trained on the world as it looks now, replaces the old snapshot.

The three ways the world moves away from the snapshot

Each of January's failures has a name, and you will meet all three again in Chapter 12.12.5.

Concept drift is when the relationship between inputs and the right answer changes. The aged-account trick is concept drift: an old account used to be a sign of a trustworthy customer, and now it can be a sign of fraud. The inputs look the same as before, but what they mean has changed.

Data drift is when the inputs themselves change, even if what they mean does not. The arrival of "buy now, pay later" is data drift: a whole new kind of order appeared with a missing field the model had never seen. Data drift can be caught immediately, because you can see the inputs as they arrive. Concept drift can only be caught once the labels come back, which for fraud takes weeks.

A feedback loop is when the model's own decisions change the data it will later learn from. This is the subtlest of the three, and it deserves a closer look, because it shows up in nearly every system where a model makes decisions.

The feedback loop, seen in code

Here is roughly what Cartwise's weekly training job did:

orders = load_orders(last_90_days)                     # (1) every order from the period
labelled = orders[orders["status"] == "shipped"]       # (2) only shipped orders can be disputed
labelled["y"] = labelled["was_disputed"].astype(int)   # (3) dispute = fraud (label 1)
model = train(labelled)                                # (4) learn from what remains

Line (1) loads three months of orders. Line (2) keeps only those that shipped, because the label in line (3) comes from disputes, and an order that was blocked never shipped and so could never be disputed. Line (4) trains on what is left.

Read line (2) again with the model in mind. The orders the model blocked are exactly the ones it thought were fraud. They are removed before training, every single week. So the next model learns only from orders the previous model was happy with, and it never gets to find out whether its blocks were right. If the old model wrongly blocked every customer in one city, that mistake can never be corrected, because those customers never appear in the training data again. The model is grading its own homework using only the questions it chose to answer.

The standard fix is to deliberately spend a little money to buy information:

def decide(order, score):
    if score > THRESHOLD:
        if random.random() < 0.01:                     # (1) 1 in 100 would-be blocks…
            return "manual_review"                     # (2) …goes to a human instead
        return "block"
    return "approve"

Line (1) picks one out of every hundred orders the model wanted to block, completely at random. Line (2) sends that order to a human reviewer instead, who checks it properly and records the true answer. Those reviewed orders become labels for exactly the region the model cannot otherwise see. Because the sample is random, it is a fair picture of everything the model blocks, so you can measure how often the blocks are right. The cost is a little reviewer time and a small amount of fraud that slips through review. The benefit is a model that can find and fix its own mistakes. This idea, keeping a small random slice of decisions out of the model's hands so you can keep learning, comes back in recommendation systems in Chapter 12.12.6, where the same loop decides which products a customer ever gets to see.

2. The model is the small part

In 2015 a group of engineers at Google published a paper titled "Hidden Technical Debt in Machine Learning Systems". Its most quoted observation was a picture. In a real production ML system, the code that defines and trains the model is a tiny box in the middle. Around it sit much bigger boxes: collecting data, checking it, computing features, serving, monitoring, managing settings and running the machines.

Redrawn after Sculley et al. (2015). The purple box in the middle is the model code that tutorials teach. Everything else is what keeps it working in production, and it is where most of the engineering time goes. The box sizes show the idea, not exact measurements.

The paper's point was not that models are unimportant. It was that a team which spends all its effort on the purple box and none on the rest ends up with a system that is fragile and expensive to change. This is why this Part spends as much time on data, evaluation and serving as it does on models.

3. The same loop for LLM applications

When you build "Ask Cartwise" on top of an LLM you did not train, you might think the loop goes away. It does not. It changes shape.

The loop for an LLM application. There is no training stage, but every other part of the loop survives: you still need a test set, you still watch production, and real failures still drive the next version.

Where the classical loop retrains a model, the LLM loop changes the prompt, the documents retrieved, the tools offered, or which model is called. Where the classical loop has labels arriving from the real world, the LLM loop has traces: complete logs of each conversation, including what was retrieved and every tool call, which you read to find out why an answer went wrong. And the role of the test set is played by an eval set: a collection of real questions with known good answers that every new version must pass before it ships. The drift problems survive too. Cartwise changes its returns policy, and an assistant that was right last week is now confidently wrong, even though nothing in the code changed. Chapter 12.8.6 and Chapter 12.10.9 build this loop properly.

4. Where each stage is taught

The four Cartwise problems each exercise a different part of the loop, and following them is the easiest way to keep your place in this Part.

Stage of the loop	Where it is taught	Cartwise thread
The maths underneath	12.2	all of them
Training classical models	12.3	fraud model
Data, labels, features	12.4	fraud model
Training neural networks	12.5	product photos
Representing meaning as numbers	12.6	product search
How LLMs work inside	12.7	Ask Cartwise
Building on LLMs, evals	12.8, 12.9	Ask Cartwise
Models that take actions	12.10	refund agent
Adapting a model to you	12.11	Ask Cartwise
Serving, cost, monitoring	12.12	all of them
Images, speech	12.13	product photos, voice
Security, fairness	12.14	refund agent
Whole systems, end to end	12.15	all of them

You do not need to remember this table. Each page will say which thread it is advancing and link back to where the earlier piece was built.

Recall

• In ordinary software, behaviour comes from code. In ML, behaviour comes from data, and the world keeps changing the data, so a shipped model starts going stale immediately.
• The ML loop: collect data → features and labels → train and evaluate → serve → monitor → feedback, round and round. The model is the smallest part of the system.
• Concept drift: what the inputs mean changes. Data drift: the inputs themselves change. A feedback loop: the model's own decisions decide what it learns from next.
• A model that blocks orders never sees labels for the blocked ones. Sending a small random slice of would-be blocks to human review buys back that missing information.
• LLM applications keep the loop with different parts: design, eval set, ship, traces, failure analysis, and back to design.

Self-test: Which of the three June failures could have been caught on the first day it happened, and which only weeks later? Why does the 1 percent manual-review sample have to be chosen at random rather than, say, the orders the model was least sure about? What plays the role of retraining in an LLM application?

Next: 12.2.1 Vectors and the Dot Product — every model on this page, from the fraud score to the LLM, is built out of one operation applied billions of times, and that operation is where the maths begins.