12.1.3 — From Business Problem to ML Problem

The head of operations at Cartwise sends you one line: "Can we use AI to reduce late deliveries?"

That sentence is not something you can build. It does not say what the model should predict, when it should predict it, what anyone will do with the prediction, or how you would know it worked. Most failed ML projects fail right here, before any model exists. The team picks something to predict that looks related to the goal, builds a model that scores well on a test set, ships it, and then discovers that nothing in the business changed, because no decision was ever wired to the prediction.

This page is the procedure for turning a vague request into a problem you can build and measure. It has six steps, and we will work through all six on the late-delivery request.

The framing funnel. Each step narrows the request until what is left is specific enough to build and to measure. Skipping a step does not save time; it moves the discovery of the problem to after launch.

① Name the decision the prediction will change

A prediction on its own does nothing. Somebody, or some piece of code, has to act differently because of it. So the first question is never "what can we predict?" It is "what decision will be made differently, by whom, and when?"

Ask that about late deliveries and you get several very different answers, each of which leads to a different project:

• Choose the courier at the moment of shipping. The warehouse picks between FastShip, SlowPost and BulkCargo for every parcel. If a model can predict which courier will be fastest for this parcel to this address, the warehouse system can pick that one automatically.
• Warn the customer early. If a parcel is likely to be late, send a message on day one rather than letting the customer discover it on day five. That will not make the parcel faster, but it can cut angry support calls.
• Promise an honest date at checkout. The checkout page currently says "3–5 days" for everything. A model could show a specific date for each order, which affects whether people buy.

Notice how different these are. The first one changes an action inside the warehouse. The second one changes a message. The third one changes what the customer sees before they pay, which means a wrong prediction could lose sales. When you take a vague request back to its owner and ask which of these they mean, you will often find that they had not decided, and that conversation is the most valuable hour of the project.

For the rest of this page, assume operations chose the first option: pick the courier automatically for each parcel.

② Define the prediction, and the moment it is made

Now the prediction can be stated exactly: for this parcel and each available courier, how many days will delivery take? The warehouse will then pick the courier with the smallest predicted number. That is a regression problem from 12.1.2, predicted once per courier, followed by picking the smallest.

The second half of this step is the one people forget, and it causes the most expensive bug in machine learning. You must state the exact moment the prediction is made, because only information that exists at that moment is allowed into the model.

Everything left of the purple line can be fed to the model. Everything right of it is unknown when the warehouse has to choose a courier, so a model that sees it during training is learning from the future.

Here is how the bug happens. The historical delivery table has a column called first_scan_delay_hours: how long after handover the courier first scanned the parcel. It is a wonderful predictor of lateness. A model trained with it will score brilliantly on historical data. But at the moment the warehouse chooses a courier, the parcel has not been scanned yet, so that column does not exist. In production the model gets an empty value, and it performs terribly.

Using information during training that will not be available at prediction time is called leakage, because the answer has leaked into the inputs. It is the single most common reason a model that looks excellent in testing fails in production. The defence is exactly this step: write down the prediction moment, and for every input, ask "would I know this at that moment?" Chapter 12.4.2 covers the subtler forms, such as a customer's average delay computed using deliveries that happened after this one.

③ Define the label exactly

The label is the right answer the model learns from, and it needs a definition precise enough that two engineers would compute the same value from the same records. "Days to deliver" sounds precise. It is not. Is it calendar days or working days? Measured from when the customer ordered, or from handover to the courier? What about a parcel that was never delivered because it was lost? What about a parcel the customer refused at the door?

Each of these choices changes what the model learns. If you measure from the order time, the model will partly be learning how slow your own warehouse is, which is not something the courier choice can fix. So for this decision, the right label is working days from handover to the courier until the delivered scan, with lost parcels kept in the data with a large penalty value, because a courier that loses parcels should be avoided. Write the definition down. You will need it when somebody asks in six months why the model behaves the way it does.

④ Choose how success is measured, at two levels

There are always two levels of measurement, and you need both.

The business metric is what operations actually cares about: the percentage of parcels delivered late, and the number of "where is my order" support contacts. This is what decides whether the project was worth doing.

The model metric is what you can compute quickly on historical data to compare one model with another, without launching anything. For a regression on days, a natural one is the mean absolute error, or MAE. It is simply the average distance between what the model predicted and what actually happened. If the model predicted 3 days and the parcel took 5, the error on that parcel is 2 days. Average that over every parcel in a test set, and you have the MAE. An MAE of 0.6 means "on average, the prediction is off by just over half a day". It has the big advantage of being in the same units as the thing you care about, so anyone can understand it.

The two levels must be connected by a story you can explain: "a lower MAE means we pick the truly fastest courier more often, which means fewer late parcels." If you cannot tell that story, the model metric is the wrong one. Chapter 12.3.2 and Chapter 12.12.4 return to this, including how to check the business metric with a live experiment.

⑤ Build the simplest possible baseline first

Before any machine learning, build the dumbest thing that could possibly work and measure it with the metric from step ④. This simple reference point is called a baseline. It has three jobs.

First, it tells you how hard the problem really is. Second, it gives the ML model something to beat: a model is only worth its cost if it clearly beats the baseline. Third, and surprisingly often, the baseline turns out to be good enough, and you ship it and move on.

Here is Cartwise's data: eight past parcels to learn from, and four more parcels held back for testing.

import pandas as pd

train = pd.DataFrame({
    "courier": ["FastShip", "FastShip", "FastShip", "SlowPost",
                "SlowPost", "SlowPost", "FastShip", "SlowPost"],
    "region":  ["metro", "metro", "rural", "metro",
                "rural", "rural", "rural", "metro"],
    "days":    [1.5, 2.5, 4.0, 3.0, 6.0, 5.0, 5.0, 4.0],
})
test = pd.DataFrame({
    "courier": ["FastShip", "SlowPost", "FastShip", "SlowPost"],
    "region":  ["metro", "rural", "rural", "metro"],
    "days":    [2.0, 7.0, 4.0, 3.0],
})

global_avg = train["days"].mean()                                   # (1)
group_avg = train.groupby(["courier", "region"])["days"].mean()     # (2)

test["pred_global"] = global_avg                                     # (3)
test["pred_group"] = [group_avg[(c, r)]                              # (4)
                      for c, r in zip(test["courier"], test["region"])]

def mae(pred):                                                       # (5)
    return (test["days"] - pred).abs().mean()

print(global_avg)                      # → 3.875
print(mae(test["pred_global"]))        # → 1.5
print(mae(test["pred_group"]))         # → 0.625

Here is what each marked line does and why it is there.

Line (1) computes the plainest baseline imaginable: the average delivery time over all past parcels, ignoring everything about them. The eight values add up to 31, and 31 divided by 8 is 3.875 days. This baseline predicts 3.875 for every parcel.

Line (2) computes a slightly smarter baseline: the average for each combination of courier and region. groupby splits the rows into groups that share the same courier and region, and .mean() averages the days within each group. FastShip in a metro area averages (1.5 + 2.5) / 2 = 2.0 days. FastShip in a rural area averages (4.0 + 5.0) / 2 = 4.5. SlowPost metro averages (3.0 + 4.0) / 2 = 3.5, and SlowPost rural averages (6.0 + 5.0) / 2 = 5.5.

Line (3) gives every test parcel the global average as its prediction. Line (4) looks up the right group average for each test parcel: the first test parcel is FastShip metro, so it gets 2.0, the second is SlowPost rural, so it gets 5.5, and so on.

Line (5) defines the mean absolute error: subtract the prediction from the truth, take the absolute value so that being 1 day too high and 1 day too low both count as 1 day of error, and average.

Now the arithmetic behind the printed results. For the global baseline, the four errors are |2.0 − 3.875| = 1.875, |7.0 − 3.875| = 3.125, |4.0 − 3.875| = 0.125 and |3.0 − 3.875| = 0.875. They add to 6.0, and 6.0 divided by 4 is 1.5. For the group baseline, the errors are |2.0 − 2.0| = 0, |7.0 − 5.5| = 1.5, |4.0 − 4.5| = 0.5 and |3.0 − 3.5| = 0.5. They add to 2.5, and 2.5 divided by 4 is 0.625.

So a lookup table of averages, which took two lines of code and no machine learning at all, cut the error from 1.5 days to 0.625 days. That 0.625 is now the number to beat. If a gradient-boosted model from Chapter 12.3.4, which can also use weight, distance, day of the week and the weather, reaches 0.45, it is earning its place. If it reaches 0.6, it is not worth the extra pipeline, the retraining and the monitoring, and the lookup table should ship instead.

Two things about this test are worth noticing because they are habits you should keep. The test parcels were kept apart from the training parcels, so the score is honest: a method is always judged on examples it did not learn from. And the second test parcel, SlowPost rural at 7.0 days, is worse than anything in training. No baseline and no model will get that one right, and that is a useful reminder that some of the error is simply randomness in the world that no model can remove.

What the baseline looks like for an LLM problem

For problems in the LLM world, the baseline idea still holds but takes a different form. Suppose Cartwise wants to route support emails into four teams. The natural modern baseline is to send each email to a capable LLM with a short instruction: "Classify this email as refund, delivery, account or payment." This requires no training data at all, which is called zero-shot use of the model, because it gets zero examples of the task.

That baseline may be excellent. But measure it, and then also price it. At 40,000 emails a week, each LLM call costs a little money and takes perhaps a second. A small classifier trained on a few thousand labelled emails costs almost nothing per email and answers in milliseconds. If the small model matches the LLM's accuracy, it wins. If it is much worse, the LLM wins. You can only make that call once you have measured both. This trade-off appears again in Chapter 12.10.10 and Chapter 12.11.

⑥ Check the data, and price each kind of mistake

The last step has two halves.

The data. Do you have enough labelled examples, are they recent enough to reflect how things work now, and will you keep getting new ones? For couriers, Cartwise has hundreds of thousands of past deliveries, and every new delivery produces a new label within a week, so the data is healthy. For a brand new courier with no history, the model will have nothing to learn from, and you need a plan for that case: perhaps fall back to the baseline until the new courier has a few hundred deliveries.

The cost of each kind of mistake. Wrong predictions are not all equally bad. Predicting a courier will take 3 days when it takes 4 is a small problem. Choosing a courier that loses the parcel is a large one. For classification problems this asymmetry is even sharper. If the fraud model wrongly blocks a real customer, Cartwise loses one sale and maybe that customer forever. If it wrongly lets fraud through, Cartwise loses the goods and pays the dispute fee. Those two costs are almost never equal, and the threshold you choose from 12.1.2 should come from them. Write the costs down in rupees before you build anything.

When not to use machine learning at all

Sometimes the honest outcome of these six steps is that ML is the wrong tool. Here are the cases where you should say so plainly.

When a rule is good enough and stable. "Orders to an address on the blocked list are rejected" is a rule. It is exact, it never needs retraining, and anyone can read it. Turning it into a model would make it less accurate and harder to explain.

When the answer must be exactly right every time. Tax calculation, the total of a shopping cart, whether a user has permission to see a document. These have correct answers defined by rules. A model that is right 99.5 percent of the time is a bug generator here. This is especially worth remembering with LLMs: never ask an LLM to add up an invoice or to decide who can access a file. Compute those with ordinary code, and use the model for the parts that are genuinely fuzzy.

When there is no data, or the labels are unreliable. A model learns whatever its labels say. If the labels are wrong in a consistent way, the model will learn that mistake faithfully and repeat it at scale.

When you must explain every decision and the model cannot. Some decisions, like refusing someone a loan, may legally require a reason a person can understand. Simpler models can give one. Complex ones often cannot without extra tools, which Chapter 12.3.3 and 12.14.3 discuss.

When the gain does not pay for the upkeep. A model is not a one-time cost. It needs a pipeline to retrain it, monitoring to catch when it goes stale, and someone on call when it misbehaves. Chapter 12.12 covers that upkeep. If the model beats the baseline by a sliver, the baseline is the better engineering decision.

The one-page problem card

All six steps fit on one page, and writing it before you start is a habit worth keeping for life. Here is the completed card for the courier problem:

DECISION     Warehouse picks a courier for each parcel automatically.
PREDICTION   Working days from handover to delivery, for each available courier.
MOMENT       At handover. Inputs: weight, size, destination, distance, courier,
             day of week, public holiday flag. NOT scans or delays (future).
LABEL        Working days from handover to the delivered scan.
             Lost parcels: 15 days (penalty). Refused at door: excluded.
METRICS      Model: mean absolute error on the last 4 weeks, held out.
             Business: % of parcels late; "where is my order" contacts per 1,000.
BASELINE     Average days per (courier, region): MAE 0.625 on the test sample.
DATA         ~400k deliveries/year; new labels within 7 days. New courier:
             fall back to baseline until 300 deliveries.
ERROR COSTS  A day late ≈ ₹40 in support load. Lost parcel ≈ ₹2,500.
SHIP IF      Model MAE ≤ 0.45 AND late-rate drops in a 2-week live test.

Every line on that card answers a question that would otherwise come up after launch, at a much worse moment.

Recall

• Start from the decision the prediction will change. A prediction that no decision uses is a wasted project.
• State the prediction moment, and allow only inputs known at that moment. Using information from after that moment is leakage, the most common reason models that test well fail in production.
• Define the label precisely enough that two people compute the same value. Measure success at two levels: a business metric and a model metric such as mean absolute error, connected by a story you can explain.
• Build a baseline first and judge it on held-out data. A two-line lookup table cut the error from 1.5 to 0.625 days, and that is the number any model must beat. For LLM problems, a zero-shot prompt is the baseline, and it must be priced as well as scored.
• Price each kind of mistake. Skip ML when a stable rule works, when answers must be exactly right, when labels are unreliable, or when the gain does not pay for the upkeep.

Self-test: Why is first_scan_delay_hours a dangerous input for the courier model? Compute the MAE of predicting 4.0 days for every test parcel above. Name one problem at an online store that should never be handed to an LLM, and say why.

Next: 12.1.4 The Living System — a shipped model is not finished; it lives inside a loop of data, training, serving and feedback, and that loop is what the rest of this Part teaches you to build.