12.3.7 — Case Study: Fraud Detection

This page builds Cartwise's real fraud system from start to finish, using every tool from Chapters 12.1 to 12.3. It is written the way a design document is written at a real company: the brief, the numbers, the decisions, the rejected alternatives, and what goes wrong. Read it as the place where the separate pieces finally connect.

1. The brief, in numbers

Cartwise processes about 40,000 orders a day. About 0.5 percent are fraud, so roughly 200 fraudulent orders a day. The average fraudulent order costs Cartwise about ₹4,000 in lost goods and dispute fees, which is about ₹8 lakh a day, or close to ₹30 crore a year. Today a set of 140 hand-written rules catches about 45 percent of it, and blocks so many genuine customers that the support team has a dedicated queue for "my order was cancelled for no reason".

The constraints are just as concrete:

• Time. The fraud check sits inside the payment step of checkout. It gets 50 milliseconds, including fetching whatever information it needs.
• People. A team of reviewers can examine about 300 orders a day by hand.
• Explanations. When a reviewer looks at a flagged order, they need to see why it was flagged, or they cannot review it quickly.
• Delay in the truth. A fraudulent order is usually confirmed only when the real cardholder disputes the charge, which can take up to 90 days.

2. The problem card

Following the six steps of 12.1.3:

DECISION     For every order at payment: APPROVE, send to REVIEW, or BLOCK.
PREDICTION   Probability that the order is fraudulent.
MOMENT       After the customer clicks "Pay", before the payment is confirmed.
             Inputs: the order, the account, the device, and counts of recent
             activity up to this instant. Nothing that happens after this moment.
LABEL        1 if a fraud dispute arrives within 90 days, or a reviewer confirms
             fraud. 0 if 90 days pass with no dispute. Orders under 90 days old
             without a dispute are NOT labelled 0 yet: their label is unknown.
METRICS      Model: area under the precision–recall curve on the most recent
             held-out month; recall at a review capacity of 300 orders a day.
             Business: fraud losses per month; genuine orders blocked per 1,000.
BASELINE     Current rules: about 45% recall; about 9 genuine orders blocked
             for every fraud blocked.
ERROR COSTS  Missed fraud ≈ ₹4,000. Wrongly blocked genuine order ≈ ₹300.
             Review ≈ ₹60 per order in staff time.

Two lines on that card deserve a closer look, because they are where fraud systems most often go wrong.

Three outcomes, not two. A yes-or-no model forces a choice between blocking a customer and letting fraud through. Adding a middle option, human review, lets the model act automatically only where it is very sure, and send the uncertain middle to people. That changes the threshold question from "where do I cut?" into "where do I cut twice?", which section 6 answers.

The label is not known for recent orders. An order from last week that has not been disputed is not "genuine" yet. It might be disputed next month. If the training data treats every undisputed recent order as genuine, it teaches the model that recent fraud looks genuine, which is exactly the newest fraud you most need to catch. The rule is simple and must be enforced in code: only orders older than the 90-day window count as genuine. The cost is that the model always learns from patterns at least three months old, which is why the faster labels from reviewers matter so much.

3. The features

The fraud score can only be as good as the information it sees. Cartwise's team grouped its features into five families, each answering a different question a human investigator would ask.

Family	The question	Examples
Order	What is being bought?	amount, number of items, share of easily resold goods (phones, gift cards)
Account	Who is buying?	account age, number of past orders, past disputes, email address age
Velocity	How fast is this happening?	orders from this card in the last hour; new addresses on this account today
Consistency	Do the details agree?	card country vs shipping country; device language vs claimed country
Network	Who else is connected?	other accounts using this address, card or device in the last 30 days

The velocity family is usually the most powerful, and it is worth seeing why. A stolen card is valuable for a short time, before the real owner notices, so fraudsters use it quickly: many orders, many addresses, in a short burst. A genuine customer almost never does that. "Orders on this card in the last hour" captures that urgency directly.

Velocity features are also the easiest to get wrong. They must count only activity before the current order, never including it and never including anything after it. Here is how that looks in code on four orders from one card:

import pandas as pd

orders = pd.DataFrame({
    "card": ["c1", "c1", "c1", "c1"],
    "ts": pd.to_datetime(["2026-10-01 10:00", "2026-10-01 10:20",
                          "2026-10-01 10:45", "2026-10-01 12:00"]),
    "amount": [1500, 2200, 9800, 400],
}).set_index("ts")

orders["card_orders_last_1h"] = (
    orders.groupby("card")["amount"]
          .rolling("1h", closed="left")       # (1) the hour BEFORE this order
          .count()
          .fillna(0)                          # (2) no earlier orders → 0
          .values
)
print(orders["card_orders_last_1h"].tolist())  # → [0.0, 1.0, 2.0, 0.0]

Line (1) counts orders in a sliding one-hour window, and closed="left" is the most important argument in the snippet: it means the window ends just before the current order, so an order never counts itself. Without it, every order would see at least one order in its own window, the model would learn a slightly wrong rule, and the feature computed live at checkout (where the current order has not been saved yet) would differ from the one it was trained on. Line (2) turns "no earlier orders in the window" into a 0. The output reads naturally: the 10:45 order had two earlier orders in the previous hour (10:00 and 10:20), and the 12:00 order had none, because the window from 11:00 to 12:00 is empty.

The general rule is called point-in-time correctness: every feature for a past order must be computed exactly as it would have been known at that order's moment. Chapter 12.4.2 covers the subtler forms. The team also maintains a list of forbidden features, the shortcuts of 12.3.3: any column written by the review team, the order's cancellation status, refund flags, and anything from the courier. All of them are known only after the decision, and all of them would make the model look brilliant in testing and useless in production.

4. Climbing the ladder of models

The team trained three candidates on orders from January to August, validated on September, and tested once on October, the time-based split from 12.3.3. Results on the October test month (numbers are illustrative, but the pattern is typical):

Model	PR-AUC	Recall at 300 reviews/day	Notes
Current rules	about 0.18	45%	baseline, hand-maintained
Logistic regression	about 0.41	58%	60 features, standardised
Gradient-boosted trees	about 0.62	74%	same 60 features, early stopping

Every step up the ladder from 12.3.4 was justified by a clear gain. The boosted trees won by a wide margin, mainly because fraud is full of interactions ("a new account is only risky when combined with high-value electronics and a velocity burst") and thresholds ("risk jumps once the card has been used three times in an hour"), which trees capture directly.

The logistic regression was not thrown away. It became the fallback model: if the boosted model ever fails to load or answers too slowly, the service switches to the logistic regression, which is a single dot product and cannot be slow. It is also what the team uses to explain the overall shape of risk to the finance department, because its odds multipliers from 12.3.1 are easy to present.

Training on rare fraud

With only 0.5 percent of orders as fraud, a model can reach low loss by mostly ignoring the fraud. The team handled this with class weights: each fraud example counts about as much as 20 genuine ones in the loss, so mistakes on fraud are punished harder during training. The alternative, duplicating or synthesising fraud examples, is covered with its trade-offs in Chapter 12.4.3.

Weighting has a side effect that catches many teams. A model trained to care twenty times more about fraud will output probabilities that are too high: it has learned the wrong base rate, as 12.2.5 warned. Its ranking of orders is still excellent, but a score of 0.3 no longer means 30 percent. So after training, the team fits a calibration step on the September validation month, the Platt or isotonic correction from 12.3.2, which maps the raw scores back to honest probabilities. That matters because the next step uses the probabilities as money.

from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.calibration import CalibratedClassifierCV
from sklearn.frozen import FrozenEstimator

model = HistGradientBoostingClassifier(
    learning_rate=0.05, max_leaf_nodes=31, max_iter=2000,
    early_stopping=True, class_weight={0: 1, 1: 20},     # (1)
)
model.fit(X_jan_aug, y_jan_aug)                           # (2)

calibrated = CalibratedClassifierCV(FrozenEstimator(model), method="isotonic")
calibrated.fit(X_sep, y_sep)                              # (3)

p_oct = calibrated.predict_proba(X_oct)[:, 1]             # (4) honest probabilities

Line (1) sets the class weights: each fraud example counts twenty times. Line (2) trains on January to August. Line (3) fits the calibration on September, a month the model did not train on, so the correction is honest; wrapping the model in FrozenEstimator tells scikit-learn the model is already trained and only the calibration should be learned. Line (4) scores the October test month with probabilities that can now be trusted in cost calculations. (X_jan_aug and the others stand for the feature tables built in section 3.)

5. The decision layer: two thresholds from money and people

With calibrated probabilities, each order's decision comes from comparing expected costs, the reasoning from 12.2.5 and 12.3.2, applied to three options:

• Approve costs, on average, p \times ₹4{,}000: the chance it is fraud times what fraud costs.
• Review costs about ₹60 of staff time, plus a small cost for the delay to a genuine customer, and catches the fraud if there is any.
• Block costs, on average, (1 - p) \times ₹300: the chance it is genuine times what a wrongly blocked customer costs.

Review is cheaper than approving once p \times 4000 > 60, which is when p exceeds about 0.015. Block is cheaper than review once (1 - p) \times 300 < 60, which is when p exceeds about 0.8. So by cost alone: approve below 1.5 percent, review between 1.5 and 80 percent, block above 80 percent.

Now the people constraint. On the October data, about 1,100 orders a day scored above 0.015, nearly four times the 300 the reviewers can handle. So the review threshold has to rise until only about 300 orders a day land in the middle band. On October's scores that happened at about 0.06. Orders between 0.06 and 0.8 go to review; orders above 0.8 are blocked automatically, because at that level precision was above 90 percent; everything else is approved.

REVIEW_AT, BLOCK_AT = 0.06, 0.80          # (1) from cost, then raised to fit capacity

def decide(p: float) -> str:
    if p >= BLOCK_AT:
        return "BLOCK"
    if p >= REVIEW_AT:
        return "REVIEW"
    return "APPROVE"

Line (1) is where the business lives. The two numbers are not fixed forever: when the review team grows, the review threshold can come down and catch more fraud; during a sale, when order volume triples, it must go up. This is why the thresholds live in configuration that the fraud operations team can change, not in the model.

The final effect in the October test, illustrative but typical: fraud caught (blocked or confirmed in review) rose from 45 percent to about 76 percent, and genuine orders wrongly blocked fell from about 9 per caught fraud to under 1, because the uncertain middle now goes to people instead of being blocked.

6. Serving it inside 50 milliseconds

The fraud system inside checkout. Steps ① to ⑤ happen within 50 milliseconds. Everything the service sees and decides is logged, because those logs, joined with disputes months later, become the next model's training data.

Walk through one order. ① Checkout sends the order to the fraud service. ② The service fetches the slow-changing features (account history, network links) from an online feature store, a fast key-value database that holds precomputed features ready to read, and the fast-changing velocity counts from a streaming service that updates counters as every order event arrives. Each lookup takes a couple of milliseconds. ③ The service runs the boosted model, which for a few hundred trees takes about a millisecond on an ordinary processor. ④ It applies the two thresholds. ⑤ It returns the decision to checkout.

Three engineering decisions in this design matter as much as the model.

What happens if the fraud service is slow or down? Checkout cannot wait. The team had to choose between failing open (approve the order if no answer arrives in time) and failing closed (hold the order). Failing closed during an outage would stop every sale, so Cartwise fails open for orders under ₹20,000 and holds larger ones for later review. That is a business decision, written down in the design, and the kind of question an interviewer will ask.

The features at serving time must match the features at training time. If the training pipeline computes "orders in the last hour" slightly differently from the streaming counter (for example, by including the current order, the exact bug the code in section 3 avoided), the model silently sees different numbers in production than it learned from. This mismatch is called training-serving skew, and it is one of the most common causes of a model that tests well and underperforms live. The cure is to compute both from the same definitions, which is the main job of a feature store, covered in Chapter 12.4.4.

Log the exact features used for each decision. Months later, when the disputes arrive, the training data must be rebuilt with the features as they were seen at decision time, not recomputed from today's database, where the account might since have changed. Logging them at decision time is the simplest way to guarantee point-in-time correctness.

7. Explanations for reviewers

A reviewer with 300 orders a day has about a minute and a half per order. A bare score of 0.34 does not help them. So the review tool shows the three features that pushed each order's score up the most, computed with the SHAP method from 12.3.3:

Order 88213   fraud probability 0.34   → REVIEW
  + card used on 4 other accounts this month     (+0.15)
  + shipping address first seen today            (+0.09)
  + 3 orders on this card in the last hour       (+0.06)

The reviewer now knows what to check: the linked accounts first. Reviews got faster and more consistent once explanations were added, and the explanations also became an early warning system: when a strange feature started appearing in the top reasons, it was usually either a new fraud pattern or a data bug.

8. Keeping it working

The system went live in November. The loop of 12.1.4 is what keeps it working.

Fraudsters adapt. Within weeks of launch, fraud shifted towards smaller orders, spread over more cards, to stay under the velocity counts. That is concept drift. The team watches the fraud rate among approved orders, using reviewer-confirmed fraud as an early signal before disputes arrive, and retrains every week on the newest matured labels.

The model starves itself of labels. Blocked orders never ship, so they never get a dispute label: the feedback loop from 12.1.4. The team sends 1 percent of would-be blocks, chosen at random, to review instead, which keeps a fair measurement of how often blocks are right.

Inputs change shape. Each week, an automatic check compares the distribution of every feature with its training distribution and alerts when one shifts sharply, such as a new payment method leaving the card-country feature empty. That is data drift, and Chapter 12.12.5 builds the monitoring for it.

Rules still exist, but in their place. Twelve of the old 140 rules survived as hard rules in front of the model: blocked cards, sanctioned countries, known stolen-card lists. These are facts, not patterns, and as 12.1.3 said, facts belong in rules. Everything else became a feature the model can weigh.

What the interviewer will push on

"Design a fraud detection system." They are checking structure: the decision and the label definition first, then features, model, thresholds, serving, and the feedback loop. Starting with "I would use XGBoost" is the tell of a weak answer. A strong one opens with the three-way decision and the 90-day label delay.

"How do you handle the class imbalance?" They want class weights or resampling, and the consequence for calibration, and the right metric (PR-AUC, recall at review capacity). The common wrong answer is to oversample and then quote accuracy.

"How do you pick the threshold?" The answer is cost-based thresholds, then adjusted to review capacity, held in configuration so operations can move them. A fixed 0.5 is the wrong answer.

"What if the fraud service goes down?" They are checking that you treat the model as a dependency with failure modes: a fallback model, a timeout, and an explicit fail-open or fail-closed policy that depends on order value.

"Your model's performance dropped two months after launch. Why?" They want the three causes from 12.1.4 in this setting: fraudsters adapting, the self-starving feedback loop, and input drift, each with how you would detect it.

Volunteer this: the label definition is the most important line in the design. Treating undisputed recent orders as genuine teaches the model that the newest fraud is safe, and it is one of the most common real-world mistakes in fraud modelling.

Recall

• Start from the decision: approve, review or block, with a review capacity of 300 a day and a 50 millisecond budget. A three-way decision lets the model act alone only where it is sure.
• Labels arrive up to 90 days late, so only orders past the window count as genuine. Features fall into order, account, velocity, consistency and network families; velocity is usually strongest and must be point-in-time correct (closed="left").
• Climb the ladder: rules, then logistic regression (kept as the fallback), then boosted trees. Train with class weights, then recalibrate on a later month so the probabilities can be used as money.
• Two thresholds come from expected cost (about 0.015 and 0.8 here), then the review threshold is raised to fit capacity (about 0.06). Thresholds live in configuration.
• Serving needs an online feature store, a fail-open or fail-closed policy, and no training-serving skew. Log exact features per decision; show reviewers the top SHAP reasons; retrain weekly; keep 1 percent random review of blocks; keep true facts as hard rules.

Self-test: Why would labelling last week's undisputed orders as genuine make the model worse at the newest fraud? Using the costs above, at what probability does blocking become cheaper than approving if review did not exist? What breaks if the training pipeline's "orders in the last hour" includes the current order but the live counter does not?

Next: 12.4.1 Labels and Annotation — the case study showed that labels, not algorithms, decide most of a model's quality; Chapter 12.4 is about getting the data right, starting with where labels come from and how to make them trustworthy.