12.4.2 — Features and Leakage

In its first week, a new version of Cartwise's fraud model scored a precision–recall AUC of 0.97 on the test month, up from 0.62. The team was preparing a celebration when one engineer asked to see which feature mattered most. It was account_status, a column from the accounts table. Fraudulent accounts had status suspended. Genuine ones had status active.

The problem was timing. An account is suspended after the fraud is discovered, which is days or weeks after the order. The training pipeline had joined each past order to the accounts table as it looks today, so every fraudulent order in history arrived carrying the word "suspended", written there by the investigation the model was meant to replace. At checkout, when a real fraudster's order arrives, their account is still active. In production, that feature carried no information at all, and the model that scored 0.97 in testing would have been worse than the one it replaced.

This is leakage, which 12.1.3 first warned about: information in the training data that will not be available at prediction time. It is the most expensive bug in applied machine learning because it makes a broken model look excellent, and the bigger the leak, the better it looks. This page covers both halves of working with inputs: turning raw records into good features, and making sure none of them leak.

1. From raw records to features

A model only accepts numbers, arranged as one row per example. Raw business data is not like that. An order record has a timestamp, a payment method name, a free-text address, an amount, and links to an account with its own history. Feature engineering is the work of turning such records into informative numeric columns. For tabular problems it is still where most of the accuracy comes from, often more than the choice of model.

Here are the main kinds of raw value and what is usually done with each.

Numbers. Usable directly, with two common adjustments. Many money-like quantities are heavily skewed: most orders are small, a few are enormous. Taking the logarithm, \log(1 + \text{amount}), squeezes the long tail so that the difference between ₹500 and ₹5,000 counts about as much as between ₹5,000 and ₹50,000, which usually matches how risk behaves. And for linear models and anything distance-based, standardise, as in 12.2.4. Trees, from 12.3.4, need neither.

Categories. Values like payment method (card, upi, wallet, cod) are names, not numbers. You cannot simply number them 1 to 4, because a linear model would then believe that wallet is "three times" card. The standard fix is one-hot encoding: one new column per category, holding 1 if the row has that category and 0 otherwise. For a column with thousands of values, like city or product, one-hot creates too many columns, and two other approaches are used. Hashing maps each value to one of a fixed number of columns using a hash function, accepting that a few values will share a column. Target encoding replaces each category with the average label for that category, for example each city with its historical fraud rate. Target encoding is powerful and is also one of the easiest ways to leak, which section 4 explains.

Dates and times. A raw timestamp is almost never useful by itself, because the model would learn "orders from March 2026", which will never happen again. What helps is extracting the parts that repeat: hour of the day, day of the week, whether it is a public holiday, days until a big sale. One subtlety: hours are circular. Hour 23 and hour 0 are one hour apart, but as plain numbers they look 23 apart. The trick is to place the hour on a circle using two columns, \sin(2\pi h/24) and \cos(2\pi h/24). Hour 0 becomes (0, 1), hour 23 becomes about (-0.26, 0.97), very close, and hour 12 becomes (0, -1), on the opposite side.

Text. An address, a product title or a review needs turning into numbers too: the TF-IDF vectors of 12.3.5, or the embeddings of Chapter 12.6, or simple hand-made signals such as "the address contains a flat number".

Aggregates. Often the most powerful features summarise history: the account's number of past orders, its share of returned orders, the number of orders on this card in the last hour from the case study in 12.3.7. Ratios are especially useful: "this order's amount divided by the account's average order amount" is far more telling than either number alone, because a ₹40,000 order is normal for one customer and alarming for another.

Missing values

Real columns have gaps, and the right treatment depends on why a value is missing. Statisticians distinguish three cases:

• Missing completely at random. The gap has nothing to do with anything, such as a logging glitch that dropped one value in a thousand. Filling it with the median, or dropping the row, does little harm.
• Missing at random. The gap depends on something you can see. Device information is missing for orders placed through the old app version. You can handle it using the column that explains it.
• Missing not at random. The gap depends on the missing value itself, and this is the dangerous one. Fraudsters often block the browser fingerprinting script, so "device fingerprint missing" is itself a fraud signal. Filling it with an average would erase exactly the information you need.

So the safe habit is: fill the gap, and also add a yes-or-no column saying the value was missing. The model can then learn whether missingness matters. Gradient-boosted tree libraries effectively do this automatically.

In code

import numpy as np
import pandas as pd

raw = pd.DataFrame({
    "amount":  [499.0, 52_000.0, 2_300.0],
    "payment": ["upi", "card", "cod"],
    "hour":    [23, 3, 14],
    "device":  ["a91f", None, "77c2"],
})

feat = pd.DataFrame({
    "log_amount":     np.log1p(raw["amount"]).round(2),              # (1)
    "hour_sin":       np.sin(2 * np.pi * raw["hour"] / 24).round(2), # (2)
    "hour_cos":       np.cos(2 * np.pi * raw["hour"] / 24).round(2),
    "device_missing": raw["device"].isna().astype(int),              # (3)
})
feat = feat.join(pd.get_dummies(raw["payment"], prefix="pay", dtype=int))  # (4)
print(feat)
#    log_amount  hour_sin  hour_cos  device_missing  pay_card  pay_cod  pay_upi
# 0        6.21     -0.26      0.97               0         0        0        1
# 1       10.86      0.71      0.71               1         1        0        0
# 2        7.74     -0.50     -0.87               0         0        1        0

Line (1) takes the logarithm of 1 + \text{amount} (log1p stands for "log of one plus", which avoids trouble when an amount is 0); the ₹52,000 order becomes 10.86 rather than a number a hundred times larger than the first. Line (2) puts the hour on a circle: 23:00 sits at (-0.26, 0.97), close to midnight, as described above. Line (3) adds the "was it missing" flag, so the second order's missing device fingerprint becomes a usable signal instead of a gap. Line (4) one-hot encodes the payment method, producing one column per method.

2. Leakage: four ways the answer sneaks in

Every form of leakage breaks the same rule: at training time, the model may only see what it would see at prediction time. It breaks in four common ways.

Target leakage: the answer, or a copy of it, is a feature

A feature that is caused by the label, or recorded because of it, will predict the label perfectly in history and be useless or absent in production. account_status was this. So would be a refund_issued flag in a fraud model, a cancellation_reason field in a model predicting which customers will cancel, or a courier_complaint column in the delivery-time model. The question to ask of every feature is the one from 12.1.3: at the exact moment of prediction, would this value already exist, and would it have this value?

Temporal leakage: the right feature, from the wrong time

This is subtler, because the feature is legitimate; it is just read at the wrong moment. "Number of past orders on this account" is a fine feature. But if the training pipeline reads it from today's accounts table, an account that placed one order in March and forty more by October will appear, in its March order, to have 41 past orders. The model learns from a version of the past that never existed. The account_status case was also temporal: the status at checkout was active; it was the status today that said suspended.

The cure is the point-in-time join: for each training example, attach every feature exactly as it was at that example's moment, ignoring all later changes. Section 3 shows how.

Contamination: preprocessing that peeked at the test set

Any step that learns something from the data, an average for filling gaps, a mean and spread for standardisation, the vocabulary for TF-IDF, a category's fraud rate for target encoding, must be learned from the training set only and then applied unchanged to validation and test data. If it is computed on everything before splitting, a little information about the test set flows into the training inputs, and the test score is slightly optimistic. With target encoding it is far from slight, as section 4 shows. The cure is mechanical, and section 5 shows it.

Group leakage: the same thing on both sides of the split

If one customer's orders appear in both the training and test sets, the model can learn to recognise that customer, by their address or device, rather than learning anything about fraud. The test score then measures recognition, which will not help with new customers. The same happens with near-duplicate product listings, several photos of the same item, or several emails in one support thread. The cure is to split by the group, so all of one customer's rows land on the same side. scikit-learn's GroupKFold does this for cross-validation.

3. The point-in-time join

A point-in-time join. Each order (amber) takes the account's values from the most recent snapshot at or before its own moment (green arrows). The red arrow is the leak: joining to the account as it is today hands March's orders a status that was only written on March 20.

To do this you need the history of each feature, not just its latest value: a table where every change is a new row with the date it became true. Many companies only store the latest value in their main database, which is why a feature store or a regularly saved snapshot of the tables, covered in Chapter 12.4.4, is so valuable for ML. With the history in hand, pandas can do the join in one call:

import pandas as pd

orders = pd.DataFrame({
    "order": ["o1", "o2", "o3"],
    "acct":  ["A", "A", "B"],
    "ts":    pd.to_datetime(["2026-03-01", "2026-03-10", "2026-03-12"]),
})
snapshots = pd.DataFrame({
    "acct":        ["A", "A", "A", "B"],
    "valid_from":  pd.to_datetime(["2026-01-01", "2026-03-05", "2026-03-20", "2026-02-01"]),
    "past_orders": [3, 4, 4, 10],
    "status":      ["active", "active", "suspended", "active"],
})

train_rows = pd.merge_asof(
    orders.sort_values("ts"), snapshots.sort_values("valid_from"),
    left_on="ts", right_on="valid_from",     # (1) match on time…
    by="acct",                               # (2) …within the same account
    direction="backward",                    # (3) only look into the past
)
print(train_rows[["order", "acct", "past_orders", "status"]])
#   order acct  past_orders  status
# 0    o1    A            3  active
# 1    o2    A            4  active
# 2    o3    B           10  active

merge_asof is pandas' "as of" join: for each row on the left, it finds the closest row on the right in time. Line (1) says the matching is by time, comparing each order's timestamp with each snapshot's start date. Line (2) restricts matches to the same account. Line (3) is the guard: direction="backward" means only snapshots at or before the order count, so nothing from the future can attach. Both tables must be sorted by time first, which is why each one is passed through sort_values.

Read the output. Order o1, on 1 March, gets the January snapshot: 3 past orders. Order o2, on 10 March, gets the 5 March snapshot: 4 past orders. Neither sees suspended, because that was written on 20 March. A plain join on acct against today's table would have given both of them suspended and recreated the leak from the opening. Databases and feature stores offer the same operation, often called an as-of join.

4. Target encoding, and why it leaks so easily

Return to the city column. Cartwise ships to 4,000 cities, too many for one-hot encoding. Target encoding replaces each city with its average fraud rate. A city with 2 frauds in 1,000 orders gets 0.002; one with 30 in 1,000 gets 0.03. It is a compact, powerful feature.

Now consider a tiny town with exactly one order in the training data, and that order was fraud. Its fraud rate is 1/1 = 1.0. When the model trains on that order, the feature "city fraud rate" for it is 1.0, a perfect signal, but only because the feature was computed from this very row's own label. The model learns that a city rate of 1.0 means fraud. In production, the new orders from that town were not part of the average, and the town's "1.0" is just one unlucky data point. Every rare category has this problem to some degree, and it can inflate test scores dramatically.

The fixes are standard, and libraries implement them:

• Compute it out of fold. Split the training data into folds as in k-fold cross-validation from 12.3.3. Each row's city rate is computed only from the other folds, so no row ever sees its own label.
• Shrink rare categories towards the overall average. A city with few orders gets a rate close to the global fraud rate, and only well-observed cities get their own number. A common form is \dfrac{\text{frauds} + m \cdot \bar{p}}{\text{orders} + m}, where \bar{p} is the overall fraud rate and m is how many "imaginary average orders" to blend in. With m = 50 and \bar{p} = 0.005, the one-order town gets (1 + 0.25)/(1 + 50) \approx 0.025 instead of 1.0.

scikit-learn's TargetEncoder does both of these automatically when used with fit_transform on the training data.

5. Making leakage hard to commit: pipelines

Contamination happens because preprocessing and model training are written as separate steps, and it is easy to fit the preprocessing on the full dataset first. The structural fix is to bundle every learned step together with the model in a single object, so that fitting it on training data fits everything on training data only, and applying it to test data applies everything with the training statistics.

from sklearn.pipeline import make_pipeline
from sklearn.compose import make_column_transformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression

prep = make_column_transformer(
    (make_pipeline(SimpleImputer(strategy="median", add_indicator=True),   # (1)
                   StandardScaler()),
     ["log_amount", "account_age_days", "orders_last_1h"]),
    (OneHotEncoder(handle_unknown="ignore"), ["payment", "device_os"]),     # (2)
)
model = make_pipeline(prep, LogisticRegression(max_iter=1000))             # (3)

# model.fit(train_df, y_train)        → medians, means, categories learned from train only
# model.predict_proba(test_df)        → the same training statistics applied to test

Line (1) handles the numeric columns: fill gaps with the median, add a "was missing" flag for each (add_indicator=True), then standardise. Line (2) one-hot encodes the category columns, and handle_unknown="ignore" means a payment method that appears in production but never appeared in training produces all zeros instead of crashing the service. Line (3) joins the preprocessing and the model into one object. Because the medians, means and category lists are learned inside fit, it is impossible to accidentally compute them on the test set. Cross-validation tools in scikit-learn accept the whole pipeline, so every fold gets its own correctly fitted preprocessing too.

A pipeline like this also cures training-serving skew from 12.3.7 for these steps: you save the fitted pipeline and load the very same object in the serving code, instead of re-implementing the preprocessing a second time in another language.

6. How to catch a leak before it ships

Leaks rarely announce themselves, but they leave fingerprints.

• Too good to be true. A jump from 0.62 to 0.97 on a hard problem should trigger suspicion before celebration. Real improvements are usually modest.
• One feature dominates. Run the permutation importance from 12.3.3. If one feature carries most of the model, ask exactly when and by whom its value gets written.
• The feature is not available in the serving code. Before training, write down where each feature will come from at prediction time. A feature with no live source is a feature from the future.
• Offline and online disagree. After launch, compare the live score distribution and the live precision (once labels arrive) with the test results. A big gap is a leak, a skew, or drift, and all three need investigating.
• Shadow the model first. Run the new model alongside the old one in production for a couple of weeks, logging its decisions without acting on them, then compare. This catches nearly every leak that the other checks missed, and Chapter 12.12.4 describes how.

What the interviewer will push on

"What is data leakage? Give examples." They want the defining rule (training sees something prediction will not) and at least three distinct kinds: target leakage, temporal leakage, preprocessing contamination, and group leakage. A memorised single example is the weak answer; a strong one gives an example of each.

"How would you build training data for a model that uses account history?" This checks for the point-in-time join. The tell is mentioning that you need a history of the feature values, not just the current table, and an as-of join that looks only backwards.

"How do you encode a category with ten thousand values?" They want hashing or target encoding, and for target encoding they want the leak: out-of-fold computation and shrinkage for rare categories. Proposing one-hot for ten thousand values without comment is the wrong answer.

"How do you handle missing values?" A good answer asks why the value is missing, explains why missing-not-at-random values carry signal, and adds a missing-indicator column. "Fill with the mean" alone is the weak answer.

Volunteer this: the most reliable leak detector is boring: for every feature, write down which system writes it and at what moment relative to the prediction. Teams that keep that table rarely ship leaks.

Recall

• Feature engineering turns records into numeric columns: log-transform skewed numbers, one-hot encoding for small categories, hashing or target encoding for large ones, cyclical sine and cosine for hours, ratios and windowed counts for history.
• Missing values: ask why they are missing. Missing-not-at-random gaps carry signal, so fill the gap and add a "was missing" column.
• Leakage means training sees what prediction will not. Four forms: target leakage (the answer as a feature), temporal leakage (a feature read at the wrong time), contamination (preprocessing fitted on test data), and group leakage (the same customer on both sides).
• The point-in-time join attaches each feature as it was at the example's moment: merge_asof with direction="backward", on a history of feature values.
• Target encoding leaks through rare categories; compute it out of fold and shrink towards the overall rate. Bundle preprocessing and model into one pipeline. Catch leaks by suspicion, feature importance, a "who writes this and when" table, and shadow runs.

Self-test: Why does "number of past orders" read from today's table leak, even though it is a legitimate feature? With m = 50 and an overall rate of 0.005, what target encoding does a city with 10 orders and 2 frauds get? Why does handle_unknown="ignore" matter in production?

Next: 12.4.3 Imbalance, Sampling and Augmentation — when the thing you care about is half a percent of the data, the model can do well by ignoring it; the next page covers how to make rare cases count, and how to stretch a small dataset further.