Appearance
12.4.3 — Imbalance, Sampling and Augmentation
Two of Cartwise's data problems look different but have the same root.
The first: fraud is half a percent of orders. In a training set of 2 million orders there are 10,000 frauds and 1,990,000 genuine orders. When gradient descent from 12.2.4 adds up the loss, 99.5 percent of the terms come from genuine orders. The cheapest way to make the total loss small is to get the genuine orders very right and barely bother with fraud. The model drifts towards "everything is genuine", the useless 99.5-percent-accurate model from 12.3.2.
The second: the support router from 12.4.1 has a fifth team, legal, for emails threatening lawsuits or reporting safety hazards. Only 40 such emails exist in the labelled set. There is simply not enough of them for any model to learn what they look like.
Both are problems of too few examples of the thing that matters. The first is called class imbalance. The second is scarcity. This page covers the tools for each, in the order you should reach for them, because the most commonly suggested tool, generating synthetic examples, is usually the last one you need.
1. First, check whether imbalance is actually your problem
Very often a team says "our model suffers from class imbalance" when what is really wrong is the measurement or the threshold. Two checks come first, before touching the data at all.
Are you measuring with the right metric? If accuracy or ROC-AUC is the headline number, the model may be fine and the report misleading. Switch to the precision–recall curve and recall at your operating capacity, as 12.3.2 explained.
Are you using the right threshold? A model trained on imbalanced data, if it is reasonably calibrated, will give fraud low probabilities, because fraud really is rare. A 3 percent fraud probability is very suspicious when the base rate is 0.5 percent: it is six times the normal risk. If you are still cutting at 0.5, the model will flag almost nothing, and it will look as if it never learned fraud. Lower the threshold using the cost reasoning of 12.3.2 and the same model may be perfectly good.
Strong models such as gradient-boosted trees, trained on enough data, often handle imbalance well on their own once these two things are fixed. Only if the model still ranks fraud poorly, meaning the precision–recall curve itself is weak, do you need the tools below.
2. Class weights: make rare mistakes cost more
The least invasive fix changes only the loss. Give each example a weight, and multiply its contribution to the loss by that weight. If each fraud counts as much as 200 genuine orders, the 10,000 frauds weigh about as much in total as the 1,990,000 genuine orders, and the model can no longer ignore them.
scikit-learn's class_weight="balanced" picks the weights for you, with the formula
w_c = \frac{n}{k \cdot n_c}
where n is the total number of examples, k the number of classes, and n_c the number in class c. For Cartwise: genuine gets \frac{2{,}000{,}000}{2 \times 1{,}990{,}000} \approx 0.503 and fraud gets \frac{2{,}000{,}000}{2 \times 10{,}000} = 100. Fraud mistakes now count about 199 times as much as genuine ones, exactly enough to make the two classes weigh the same in total.
python
import numpy as np
from sklearn.utils.class_weight import compute_class_weight
y = np.array([0] * 1_990_000 + [1] * 10_000)
print(compute_class_weight("balanced", classes=np.array([0, 1]), y=y).round(3))
# → [ 0.503 100. ]"Balanced" is a starting point, not a law. The fraud case study in 12.3.7 used a weight of 20, found on the validation set, because full balancing over-corrected and produced a model that flagged too eagerly. Treat the weight as a hyperparameter.
The one cost of weighting, which the case study also met, is calibration. The model has been told fraud matters 20 or 100 times more than it really occurs, so its probabilities come out too high. Rankings stay good; the numbers need correcting with a calibration step on unweighted validation data before they can be used as money.
3. Resampling: change the data instead of the loss
Resampling changes how many examples of each class the model sees.
Undersampling keeps every fraud and a random fraction of the genuine orders. Keep 5 percent of the genuine orders and you have 10,000 frauds against 99,500 genuine, about 1 in 11, and a dataset twenty times smaller. That is its real strength: with billions of rows, as in advertising click prediction, training on everything is too slow, and dropping most of the boring examples costs little accuracy. The obvious cost is that you throw information away. A good compromise is to drop only the easy genuine orders, but random undersampling is the common default.
Oversampling repeats the fraud examples, perhaps ten times each. It throws nothing away, but the model sees the identical rows again and again and can memorise them, which is the overfitting of 12.3.3 in its most direct form. For gradient-based models, repeating an example ten times is mathematically almost the same as giving it a weight of ten, so class weights usually achieve the same thing more cheaply.
SMOTE, the synthetic minority oversampling technique, published by Nitesh Chawla and colleagues in 2002, tries to oversample without exact repetition. For each rare example, find a few of its nearest rare neighbours (using the distance ideas of 12.3.5), pick one, and create a new example at a random point on the straight line between them:
\mathbf{x}_{\text{new}} = \mathbf{x} + \lambda\,(\mathbf{x}_{\text{neighbour}} - \mathbf{x}), \qquad \lambda \text{ random between 0 and 1}
For two fraud orders described by (amount in ₹10,000s, share of easily resold goods), \mathbf{x} = (2.0, 0.8) and its neighbour (3.0, 0.6), with \lambda = 0.4: the new example is (2.0 + 0.4 \times 1.0,\ 0.8 + 0.4 \times (-0.2)) = (2.4, 0.72), an order of ₹24,000 with 72 percent resellable goods, plausibly between the two.
python
import numpy as np
x, neighbour, lam = np.array([2.0, 0.8]), np.array([3.0, 0.6]), 0.4
print(x + lam * (neighbour - x)) # → [2.4 0.72]SMOTE is famous and often recommended, but use it with care. The invented points are guesses: a straight line between two frauds may pass through a region full of genuine orders, so SMOTE can create "fraud" right where genuine customers live. It does not work for categories (what is halfway between upi and card?). And in published comparisons, with strong models like boosted trees, it frequently gives little benefit over simple class weights. Try it, measure it on an untouched validation set, and keep it only if it wins.
The rule that keeps resampling honest
Resample only the training data. The validation and test sets must keep the real-world proportions, because they exist to tell you how the model will behave on the real stream of orders. A test set that has been rebalanced to 50 percent fraud will report a precision far higher than anything you will see in production. And when resampling inside cross-validation, resample inside each fold, after the split, never before, or copies of the same fraud end up on both sides of the split.
Fixing the probabilities after undersampling
Undersampling has a predictable effect on probabilities, and unlike most calibration problems, it has an exact correction. If you keep only a fraction r of the genuine orders, every genuine order in the training data stands for 1/r real ones, so the model sees fraud as 1/r times more common than it really is. Its odds of fraud, from 12.2.6, come out 1/r times too high. To undo it, multiply the odds by r. Writing p' for the model's probability and p for the corrected one:
\frac{p}{1 - p} = r \cdot \frac{p'}{1 - p'} \quad\Longrightarrow\quad p = \frac{r\,p'}{r\,p' + (1 - p')}
With r = 0.05 (we kept 5 percent of genuine orders) and a model output of p' = 0.5, the corrected probability is \frac{0.05 \times 0.5}{0.05 \times 0.5 + 0.5} = \frac{0.025}{0.525} \approx 0.048. The model's "coin flip" is really about a 5 percent chance, which matches intuition: in the undersampled data, fraud was about one in eleven; in reality it is one in two hundred.
python
def correct_for_undersampling(p_model, keep_rate):
return keep_rate * p_model / (keep_rate * p_model + (1 - p_model))
print(round(correct_for_undersampling(0.5, 0.05), 4)) # → 0.0476This is Bayes' rule from 12.2.5 at work: the model learned the likelihoods correctly but with the wrong prior, and the formula swaps in the right prior.
4. Stratified sampling: make sure the rare class shows up
Even without rebalancing, splitting imbalanced data randomly is risky. A test set of 10,000 orders drawn at random from data with 0.5 percent fraud will contain about 50 frauds on average, but any particular draw might contain 38 or 63. With validation folds of a few thousand rows, some folds might contain a handful of frauds, and the scores from fold to fold jump around for no real reason.
Stratified sampling fixes that by sampling each class separately, so every split has exactly the overall proportion. "Stratum" is Latin for "layer": the data is divided into layers by class, and each layer is sampled in proportion.
python
from sklearn.model_selection import train_test_split, StratifiedKFold
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2,
stratify=y, random_state=0) # (1)
folds = StratifiedKFold(n_splits=5, shuffle=True, random_state=0) # (2)Line (1) splits off a test set whose fraud rate matches the full data exactly. Line (2) does the same for every fold of cross-validation. For time-ordered data like fraud, the time-based split of 12.3.3 still comes first; within a time window, stratification is how you sample.
5. Scarcity: the 40 legal emails
Weights and resampling rebalance what you have. With only 40 legal emails, the deeper problem is that there is not enough variety for a model to learn from, however you weight them. Three tools help, in this order.
Collect more real examples. The best fix, and often cheaper than it sounds. Search the unlabelled inbox for emails containing "lawyer", "court", "consumer forum" or "fire", have someone label the matches, and you may find another 300 within a day. This is targeted labelling, a close relative of the active learning in 12.4.1. No synthetic trick is worth as much as real examples.
Use a model that already knows language. A model pre-trained on huge amounts of text already knows what a legal threat sounds like, so it needs only a few examples to learn the category. This is the transfer learning from 12.1.2, and for text and images it is by far the most effective answer to scarcity. An LLM asked "does this email threaten legal action or report a safety hazard?" may need no legal examples at all, and the 40 you have become a test set to check it.
Augment. Create new training examples by changing existing ones in ways that do not change the label. This is data augmentation, and it is the next section.
6. Data augmentation: variations that keep the label
The idea behind augmentation is to teach the model what does not matter. A photo of a kettle is still a kettle if it is flipped left to right, slightly rotated, cropped a little, or a bit darker. Showing the model those variations teaches it to ignore them, a property called invariance, and it effectively multiplies the dataset.
Images are where augmentation works best and is used almost always. During training, every time a product photo is used, it is randomly flipped, cropped, rotated a few degrees, and has its brightness and colour nudged, so the model never sees exactly the same picture twice. Chapter 12.5.5 shows it in code.
Text is harder, because small changes can change the meaning. Common methods:
- Back-translation: translate the email into another language and back, which rewords it while keeping the meaning: "I will take you to consumer court" might come back as "I am going to take you to the consumer court".
- Synonym replacement: swap some words for synonyms. Cheap, but it easily produces unnatural text.
- Paraphrasing with an LLM: ask a model to rewrite each legal email five different ways, or to write new emails in the same style. Today this is the most common approach. It is also the riskiest, because LLM-written text tends to be more polished and more similar to itself than real customer emails, so a model trained mostly on it can do well on synthetic examples and worse on real ones. Keep the test set entirely real, and measure whether the synthetic data helps on it.
Tables are the hardest. There is no "flip" for an order. SMOTE from section 3 is the main tool, with the caveats given there.
Two rules apply to every kind of augmentation. First, it must preserve the label. Flipping a photo of a shoe is fine; flipping a photo of a street sign turns readable text into mirror writing, and rotating a photo of the digit 6 by 180 degrees makes it a 9. Before adding any transformation, ask whether a careful human would still give the changed example the same label. Second, augment only the training data, for the same reason as resampling: the test set must look like the real world.
Recall
- With class imbalance, the loss is dominated by the common class and the model can learn to ignore the rare one. First check the metric (precision–recall, not accuracy) and the threshold; a strong model may already be fine.
- Class weights scale each example's loss; "balanced" uses n/(k\,n_c), giving fraud about 199 times genuine's weight here. Tune the weight, and recalibrate afterwards.
- Undersampling discards common examples (fast, loses information); oversampling repeats rare ones (risks memorising); SMOTE interpolates between rare neighbours (may invent fraud in genuine territory). Resample only training data, inside each fold. After undersampling at rate r, correct with p = r p' / (r p' + 1 - p').
- Stratified sampling keeps each split's class proportions equal to the whole.
- For scarcity, collect targeted real examples first, then use pre-trained models, then data augmentation: transformations that preserve the label (image flips and crops, back-translation, LLM paraphrase), applied to training data only.
Self-test: A model trained with genuine orders undersampled to 10 percent outputs 0.2 for an order. What is the corrected probability? Why is a rebalanced test set misleading? Name an image augmentation that would destroy the label for one Cartwise task and keep it for another.
Next: 12.4.4 Pipelines, Validation and Feature Stores — every step in this chapter has to run again every week, automatically and identically; the last data page is about the pipelines, checks and stores that make that possible.