Appearance
12.3.6 — Learning Without Labels
Every model so far learned from the right answers: past orders labelled fraud or genuine, deliveries with their actual days. But much of the data Cartwise holds has no labels at all, and some of the most useful questions do not have right answers waiting to be learned:
- Marketing asks: "What kinds of customers do we have?" Nobody has labelled customers by type. There may not even be "true" types.
- The data team asks: "We track 300 behaviours per customer. Can we describe each customer with far fewer numbers without losing much?"
- The fraud team asks: "Is there a new kind of fraud happening that our model has never been trained on?" By definition, nothing is labelled with the new kind yet.
These are the three main jobs of unsupervised learning from 12.1.2: clustering finds groups, dimensionality reduction finds simpler descriptions, and anomaly detection finds the cases that do not fit. They are harder to judge than supervised models, because there is no test set of right answers to score against, and that difficulty runs through this whole page.
1. Clustering with k-means
Describe six Cartwise customers by two numbers: how many orders they place per month, and their average basket value in thousands of rupees.
| Customer | P1 | P2 | P3 | P4 | P5 | P6 |
|---|---|---|---|---|---|---|
| Orders per month | 1 | 2 | 1.5 | 8 | 9 | 10 |
| Average basket (₹000) | 8 | 9 | 7 | 1 | 2 | 1.5 |
Looking at the table, you can probably see two groups: P1 to P3 order rarely and spend a lot (perhaps they buy electronics), and P4 to P6 order often and spend a little (perhaps groceries). k-means is an algorithm that finds groups like this automatically, and it is the most widely used clustering method. You tell it how many groups to find, k, and it repeats two simple steps.
- Assign: put every point in the group whose centre is nearest.
- Update: move each group's centre to the average position of the points now in it.
Repeat until nothing changes. The centre of a group is called its centroid, which just means "average point". The name k-means says what it does: it finds k means.
Watching it run
Let us run it with k = 2, and deliberately start badly, with the two centroids placed on P1 (1, 8) and P2 (2, 9), both inside the same natural group.
Round 1, assign. Each customer joins the nearer centroid. P1 and P3 are nearest the blue centroid at P1. P2 is the amber centroid itself. The surprise is the far-away customers: P4 at (8, 1) is a distance of \sqrt{7^2 + 7^2} \approx 9.90 from blue and \sqrt{6^2 + 8^2} = 10 from amber, so it joins blue by a whisker, while P5 and P6 are slightly nearer amber. Panel ① shows the mess: each group has a mix of both kinds of customer.
Round 1, update. Move each centroid to the average of its members. Blue has P1, P3 and P4: the average is \left(\frac{1 + 1.5 + 8}{3}, \frac{8 + 7 + 1}{3}\right) = (3.5, 5.33). Amber has P2, P5 and P6: \left(\frac{2 + 9 + 10}{3}, \frac{9 + 2 + 1.5}{3}\right) = (7, 4.17). Both centroids have moved to the middle, as in panel ②.
Round 2, assign. Now P1, P2 and P3 are all nearer blue, and P4, P5 and P6 are all nearer amber. For example, P2 at (2, 9) has squared distance 1.5^2 + 3.67^2 \approx 15.7 to blue and 5^2 + 4.83^2 \approx 48.4 to amber. The groups are now correct.
Round 2, update. Blue moves to (1.5, 8) and amber to (9, 1.5), the centres of the two natural groups. Round 3 changes no assignment, so the algorithm stops, as in panel ③.
python
import numpy as np
from sklearn.cluster import KMeans
X = np.array([[1, 8], [2, 9], [1.5, 7], [8, 1], [9, 2], [10, 1.5]])
km = KMeans(n_clusters=2, n_init=10, random_state=0).fit(X) # (1)
print(km.labels_) # (2) → e.g. [1 1 1 0 0 0]
print(km.cluster_centers_) # → [[9. 1.5] [1.5 8. ]] (order may differ)
print(round(km.inertia_, 2)) # (3) → 5.0Line (1) runs k-means. n_init=10 runs it ten times from different random starting centroids and keeps the best result, because, as our bad start showed, the outcome can depend on where you begin, and on harder data a bad start can get stuck in a poor grouping. Line (2) gives each customer's group number. The numbers are arbitrary labels: whether the big spenders are "0" or "1" means nothing. Line (3) is the inertia: the total squared distance from each customer to its centroid, the quantity k-means tries to make small. Here it is 5.0. You can check it by hand: in the big-basket group, with its centroid at (1.5, 8), P1 is a squared distance of 0.5^2 = 0.25 away, P2 is 0.5^2 + 1^2 = 1.25, and P3 is 1^2 = 1, which adds to 2.5. The other group contributes another 2.5 in the same way.
Choosing k, and what k-means cannot do
You have to choose k, and the data does not tell you directly. The common trick is the elbow method: run k-means for k = 1, 2, \ldots, 10 and plot the inertia. It always falls as k grows, since more centroids means every point is closer to one, reaching zero when every customer has a centroid of its own. But it usually falls steeply at first and then flattens. The bend, the "elbow", is a reasonable choice: past it, extra groups buy very little. In practice the more important test is whether the groups are useful: can marketing describe each one in a sentence and treat it differently? Three clear groups that drive three different campaigns beat seven that nobody can tell apart.
That points at the real limitation of all clustering: the algorithm returns groups, not meanings. It says "group 0" and "group 1". A person has to look at each group's centroid and members and decide what it is, and sometimes the honest answer is "nothing interesting". Clustering is a tool for exploring and summarising data, not a source of truth.
k-means also assumes something about the shape of groups: it measures straight-line distance to a centre, so it finds round blobs of similar size. Give it two crescent-shaped groups, or one tight group next to one sprawling group, and it will cut them in the wrong place. Two other methods are worth knowing by name for those cases. Hierarchical clustering starts with every customer in its own group and repeatedly merges the two closest groups, building a tree you can cut at any level, so you can look at the structure before choosing how many groups you want. DBSCAN (density-based spatial clustering of applications with noise) grows groups outward through densely packed regions, finds groups of any shape, works out the number of groups by itself, and marks isolated points as noise, which also makes it useful for the anomaly detection in section 3. And as with k-nearest neighbours in 12.3.5, every distance-based method needs its features standardised first.
2. Dimensionality reduction with PCA
Cartwise tracks 300 behaviours per customer. Many of them say nearly the same thing: number of items bought, total spend, number of orders, and days active all rise together for a heavy shopper. Feeding 300 overlapping numbers into a model makes it slower and, as the curse of dimensionality in 12.3.5 showed, makes distances less meaningful. Can we describe each customer with, say, 20 numbers that keep most of the information?
Principal component analysis, or PCA, does exactly that. It finds new axes, each a blend of the original features, ordered by how much the data spreads out along them. The first axis is the single direction of greatest spread. The second is the direction of greatest remaining spread at a right angle to the first, and so on. These new axes are the principal components. Keep the first few, drop the rest, and you have a shorter description that keeps most of the variation. The method goes back to the statistician Karl Pearson in 1901 and was developed further by Harold Hotelling in the 1930s.
A worked example in two dimensions
Take six customers described by two correlated features: average items per order and average basket value in thousands of rupees.
| Customer | 1 | 2 | 3 | 4 | 5 | 6 |
|---|---|---|---|---|---|---|
| Items per order | 1 | 2 | 3 | 4 | 5 | 6 |
| Basket (₹000) | 1.2 | 1.9 | 3.2 | 3.8 | 5.1 | 5.8 |
The points lie almost on a diagonal line, because customers who buy more items per order also spend more. PCA finds that diagonal. Working it out (the method is below), the first principal component points in the direction (0.72, 0.69): roughly equal parts of each feature, which you can read as "overall basket size". Along it, the data has a variance of about 6.66. Across it, in the second direction, the variance is only about 0.014. So the first component carries 6.66 / (6.66 + 0.014) \approx 99.8 percent of all the spread. That fraction is called the explained variance ratio. Projecting each customer onto PC1 gives one number each: -3.40, -2.19, -0.57, 0.57, 2.19, 3.40. Two columns have become one, and almost nothing has been lost.
How are the directions found? First, subtract each feature's average so the data is centred on zero. Then compute the covariance matrix, a small square table whose entries say how much each pair of features varies together; here it is about \begin{bmatrix} 3.5 & 3.32 \\ 3.32 & 3.18 \end{bmatrix}, and the large off-diagonal 3.32 is the strong link between the two features. The principal components are the special directions of this matrix, called its eigenvectors: directions that the matrix only stretches without turning, as in the matrix geometry of 12.2.2. How much each one is stretched, its eigenvalue, is exactly the variance along it: 6.66 and 0.014 here. Volume II's chapter on eigenvalues and SVD derives why, if you want the full proof. In practice, libraries compute PCA with a closely related factorisation called the singular value decomposition, which is more numerically stable.
python
import numpy as np
from sklearn.decomposition import PCA
X = np.array([[1, 1.2], [2, 1.9], [3, 3.2], [4, 3.8], [5, 5.1], [6, 5.8]])
pca = PCA(n_components=2).fit(X) # (1)
print(pca.explained_variance_ratio_.round(3)) # (2) → [0.998 0.002]
print(pca.components_[0].round(2)) # (3) → [0.72 0.69] (sign may flip)
print(PCA(n_components=1).fit_transform(X).ravel().round(2))
# → [-3.4 -2.19 -0.57 0.57 2.19 3.4 ] (sign may flip)Line (1) fits PCA; it centres the data itself. Line (2) shows the explained variance ratio, the same 99.8 percent. Line (3) shows the first direction. The note "sign may flip" matters: (0.72, 0.69) and (-0.72, -0.69) describe the same line, and libraries may return either, so never write code that depends on the sign of a component.
What PCA is for, and where it misleads
With 300 features, you look at the explained variance ratios and keep enough components to reach, say, 95 percent. Often that is far fewer than 300, because real features overlap heavily. Uses: compressing features before a slow algorithm, plotting high-dimensional data in two dimensions to look at it, and removing noise that lives in the many small components.
Two cautions. First, always standardise before PCA unless the features already share units. Otherwise "spend in rupees", with its huge numbers, has by far the most variance and becomes the first component simply because of its units. Second, spread is not the same as usefulness. PCA keeps the directions where the data varies most, which is not necessarily where fraud and genuine orders differ. A tiny component can hold exactly the signal you care about, and PCA would throw it away. And because each component is a blend of every original feature, the new columns are much harder to explain to anyone.
PCA finds flat directions. Modern deep learning learns curved, far richer compressions of data, called embeddings, and that idea, compressing something into a short vector that keeps what matters, is the subject of Chapter 12.6.
3. Anomaly detection: finding what does not fit
In October, Cartwise's fraud model, trained on past fraud, was missing a new trick: small orders of gift cards, placed in bursts by accounts that had been dormant for a year. The model had never seen this pattern labelled, so it had nothing to learn from. What the team could see was that these orders looked unusual. Anomaly detection finds examples that differ from the bulk of the data, without needing labels for what the differences mean.
The simplest baseline: how many spreads from normal?
For a single feature, the standardisation from 12.2.4 already gives a detector. Compute each order's z-score, z = (x - \text{mean}) / \text{standard deviation}: how many spreads it sits from average. From the normal distribution in 12.2.5, only about 0.3 percent of values lie more than three spreads away, so |z| > 3 is a common first alarm.
It is a useful baseline and it fails in two predictable ways. Many real quantities, like order amounts, are skewed with long tails rather than bell-shaped, so the alarm fires constantly on perfectly normal big orders. And it looks at one feature at a time, so it misses combinations that are each normal on their own but strange together: a normal amount, from a normal account age, at a normal time, but all three at once in a way never seen before.
Isolation forest: unusual points are easy to separate
The isolation forest, introduced by Fei Tony Liu, Kai Ming Ting and Zhi-Hua Zhou in 2008, turns the idea on its head. Instead of describing what normal looks like, it measures how easy each point is to isolate.
Here is the intuition with seven order amounts, in thousands: 10, 11, 12, 12, 13, 14 and 95. Pick a random cut point somewhere between the smallest (10) and the largest (95). Any cut above 14 separates the ₹95,000 order from everything else in a single step. That happens for any cut between 14 and 95, which is 81/85 \approx 95 percent of the possible cuts. To isolate one of the ₹12,000 orders, by contrast, you need several lucky cuts that slice closely on both sides of it. Normal points sit in crowds and take many random cuts to isolate. Anomalies sit alone and fall out in one or two.
The algorithm builds many random trees, each one choosing a random feature and a random cut at every step, splitting until each point stands alone. A point's anomaly score is based on how many cuts, on average across the trees, it took to isolate it: a short average path means anomalous. Because each cut uses a different random feature, it naturally catches unusual combinations across features, not just extreme values on one.
python
from sklearn.ensemble import IsolationForest
import numpy as np
amounts = np.array([[10], [11], [12], [12], [13], [14], [95]])
iso = IsolationForest(n_estimators=200, contamination=0.15,
random_state=0).fit(amounts) # (1)
print(iso.predict(amounts)) # (2) → [ 1 1 1 1 1 1 -1]Line (1) builds 200 random isolation trees. contamination is your estimate of what fraction of the data is anomalous, and it sets the cutoff: here, roughly the most isolated 15 percent. Line (2) labels each point: 1 for normal, −1 for anomaly. The ₹95,000 order is the only one flagged. In real use you would pass many features at once, and choose the cutoff by how many cases the review team can look at each day.
Another approach worth knowing by name is the one-class SVM, which uses the margin and kernel ideas of 12.3.5 to draw a tight boundary around the normal data and flags anything outside it. And the k-nearest-neighbour idea gives a third: a point whose nearest neighbours are all far away is probably unusual.
The hard part: an anomaly is not the same as fraud
An anomaly detector finds unusual orders, and unusual is not the same as bad. A customer buying gifts for a wedding, a business placing its first bulk order, a new product launch that brings a wave of first-time buyers: all are anomalous, and all are genuine. So anomaly detection in production is almost never a blocking decision on its own. It is a way to point human attention at the strange cases, so that people can discover new fraud patterns, label them, and feed those labels back into the supervised model, which closes the loop from 12.1.4.
Judging an anomaly detector has the same problem as judging clustering: there are no labels to score against, by construction. The usual practice is to have analysts review the top-scoring cases each week and record what fraction turned out to be interesting. That fraction is the detector's real precision, and it is what decides whether the detector stays switched on.
Recall
- Unsupervised learning finds structure without labels: groups, simpler descriptions, unusual cases. With no right answers, judging the result is the hard part.
- k-means alternates assigning points to the nearest centroid and moving centroids to their members' average, until nothing changes. It needs k (elbow method), several random starts, standardised features, and round groups. Groups have numbers, not meanings.
- Principal component analysis finds new axes ordered by spread; the principal components are the covariance matrix's eigenvectors, and the explained variance ratio says how much each keeps. Two linked features became one with 99.8 percent kept. Standardise first; high spread is not the same as useful.
- Anomaly detection flags what does not fit. A z-score beyond 3 is the baseline; an isolation forest scores points by how few random cuts isolate them, catching unusual combinations.
- Unusual is not the same as bad: anomaly scores direct human review, whose labels then improve the supervised model.
Self-test: If the k-means example had started with centroids on P1 and P4, how many rounds would it need? Why must amounts in rupees and account age in years be standardised before PCA? In the isolation example, roughly what fraction of random cuts would isolate the ₹10,000 order in one step?
Next: 12.3.7 Case Study: Fraud Detection — every tool from 12.2 and 12.3 is now on the table; the case study puts them together into Cartwise's real fraud system, from raw orders to a threshold the reviewers can live with.