12.4.1 — Labels and Annotation

Cartwise wants to route its 40,000 weekly support emails automatically into four teams: refunds, delivery, account and payment. There is no historical label column to learn from, because emails were always read and forwarded by hand. So the team hires six temporary staff, gives them a one-line instruction, "Tag each email with the team that should handle it", and two weeks later has 20,000 labelled emails.

The model trained on them was disappointing, and nobody could work out why. Then somebody had two annotators label the same 200 emails. They agreed on only 71 percent. The disagreements were not careless. Take "My parcel never arrived, I want my money back." Is that refunds (they want money) or delivery (the parcel is lost)? Three annotators had said one, three the other, and each had been perfectly consistent with their own private rule. The model had been trained on two contradictory rules at once, and it learned the confusion faithfully.

A model can never be more consistent than its labels. If humans agree only 71 percent of the time, no amount of modelling will make the model agree with "the truth" much more than that, because the labels do not contain a single truth. This is why experienced teams spend more effort on labels than on algorithms, and why this Part gives labels their own page.

1. Where labels come from

Labels arrive in two very different ways, and each has its own failure.

Natural labels are produced by the world as a side effect of normal business. A dispute arriving says an order was fraud. A delivery scan says how many days a parcel took. A click says a customer found a search result interesting. They are cheap and plentiful, which is wonderful. Their problems:

• Delay. As the fraud case study in 12.3.7 showed, a dispute can take 90 days, so recent data has no label yet.
• They measure something nearby, not the thing itself. A click does not mean the customer liked the product; it might mean the image was misleading. A label that stands in for what you really care about is called a proxy, and models trained on proxies optimise the proxy. A search model trained only on clicks learns to rank eye-catching products, not useful ones.
• They are shaped by the current system. Customers can only click products they were shown, and the current model chose what they were shown. This is the feedback loop from 12.1.4 again: the labels carry the old model's biases into the new one.

Human labels are produced on purpose, by people reading or looking at examples and deciding. The support emails need this, as do product photos ("does this image show the product clearly?") and reviews ("is this review abusive?"). They are slow and expensive, and their quality depends almost entirely on how well the task was defined. The rest of this page is mostly about doing them well.

2. Guidelines: the part everyone skips

The fix for the 71 percent agreement was not more annotators or a better model. It was a better definition. The team rewrote the one-line instruction as a three-page guideline, built around the cases that had caused disagreement. A good guideline has four parts:

1. A definition of each class, in one or two sentences, saying what the team handling it can actually do. "Refunds: the customer wants money back for an order that was delivered. Delivery: anything about a parcel that has not arrived or arrived damaged, even if the customer also asks for money."
2. A rule for overlaps. When an email fits two classes, which wins? "If the parcel has not arrived, it is delivery, whatever else the customer asks for, because the delivery team can issue refunds for lost parcels."
3. Worked examples, especially hard ones. Ten typical emails per class, and then the twenty most confusing emails from the pilot, each with the correct answer and a sentence explaining why.
4. An escape hatch. A fifth label, "unclear", for emails that genuinely do not fit, so annotators are not forced to guess. Those emails are then reviewed by a senior person, and some of them become new worked examples.

With the rewritten guideline, agreement on a fresh set of 200 emails rose to 92 percent, and the model trained on the relabelled data improved more than any modelling change had managed.

3. Measuring agreement: Cohen's kappa

"92 percent agreement" sounds clear, but raw agreement has a hidden flaw: some agreement happens by pure chance. If 90 percent of emails were delivery, two annotators who both tagged everything as delivery without reading would agree 90 percent of the time, while learning nothing. So agreement is measured with a statistic that subtracts chance: Cohen's kappa, introduced by the psychologist Jacob Cohen in 1960.

\kappa = \frac{p_o - p_e}{1 - p_e}

Read it as "kappa equals observed agreement minus chance agreement, divided by one minus chance agreement". Here p_o is how often the two annotators actually agreed, and p_e is how often they would agree if each one labelled at random but with their own usual mix of labels. The bottom of the fraction is the most agreement that was available beyond chance, and the top is how much of it they achieved. So kappa is the fraction of possible above-chance agreement that the annotators actually reached.

A worked example: two annotators, Priya and Tom, each label the same 100 emails as "refund" or "not refund".

	Tom: refund	Tom: not refund	Priya's total
Priya: refund	40	5	45
Priya: not refund	10	45	55
Tom's total	50	50	100

They agreed on 40 + 45 = 85 emails, so p_o = 0.85. Now chance agreement. Priya says "refund" 45 percent of the time and Tom 50 percent. If they labelled independently at those rates, they would both say "refund" 0.45 \times 0.5 = 0.225 of the time, and both say "not refund" 0.55 \times 0.5 = 0.275 of the time, so p_e = 0.225 + 0.275 = 0.5. Then:

\kappa = \frac{0.85 - 0.5}{1 - 0.5} = \frac{0.35}{0.5} = 0.70

So the raw 85 percent agreement becomes a kappa of 0.70: they reached 70 percent of the agreement that was possible beyond chance. As a rough guide, a kappa above about 0.8 means the task is well defined; 0.6 to 0.8 is workable but worth tightening; below about 0.4 means the guidelines or the task itself are too ambiguous, and labelling more data will mostly add noise. (Kappa compares two annotators. With more than two, the usual extension is Fleiss' kappa, after the statistician Joseph Fleiss.)

from sklearn.metrics import cohen_kappa_score

priya = ["R"] * 40 + ["R"] * 5 + ["N"] * 10 + ["N"] * 45
tom   = ["R"] * 40 + ["N"] * 5 + ["R"] * 10 + ["N"] * 45
print(cohen_kappa_score(priya, tom))       # → 0.7

Each list encodes the table: the first 40 emails both said refund, the next 5 Priya said refund and Tom did not, and so on. The function returns the same 0.70.

The agreement rate also gives you a ceiling to aim at. If careful humans agree 92 percent of the time, a model scoring 91 percent against one annotator's labels is about as good as the labels allow. Pushing for 98 percent on that test set means training the model to copy one particular annotator's habits.

4. A workflow that produces good labels

A labelling workflow that produces trustworthy labels. The red loop, fixing the guidelines until agreement is high, is the step most teams skip; the purple loop, letting the model choose what to label next, is what makes labelling affordable.

Walk through it once.

• ① Write the guidelines.
• ② Run a pilot: three people label the same 200 items.
• ③ Measure kappa. If it is low, read the disagreements, fix the guidelines and run the pilot again. This red loop usually runs two or three times.
• ④ Label at full scale. But keep about 10 percent of items labelled by two people, so agreement is always being measured, and you notice if one labeller starts to drift.
• ⑤ Send disagreements to a senior person, who decides. Hard cases become new examples in the guidelines.
• ⑥ Train the model. Set aside a carefully double-checked gold set for measuring it, labelled by your most careful people. The measuring set is the one place where a wrong label directly misleads you.
• ⑦ Let the model choose what to label next, which is section 5.

The tools

You need software to show items to annotators and record their answers. Label Studio is the most widely used open-source tool for this: you run it on your own server, define a labelling interface (choose a class, draw a box on an image, highlight words in text, transcribe audio), load your items, and annotators work in their browser. It supports multiple annotators per item and review steps, which is exactly what the workflow above needs. Because it runs on your own machines, sensitive data like customer emails never leaves the company. Prodigy is a paid alternative built by the makers of the spaCy text library, designed around very fast one-item-at-a-time annotation. For very large jobs, companies use managed labelling services that supply a trained workforce. Whichever you use, the tool is the easy part; the guidelines and the agreement measurement are what decide quality.

What labels cost

Labelling is often the biggest single cost of a project, so it is worth estimating before you start. If an annotator reads and tags one email in 30 seconds, and their time costs ₹400 an hour, each label costs about ₹3.30. Twenty thousand emails, with 10 percent labelled twice, is 22,000 labels, about ₹73,000, plus the senior reviewer's time for adjudication. Drawing boxes around products in photos might take two minutes each, which at the same rate is about ₹13 per image. Numbers like these decide whether you can afford the label set you want, and they are the reason for the next two sections.

5. Labelling less: active learning

Most randomly chosen emails are easy. "Where is my parcel? Order 4471." is obviously delivery. Once the model has seen a few hundred of those, labelling another thousand teaches it almost nothing. The emails that teach it something are the ones it is unsure about.

Active learning uses that fact. Label a small starting set, train a model, run it over all the unlabelled emails, and send humans the ones where the model is least certain. Retrain, and repeat. The simplest measure of uncertainty for a yes-or-no question is how close the predicted probability is to 0.5:

import numpy as np

emails = np.array(["where is order 4471", "parcel never came, money back",
                   "refund for broken kettle", "card charged twice",
                   "cannot log in", "returned item, no refund yet"])
p_refund = np.array([0.03, 0.52, 0.96, 0.41, 0.02, 0.61])   # current model's guesses

uncertainty = 1 - np.abs(p_refund - 0.5) * 2                # (1) 1 = coin flip, 0 = certain
order = np.argsort(-uncertainty)                            # (2) most uncertain first
print(emails[order[:3]])
# → ['parcel never came, money back' 'card charged twice' 'returned item, no refund yet']

Line (1) turns each probability into an uncertainty score: a probability of 0.5 gives 1 (the model is guessing), and 0 or 1 gives 0 (the model is sure). The 0.52 email scores 0.96, the 0.41 email scores 0.82, and the 0.61 email scores 0.78. Line (2) sorts from most to least uncertain. The three emails it picks are exactly the borderline ones: the lost-parcel-with-refund case that broke the original labels, a double charge that might be payment or refund, and a returned item. The obvious ones at 0.03, 0.02 and 0.96 are skipped.

In practice, active learning often reaches the same accuracy with a fraction of the labels, though how much it saves varies a lot by problem. Two cautions. First, the most uncertain items are often genuinely ambiguous, so they are slower and more contentious to label; plan for that. Second, a model trained only on hard cases can get a distorted view of what typical data looks like, so mix in a share of randomly chosen items with every batch. And as always, the test set must still be a random sample, never chosen by the model, or it will no longer represent the real stream of emails.

6. Labelling faster: models and rules that propose labels

Model-assisted labelling. Instead of a blank form, show the annotator the model's guess and ask them to correct it. This is much faster, often several times faster for complex tasks like drawing boxes. It has one serious danger, called automation bias: people tend to accept a plausible suggestion without checking it properly, especially when tired, so the model's mistakes get copied into the labels and then learned back by the next model. The defence is to measure it. Track how often annotators change the suggestion. If that rate drops close to zero, they are probably rubber-stamping. Some teams also slip in items where the suggestion is deliberately wrong, to check that people catch them.

Weak supervision. Sometimes you can write rough rules that are often right: "if the email mentions a tracking number, it is probably delivery"; "if it contains 'charged twice', it is probably payment". Each such rule is called a labelling function. Individually they are noisy and many emails match none, but a statistical model can combine dozens of them, learning how accurate each one is from where they agree and disagree, and produce a probable label for every email. This approach was made popular by a Stanford research project and open-source tool called Snorkel. It is useful when you need millions of labels and can express much of the knowledge as rules, and it produces labels good enough to train a model that then generalises beyond the rules.

LLMs as labellers. Today the most common way to get a first set of labels is to give a capable LLM the guidelines from section 2 and ask it to label the emails. With good guidelines, LLM labels on a task like email routing often agree with careful human labels about as well as two humans agree with each other. That makes a new workflow possible: the LLM labels everything, humans review a sample and every case where the LLM was unsure or two LLM runs disagreed, kappa between the LLM and the human gold set is measured exactly like kappa between two people, and a small, cheap classifier is then trained on the result to do the routing at scale. The same rules apply as for any annotator: an LLM labeller needs clear guidelines, its agreement must be measured rather than assumed, and it must never label the test set you use to judge it. Chapter 12.8.6 treats the closely related question of using an LLM to judge other models' outputs.

7. When the labels themselves are wrong

Even with good process, some labels will be wrong. A useful rule of thumb from studies of public datasets is that a few percent of labels in carefully built datasets turn out to be mistakes. Three practical habits deal with this.

Look at the model's most confident mistakes. After training, find the examples where the model is very sure and disagrees with the label. A good share of those are label errors, not model errors. Re-checking just those items is one of the cheapest ways to clean a dataset.

Clean the test set first. Noisy training labels mostly add a little noise to the model. Noisy test labels corrupt every decision you make, because you are choosing models by how well they match the errors.

Keep the label history. Record who labelled each item, when, and under which version of the guidelines. When the guidelines change, as they did here when lost-parcel emails moved from refunds to delivery, you need to know which old labels were made under the old rule so you can relabel them. That is part of data versioning, covered in Chapter 12.4.4.

Recall

• A model cannot be more consistent than its labels. Two annotators agreeing only 71 percent of the time puts a ceiling on the model.
• Natural labels are cheap but delayed, often proxies, and shaped by the current system. Human labels depend on guidelines: class definitions, overlap rules, hard worked examples, and an "unclear" escape hatch.
• Cohen's kappa = (p_o - p_e)/(1 - p_e) removes chance agreement: 85 percent raw agreement became kappa 0.70. Aim above about 0.8; fix guidelines when it is low.
• The workflow: guidelines, pilot, measure, fix, label at scale with 10 percent overlap, adjudicate, keep a gold test set. Tools like Label Studio host the work; labels cost real money per item.
• Active learning labels what the model is least sure about. Model-assisted labelling is faster but risks automation bias; weak supervision combines noisy rules; LLM labellers need guidelines and measured agreement too. Check the model's confident mistakes for label errors, and clean the test set first.

Self-test: If two annotators agree 90 percent of the time and each labels 90 percent of items as "delivery", what is kappa, and what does it tell you? Why should the test set never be chosen by active learning? What signal tells you annotators have started rubber-stamping model suggestions?

Next: 12.4.2 Features and Leakage — with trustworthy labels in hand, the next question is what the model gets to look at, and how a single careless column can make a useless model look brilliant.