12.10 — Bias, Safety and What Engineers Have to Get Right

A hiring screen is trained on ten years of applications and which candidates were hired. It learns to downrank CVs mentioning a women's college, and CVs with a two-year gap.

Nobody put gender in the model. The model learned from decisions that were made by people, and it reproduced their pattern with more consistency and more scale than any individual could. Removing the protected attribute did not remove the bias, because dozens of other fields carry the same information.

That is the central practical fact of this chapter: a model trained on historical outcomes learns the historical process, including the parts of it nobody would defend.

1. Where bias comes from

Historical bias. The data faithfully records a world with unequal outcomes. The model is accurate and the outcome is unjust, and accuracy is not a defence.

Representation bias. Groups under-represented in training data get worse performance — the speech-recognition gap in Chapter 12.7, and facial analysis systems with error rates an order of magnitude higher for some groups than others.

Label bias. The labels encode a judgement. "Was hired" is not "was qualified"; "was arrested" is not "committed a crime". The target variable is frequently a proxy for what you actually mean, and its gap from the real quantity is where bias enters.

Proxy variables. Postcode correlates with ethnicity. Name correlates with gender and ethnicity. Purchase history correlates with almost everything. This is why deleting the protected attribute does not work — it removes your ability to measure the disparity while leaving the disparity in place, which is strictly worse.

Feedback loops. Predict crime in an area, police it more, record more crime there, confirm the prediction. The model shapes the data that trains its successor (Chapter 12.9's warning, with a social consequence).

Measurement bias. A feature means different things for different groups. "Number of prior support contacts" measures dissatisfaction for one population and access to a phone during working hours for another.

2. Fairness definitions conflict, and you must choose

The uncomfortable and important result: the common fairness criteria are mathematically incompatible except in degenerate cases.

Demographic parity — the positive rate is equal across groups. The same share of each group is approved.

Equalised odds — true positive and false positive rates are equal across groups. The model is equally accurate for everyone.

Calibration — a predicted 70% means 70% for every group. The score means the same thing regardless of who you are.

If the underlying base rates differ between groups, you cannot have all three. This was proved in the debate over criminal risk assessment: one side argued the tool was biased because false positive rates differed by group; the other argued it was fair because it was calibrated. Both were correct on their own definition, and the definitions cannot both hold.

So fairness is not a property you verify; it is a choice you must make explicitly and write down. Choose demographic parity when the goal is representation in an outcome. Choose equalised odds when errors are the harm. Choose calibration when the score is consumed as a probability by a downstream decision. A team that has not made this choice has made it implicitly by default.

And measure by group, always. Disaggregated evaluation — accuracy, precision, recall and calibration per group — is the minimum. A single average is the most common way a disparity stays invisible. Report confidence intervals: small groups have noisy metrics, and a 20-person subgroup's accuracy is close to meaningless.

3. Mitigations, and their limits

Pre-processing — reweight or resample the training data, or remove the most biased features. Simple, and it cannot fix labels that encode the bias.

In-processing — add a fairness constraint or penalty to the training objective. Effective, and it costs some accuracy, which is a real trade to state rather than hide.

Post-processing — set different decision thresholds per group to equalise a chosen metric. Technically simple, effective, and it may be unlawful in your jurisdiction, because it is an explicit group-based decision rule. Take legal advice before shipping it.

And the non-technical measures that usually matter more:

Better data. Collect representative data. Expensive, slow, and it addresses the cause.

Narrow the scope. A model that ranks candidates for a human to review is a different risk from one that rejects them. Reducing what the model decides is the most reliable mitigation available.

Human review with real authority, and a genuine appeal route where a person can override with a recorded reason.

Document the limits in a model card (Chapter 12.9), including who it was not validated for.

4. The failure modes specific to language models

Hallucination is structural, not a bug to be patched. A language model is trained to produce likely continuations. It has no separate representation of "things I know" versus "things that are plausible", and a fluent invented citation is exactly as likely as a real one when the real one was never learned.

Four things genuinely reduce it:

• Retrieval with an explicit instruction to answer only from sources (Chapter 12.6.2).
• A sanctioned refusal — NOT_FOUND — so declining is an available action.
• Verification — check quoted facts, resolve cited links, validate identifiers against a real system.
• Constrained output where the answer must come from a fixed set.

Models are also poorly calibrated in words. "I am certain" and "I believe" do not reliably track accuracy, so treat expressed confidence as text, not as a measurement.

Memorisation and extraction. Models can reproduce training data verbatim, and extraction attacks have recovered personal information and long passages from published models. Consequences: do not fine-tune on personal data you cannot defend the model reproducing, and treat a model trained on sensitive data as itself sensitive (Chapter 8.7's classification applies to the weights).

Copyright and provenance. Whether training on copyrighted material is permitted is being litigated in several jurisdictions and is not settled. What is under your control: know what your training data is licensed for, review generated code as a contribution rather than as your own (Chapter 8.7), and keep provenance records.

Automation bias. People accept a machine's output more readily than a colleague's, and the effect grows as the system gets mostly right — a system that is wrong 2% of the time is more dangerous than one wrong 30% of the time, because nobody checks it. Design for review: show sources, show uncertainty, and make disagreeing easy.

Sycophancy — the model concedes when pushed, because agreement was preferred during training (Chapter 12.8). In an advisory system this is a correctness problem, not a personality quirk.

Synthetic media. Voice cloning from seconds of audio and convincing video generation are here. Consent, provenance marking and disclosure are engineering obligations for anything you build in that space.

5. Guardrails, and the jailbreak treadmill

Layers that actually get deployed:

• Training-time alignment — refusal behaviour learned during the alignment stages of Chapter 12.8.
• System prompt constraints — weak on their own; a floor, not a wall.
• Input classifiers — detect prohibited requests before they reach the model.
• Output classifiers — detect harmful, personal or off-policy content before it reaches the user.
• Tool and permission limits — the only structural control, per Chapter 12.6.3. A model with no dangerous tools cannot take a dangerous action regardless of what it is persuaded to say.

Jailbreaks work and keep working. The recurring families: role-play framings, encoded or obfuscated requests, splitting a request across turns, many-shot prompting that fills a long context with examples of compliance, translation into a lower-resource language where safety training is thinner, and adversarial suffixes found by optimisation.

Patching each one is a treadmill, and the honest conclusion is that for anything where the consequence is real, the defence must be architectural — restricted tools, human approval, sandboxing, and output validation — rather than a promise that the model will refuse.

And the opposite failure is real too. Over-refusal — declining legitimate medical, security or legal questions — makes a system useless for professionals and is measured far less often than harmful compliance. Track both, or you will optimise one into the other.

6. Alignment vocabulary, without the drama

Worth knowing precisely, because these terms are used loosely.

Specification gaming — the system optimises the objective you wrote rather than the one you meant. It is ordinary and well documented: the reward-hacked model in Chapter 12.8, the recommender maximising watch time by promoting outrage. It is a design failure, not an intention.

Goal misgeneralisation — the system learns a rule that fits training and generalises wrongly. A classifier that learned to detect snow rather than wolves is the standard illustration.

Interpretability — understanding what a model computes internally, currently an active research area with genuine partial results and no complete method.

Dangerous capability evaluations — testing whether a model can meaningfully assist with serious harms, run before release by major labs and increasingly required by regulation.

Scalable oversight — supervising systems on tasks where human verification is slower than the system's output.

The measured position: current systems produce concrete harms now — biased decisions, confident falsehoods, privacy leakage, fraud enabled by synthetic media — and speculative long-horizon risks are debated seriously by serious people. You do not have to resolve that debate to do your job. The concrete harms are the ones you can measure and reduce, and doing so is not in tension with taking the longer questions seriously.

7. Regulation

The EU AI Act is the first comprehensive framework, and it is risk-tiered:

• Unacceptable — social scoring, some biometric categorisation. Prohibited.
• High risk — employment, credit, education, essential services, law enforcement. Requires risk management, data governance, documentation, logging, human oversight and accuracy testing.
• Limited risk — transparency duties: people must be told they are interacting with an AI system, and synthetic content must be marked.
• Minimal risk — most applications, no specific obligation.

Obligations phase in across 2025–2027, with separate rules for general-purpose models above a compute threshold. If your system touches employment, credit or essential services in the EU, it is likely high risk, and that is a build-time constraint rather than a compliance afterthought.

Existing law already applies everywhere. Discrimination law does not care that the decision came from a model. Data protection (Chapter 8.7) covers training data and inference. Consumer protection covers misleading claims. "The model did it" is not a defence anyone has successfully used.

Frameworks worth knowing by name: the NIST AI Risk Management Framework (voluntary, widely referenced, structured as govern/map/measure/manage) and ISO/IEC 42001 (a certifiable management system standard). Enterprise customers increasingly ask for one of them.

8. What an engineer actually does

Not philosophy — a checklist.

Before building:

• What decision does this make, about whom, and what happens when it is wrong?
• Would a rule-based system be adequate? (Chapter 12.1.)
• Is the training data representative of who will be affected?
• Is the label actually the thing you mean, or a proxy for it?

While building:

• Evaluate per group, with confidence intervals. Never ship a single average.
• Choose and write down a fairness criterion, knowing you cannot have all of them.
• Keep a human in the loop where the consequence is material — with real authority to override.
• Log inputs, outputs and model version so a decision can be explained later.
• Constrain tools and permissions rather than relying on refusal.

Before shipping:

• Write a model card: intended use, out-of-scope use, performance by segment, known limitations.
• Red-team it: try to make it fail, and try to make it fail for a specific group.
• Decide the fallback when it is unavailable or wrong.
• Build the appeal path, and make sure a person can actually action it.

After shipping:

• Monitor by segment, not in aggregate.
• Watch drift (Chapter 12.9) and re-evaluate fairness after every retrain.
• Keep a route for people to report harm, and read it.

And the one thing worth saying plainly: you are allowed to say no. This should not be automated. This data is not adequate. This decision needs a person. Engineers are frequently the only people in the room who know what the system cannot do, and saying so early is far cheaper than saying it after an incident.

Recall

• A model trained on historical outcomes learns the historical process. Bias enters through history, representation, labels, proxy variables, feedback loops and measurement. Deleting the protected attribute removes your ability to measure the disparity, not the disparity.
• Demographic parity, equalised odds and calibration are mathematically incompatible when base rates differ. Fairness is a choice you must state, and one not stated has been made by default.
• Always evaluate disaggregated by group, with confidence intervals. A single average is how disparity stays invisible.
• Mitigations: pre-processing (cannot fix biased labels), in-processing (costs accuracy), post-processing per-group thresholds (may be unlawful — take advice). Narrowing what the model decides is the most reliable mitigation.
• Hallucination is structural — the model has no separate representation of what it knows. Reduce with retrieval, a sanctioned refusal, verification and constrained output, and treat expressed confidence as text, not measurement.
• Models memorise and can be made to reproduce training data — so weights trained on sensitive data are themselves sensitive. Automation bias makes a mostly-right system more dangerous than an obviously-wrong one.
• Guardrails layer, and only tool and permission limits are structural. Jailbreaks — role-play, encoding, many-shot, low-resource languages, adversarial suffixes — keep working, so real consequences need architecture, not refusal. Measure over-refusal too.
• Specification gaming and goal misgeneralisation are design failures, not intentions. The EU AI Act is risk-tiered and makes employment, credit and essential services high risk; existing discrimination and data protection law applies regardless. And you are allowed to say a decision should not be automated.

Self-test: Why does removing gender from the features not remove gender bias? · Which three fairness definitions conflict, and when? · Why is a 98%-accurate system sometimes more dangerous than a 70%-accurate one? · Which guardrail layer is structural rather than probabilistic? · What makes weights trained on personal data sensitive? · What must you write down before shipping a model that affects people?