12.14.3 — Privacy, Fairness and Governance

Cartwise's "pay later" service lets customers buy now and pay in instalments, and a model decides who may use it and up to what limit. Six months after launch, a journalist asked the company three questions. What personal data does the model use, and did customers agree to that? Is it fair: does it approve women and men, or customers from different states, at similar rates, and if not, why not? And who at Cartwise decided it was acceptable, and how would a customer appeal a refusal?

The engineers could answer the first question partly, the second not at all, and the third only with a list of names in a chat thread. None of these is a question about accuracy, latency or cost, the subjects of most of this Part. They are about obligations: to the people whose data a system uses, to the people its decisions affect, and to the law. This page covers all three, in the order the journalist asked them: privacy, fairness and governance.

1. Privacy: personal data in AI systems

The principles the laws share

Data protection laws differ between countries, but they rest on a shared set of principles, set out most influentially in the European Union's General Data Protection Regulation (GDPR, in force since 2018), and found in India's Digital Personal Data Protection Act of 2023 (the DPDP Act) and similar laws elsewhere. The governance chapter of Part 8 covers them for software in general. For AI systems, six matter most.

A lawful basis, usually consent. Personal data may be used only for a reason the law accepts. Under India's DPDP Act, the main basis is the person's consent, given freely for a specific purpose after a clear notice of what data is used and why.

Purpose limitation. Data collected for one purpose may not simply be reused for another. Order history collected to deliver orders is not automatically available to train a credit model. Using it for that needs its own basis.

Data minimisation. Use only the data the purpose needs. A credit model does not need a customer's browsing history of medical products, even if it would slightly improve accuracy.

Retention limits. Keep data only as long as the purpose requires, then delete it, including from logs, traces and caches.

Rights of the person. People can ask what data is held about them, correct it, and have it erased, and they can complain to a regulator: in India, the Data Protection Board.

Rights about automated decisions. GDPR's Article 22 gives people the right not to be subject to a decision based solely on automated processing when it significantly affects them, such as a refusal of credit, without the ability to obtain human review, express their view and contest the decision.

India's DPDP Rules were notified in November 2025, with most of the substantive duties (notice and consent, security safeguards, breach reporting, children's data) taking effect eighteen months later, in 2027. Penalties reach ₹250 crore for failing to protect data adequately. Dates and details of laws change; the principles above change much less, and a system built on them is ready for most of what any particular law requires.

What is different about AI

Three features of AI systems make privacy harder than in ordinary software.

Models can memorise. As 12.14.2 showed, a model trained on personal data can sometimes reproduce it. And erasing someone from a trained model is not practical today. Deleting a customer's row from a database takes a second; removing what a model learned from that row means, in practice, retraining without it. Machine unlearning, methods to remove a record's influence without full retraining, is an active research area but not yet a dependable engineering tool. The practical rule follows: keep personal data where it can be deleted, in databases and retrieval indexes, and train models only on data whose use is consented to and, where possible, aggregated or anonymised.

Data travels further. A customer's message to "Ask Cartwise" passes through the gateway, perhaps an external model provider, a trace store, a cache, a memory store and an evaluation set. Each is a copy that must respect the same purpose, retention and erasure rules, and an erasure request must reach every one. Cartwise keeps a map of where personal data flows for each AI feature, so that an erasure request deletes it from all of them, and removes personal details such as card numbers and phone numbers at the gateway before requests leave for external providers.

Inferences are personal data too. A model that predicts a customer is likely to be pregnant from their purchases, or creditworthy from their spending, creates new personal data, sometimes sensitive, that the customer never disclosed. Such inferences fall under the same principles: they need a purpose and a basis, and the customer has rights over them.

Before building an AI feature that processes personal data at scale or makes significant decisions about people, the standard practice is a data protection impact assessment: a written analysis of what data is used and why, the risks to the people involved, and how they are reduced. For the pay-later model, it would have prompted the questions the journalist asked, before launch rather than after.

2. Fairness: does the model treat groups differently?

How unfairness gets in

In 2018, Reuters reported that Amazon had abandoned an experimental tool for ranking job applicants. It had been trained on ten years of the company's hiring decisions, in an industry where most hires were men, and it had learned to prefer men: it marked down CVs containing the word "women's", as in "women's chess club captain", and downgraded graduates of two all-women colleges. Nobody had given it the applicants' gender. It learned the pattern from the decisions it was trained to imitate, and would have applied it more consistently, and at greater scale, than any individual recruiter.

The story shows the two main routes by which unfairness enters a model. Historical bias: the training labels record past human decisions, and if those decisions were unfair, a model that learns them faithfully is unfair too. Proxy variables: removing a sensitive attribute such as gender or religion from the inputs does not remove its influence, because other inputs carry the same information. Postcode correlates with community and income, first names with gender and religion, shopping patterns with almost everything. Worse, deleting the attribute removes the ability to measure the disparity, while leaving the disparity in place.

Two further routes are worth naming. Unrepresentative data: a group that is rare in the training data gets worse predictions, as the speech recognisers of 12.13.2 did for under-represented accents. And feedback loops, from 12.12.4: a group refused credit never gets the chance to show it would have repaid, so the next model's data confirms the first model's refusal.

Measuring it: several definitions, one worked example

There are several precise definitions of what it means for a model to treat two groups fairly, and they measure different things. Take the pay-later model and two groups of 1,000 applicants each. Suppose that in group A, 500 would repay if approved, and in group B, 200 would (the base rates are 50% and 20%). The model approves 450 people in group A, of whom 400 would repay, and 200 people in group B, of whom 150 would repay.

Measure	Question it asks	Group A	Group B
Approval rate	what share is approved?	450/1,000 = 45%	200/1,000 = 20%
True positive rate	of those who would repay, what share is approved?	400/500 = 80%	150/200 = 75%
False positive rate	of those who would not repay, what share is approved?	50/500 = 10%	50/800 = 6.25%
Precision	of those approved, what share repays?	400/450 = 88.9%	150/200 = 75%

Each row is a different definition of fairness.

Demographic parity asks for equal approval rates. Here they are 45% and 20%: very unequal. In United States employment law, a long-standing guideline called the four-fifths rule treats a selection rate for one group below 80% of the highest group's as evidence of adverse impact. Here 20\% / 45\% \approx 0.44, far below 0.8.

Equal opportunity asks for equal true positive rates: people who would repay should be approved at the same rate whatever their group. Here 80% and 75% are close but not equal. Equalised odds asks for equal true positive and false positive rates; the false positive rates, 10% and 6.25%, also differ.

Predictive parity, closely related to calibration, asks that an approval mean the same thing in both groups: the share of approved people who repay should be equal. Here it is 88.9% against 75%, so an approval in group B is less reliable.

Why you cannot have them all

A natural response is to demand all of them. In 2016 and 2017, two independent papers, by Jon Kleinberg, Sendhil Mullainathan and Manish Raghavan, and by Alexandra Chouldechova, proved that this is mathematically impossible whenever the groups' base rates differ, except for a perfect model. A calibrated model's scores mean the same thing in each group, so with different base rates it must approve the groups at different rates, and its error rates must differ too. Forcing equal approval rates, in turn, means approving people more likely to default in one group or refusing likely repayers in the other, which changes the error rates and the meaning of an approval.

Chouldechova's paper came out of a public argument about COMPAS, a tool used in United States courts to predict whether defendants would reoffend. In 2016, the journalism organisation ProPublica showed that its false positive rate was much higher for Black defendants than white ones: more Black defendants who did not reoffend had been labelled high risk. The tool's makers replied that it was calibrated: a given score meant the same risk in both groups. Both were right. With different base rates, both cannot be true at once.

So fairness is not a property a model simply has or lacks. It is a choice between definitions, which depends on which mistake does the most harm in the situation, and it must be made explicitly, written down, and justified. For pay-later credit, Cartwise chose to require that equally creditworthy people be approved at similar rates (equal opportunity), to monitor approval rates by group for adverse impact, and to investigate and explain any gap rather than hide it.

What to do in practice

• Measure by group, always. Compute every metric for each group that matters, with confidence intervals, because small groups give noisy numbers. This is the slice evaluation of 12.12.4 applied to people, and an overall average is the most common way a disparity stays invisible.
• Keep sensitive attributes for measurement, stored securely and with a lawful basis, even when they are not model inputs; without them you cannot see a problem caused by proxies.
• Look for proxies. If an input such as postcode predicts a sensitive attribute strongly, ask whether it belongs in the model at all, and whether its contribution is justified by what it says about the actual outcome.
• Choose and document the definition, with the reasons, before the model ships.
• Fix the cause where you can: better data for under-represented groups, labels that are not records of past bias, exploration to escape feedback loops. Adjusting thresholds per group is sometimes used, and is a legal question in many places, not only a technical one.
• For language models, test outputs across names, languages and dialects: do support replies differ in tone or helpfulness when only the customer's name changes? Swap the name in otherwise identical prompts and compare. These counterfactual checks belong in the evaluation sets of 12.8.6.

3. Governance: who decides, and how

The journalist's third question, who decided this was acceptable?, is about governance: the processes by which an organisation decides what AI it builds, checks that it works as intended, and stays accountable for it.

Knowing what you have

Governance begins with an inventory: a list of every AI system the company builds or uses, its purpose, its owner, the data it uses, and the decisions it affects. It sounds bureaucratic until you discover the systems nobody knew about, such as an internal team pasting customer data into a public chatbot to draft replies. Unapproved use of external AI tools is often called shadow AI, and finding it is the inventory's second job.

Matching care to risk

Not every AI system deserves the same scrutiny, so systems are classified by risk, and the process scales with it. Cartwise uses three tiers:

Tier	Example	Process
Low	product description drafts, internal search	owner sign-off, standard evaluation
Medium	"Ask Cartwise", refund agent	plus security review, red-teaming, monitoring plan
High	pay-later credit, fraud declines, hiring tools	plus impact assessment, fairness analysis, review board, human appeal

The same idea underlies the EU AI Act, the first comprehensive AI law, which entered into force in August 2024. It sorts AI systems into risk categories. A few uses are prohibited, such as social scoring by public authorities and certain kinds of manipulation. High-risk uses, including credit scoring, hiring, education and critical infrastructure, carry obligations for risk management, data quality, documentation, human oversight, accuracy and logging. Systems that interact with people or generate content carry transparency duties, such as telling users they are talking to an AI and marking generated media. The rest are minimally regulated. The largest general-purpose models have their own obligations, with extra ones for models trained with more than 10^{25} arithmetic operations, which the law treats as posing systemic risk. Fines for prohibited uses reach €35 million or 7% of global annual turnover. Its obligations take effect in stages; in 2026, EU lawmakers agreed to delay the main high-risk obligations, for most uses, from August 2026 to December 2027, so check the current text before relying on any date. The pay-later model, if offered to customers in the EU, would be a high-risk system.

Two widely used frameworks organise the work without being laws. The United States NIST AI Risk Management Framework (2023) arranges it into four functions: govern (policies, roles, accountability), map (understand the context and risks of each system), measure (assess them) and manage (act on them). ISO/IEC 42001 (2023) is an international standard for an AI management system, against which a company can be certified, much as ISO 27001 certifies information-security management.

Documenting what you built

Two documents, proposed by researchers and now standard, make a system's purpose and limits visible to the people who must judge it. A model card, proposed by Margaret Mitchell and colleagues in 2019, is a short document for a model: what it is for, what it must not be used for, how it was evaluated, how it performs for each relevant group, and its known limitations. A datasheet for a dataset, proposed by Timnit Gebru and colleagues, records how a dataset was collected, what it contains, what consent covers it, and what biases it may carry. The registry of 12.12.4 stores both with every model version, so that the documentation always matches what is deployed.

People with real authority

For decisions that significantly affect people, governance means a person can step in. That has three parts. Review before launch: high-tier systems go to a review group with the authority to say no, including people outside the building team, such as legal, risk and customer representatives. Human oversight in operation: the approvals and overrides of 12.10.8. Appeal: a customer refused pay-later credit can ask for a person to review the decision, receives the main reasons in plain language, and can supply information the model did not have. "The model said so" is not an answer a company can give a customer or a regulator.

Oversight has its own failure mode, already met in 12.10.8 as approval fatigue. Automation bias is the tendency of people to accept a machine's output with less scrutiny than a colleague's, and it is strongest when the machine is usually right: a reviewer who has seen the model be correct a thousand times stops checking. Oversight that works is designed for disagreement: the reviewer sees the evidence and the model's uncertainty, not only its verdict, disagreeing is as easy as agreeing, and reviewers are periodically shown cases where the model was wrong.

Being open with users

People should know when they are dealing with AI. "Ask Cartwise" says it is an AI assistant and offers a person. Generated product backgrounds are labelled, with provenance credentials, as in 12.13.3. Refused applicants are told that an automated system was involved and how to ask for review.

Content safety, in both directions

Systems that generate text or images for the public also need content safety: input and output checks, often small classifier models, that stop the system producing harmful material such as harassment, dangerous instructions or sexual content involving minors. Their failure in the other direction matters too. Over-refusal, declining legitimate requests because a filter is too cautious, makes a system useless for exactly the people it should help: a customer asking about returning a knife set, or a medical supplies seller describing a product. Measure both: how often harmful requests get through, and how often legitimate ones are refused. Optimising only one reliably makes the other worse.

And engineers can say no

The engineers who build a system are often the only people who fully understand what it cannot do. Part of governance is a culture in which saying "this decision should not be automated", or "this model is not accurate enough for this group to ship", is a normal, respected engineering judgement rather than an obstacle.

4. The pay-later model, revisited

After the journalist's questions, Cartwise worked through every section of this page for the pay-later model. A data protection impact assessment documented its data and removed two inputs the purpose did not need. A separate consent now covers the use of order history for credit, and declining it means a manual review instead. Approval rates and true positive rates are measured monthly by gender and state, with the chosen fairness definition and its reasoning in the model card; a gap found between two states was traced to a postcode-based input acting as a proxy, which was removed with little loss of accuracy. The model was classified as high tier, reviewed by a board including legal and customer advocates, and given an appeal route with plain-language reasons. The answers to the journalist's three questions are now a document, not a chat thread.

What the interviewer will push on

"How do you handle bias in a model?" They want more than "remove the sensitive attribute". The strong answer: unfairness enters through historical labels, proxies, unrepresentative data and feedback loops; removing the attribute leaves proxies and removes your ability to measure; so keep it for measurement, evaluate by group with confidence intervals, choose and document a fairness definition, and fix causes in the data.

"What fairness metrics are there, and can you satisfy them all?" Demographic parity (equal approval rates), equal opportunity (equal true positive rates), equalised odds (equal true and false positive rates), predictive parity or calibration (an approval means the same thing). With different base rates they cannot all hold at once (Kleinberg et al.; Chouldechova). Mention COMPAS, and that the choice depends on which error does the most harm.

"A user asks you to delete their data. What about the model trained on it?" Delete it everywhere it can be deleted: databases, indexes, caches, memory, traces, evaluation sets. Removing it from trained weights generally means retraining, since unlearning is not yet dependable, which is why personal data should live in deletable stores and training should use consented, aggregated or anonymised data.

"What does the EU AI Act require?" It is built around risk:

• Some uses are banned outright.
• High-risk uses carry duties: managing risk, data quality, documentation, human oversight and logging.
• Chatbots and generated media must say what they are.
• General-purpose models have their own rules, with extra ones for models trained with more than 10^{25} operations.

The duties come in gradually, and the dates have already moved.

"How do you make human oversight effective?" Show evidence and uncertainty, make disagreement easy, send only meaningful cases, watch for automation bias, and give affected people an appeal with reasons.

Something to volunteer: fairness definitions conflict mathematically, so "is the model fair?" has no answer until someone chooses which definition matters for this decision. The engineer's job is to make that choice visible, measured and documented, rather than to pretend a single number settles it.

Recall

• Privacy principles (GDPR, India's DPDP Act 2023): lawful basis (usually consent), purpose limitation, minimisation, retention limits, rights to access, correction and erasure, and rights about solely automated significant decisions (GDPR Article 22). DPDP Rules notified November 2025, main duties from 2027, penalties up to ₹250 crore.
• AI specifics: models memorise and cannot practically unlearn, so keep personal data in deletable stores; data spreads to providers, traces, caches and memory, so map flows and redact at the gateway; inferences are personal data. Do impact assessments.
• Unfairness enters via historical labels, proxy variables, unrepresentative data and feedback loops; removing the attribute hides rather than removes it.
• Definitions: demographic parity (approval rates; four-fifths rule), equal opportunity (true positive rates), equalised odds (plus false positive rates), predictive parity or calibration. With different base rates they conflict (Kleinberg et al., 2016; Chouldechova, 2017; COMPAS). Choose, document, measure by group with intervals, fix causes.
• Governance:
  • Keep a list of every AI system, including the ones staff set up on their own (shadow AI).
  • Sort them into risk tiers. Know the EU AI Act's risk categories (its high-risk duties were delayed in 2026 for most uses), NIST AI RMF (govern, map, measure, manage) and ISO/IEC 42001.
  • Write model cards and datasheets.
  • Use review boards and human oversight, and let people appeal a decision and hear the reasons.
  • Watch for automation bias.
  • Tell people when they are dealing with AI.
  • Measure both harmful output and needless refusals.
  • Engineers may say no.

Self-test: In the worked example, what is group B's approval rate as a fraction of group A's, and what does the four-fifths rule say about it? Why does removing gender from the inputs not stop a model discriminating by gender? A customer refused pay-later credit asks why; what three things does Cartwise's appeal route give them?

Next: 12.15.1 Designing an AI System — the final chapter brings the whole Part together: a method for designing any AI system from a blank page, then four complete case studies and a capstone project you build yourself.