Appearance
12.7.9 — From Predictor to Assistant
Give a freshly pretrained base model, from 12.7.5, this prompt:
Write a polite reply to a customer whose parcel is three days late.
It might write a reply. It is just as likely to continue with "Write a polite reply to a customer whose refund was rejected. Write a polite reply to…", because on the internet, an instruction like that most often appears in a list of writing exercises. Or it might continue as a forum post asking how to write such replies. All of these are plausible continuations, and continuing text plausibly is the only thing a base model was trained to do. It knows a great deal; it has no idea that you want it to do something.
The assistants you use, Claude, ChatGPT, Gemini and the "instruct" or "chat" versions of open models, are base models put through further training stages that teach them to treat text as a conversation, follow instructions, answer helpfully and honestly, and decline harmful requests. This is usually called alignment: aligning the model's behaviour with what its users and builders intend. The stages are far cheaper than pretraining, but they decide almost everything about how a model behaves, and knowing them explains the habits, strengths and failure modes of every assistant you will build on.
1. Stage one: supervised fine-tuning on conversations
The first stage is the simplest. Collect many example conversations, each an instruction or question followed by an excellent response written or carefully checked by people, and continue training the base model on them with the same next-token loss as pretraining. This is supervised fine-tuning, SFT, also called instruction tuning. Typical datasets have tens of thousands to a few hundred thousand high-quality examples, a tiny amount next to trillions of pretraining tokens.
What SFT mostly teaches is not new knowledge but a new format and role: when text looks like a user asking something, what follows is an assistant answering it, completely, in a helpful register, and then stopping. A 2023 study from Meta, titled "LIMA: Less Is More for Alignment", made the point vividly: a strong base model fine-tuned on only about 1,000 carefully chosen examples already behaved much like an assistant. The knowledge came from pretraining; SFT taught the model how to present it.
Chat templates: how a conversation becomes one string of tokens
A model only ever sees one sequence of tokens, so a conversation with roles must be flattened into one. Each model family uses a fixed chat template with special tokens marking who is speaking. Illustratively, a conversation becomes something like:
text
<|system|>You are Cartwise's support assistant. Be concise.<|end|>
<|user|>Where is my order 4471?<|end|>
<|assistant|>Generation starts right after the final <|assistant|> marker, and the model was trained to produce the assistant's reply and then emit <|end|>, which is how it knows to stop. The system prompt is the first block: instructions from the application builder that the model was trained to give priority to throughout the conversation.
When you call a hosted API, the provider applies the template for you, which is why APIs take a list of messages with roles rather than one string. When you run an open model yourself, you must use exactly the template it was trained with; the transformers library's tokenizer.apply_chat_template does it. A model given the wrong template still produces text, just noticeably worse text, with no error message: one of the most common silent bugs in self-hosted LLM applications.
2. Stage two: learning from human preferences
SFT gets you an assistant that answers in the right format. It does not reliably get you the best answers, because people find it much harder to write an excellent answer than to recognise one. So the next stage learns from comparisons.
The method that made ChatGPT possible is reinforcement learning from human feedback, RLHF. It was developed in earlier research, including a 2017 paper by Paul Christiano and colleagues, and applied to language models at scale by OpenAI's InstructGPT paper in 2022 by Long Ouyang and colleagues. That paper's most striking result was that people preferred the answers of a 1.3-billion-parameter InstructGPT model over those of the original 175-billion-parameter GPT-3: a model a hundred times smaller, made more useful by how it was trained to behave.
RLHF has three steps, shown below.
Step ①: collect comparisons. For many prompts, the SFT model writes two (or more) answers, and trained people pick the better one. Comparing is far easier and more consistent than assigning scores out of 10, which is why it is the format used; the agreement measures of 12.4.1 apply to these raters exactly as to any annotators.
Step ②: train a reward model. A separate network, usually a copy of the language model with its final layer replaced by one that outputs a single number, is trained to give the preferred answer the higher score. The standard training rule, named after Ralph Bradley and Milton Terry who proposed it for ranking in 1952, says the probability that answer A beats answer B is the sigmoid of the difference in their scores:
P(A \text{ preferred over } B) = \sigma\big(r(A) - r(B)\big)
If the reward model scores A at 1.2 and B at 0.4, it predicts A wins with probability \sigma(0.8) \approx 0.69. If people did prefer A, the loss is -\ln 0.69 \approx 0.37, and training pushes A's score up and B's down until the reward model agrees with people as often as possible. This is the logistic regression of 12.3.1 applied to the difference between two scores.
Step ③: optimise the assistant against the reward. Now the assistant generates answers to many prompts, the reward model scores them, and reinforcement learning, from 12.1.2, adjusts the assistant's weights to make high-scoring answers more likely. The algorithm usually used is PPO, proximal policy optimisation, from John Schulman and colleagues in 2017, which limits how much the model can change in each update so that training stays stable.
Reward hacking, and the leash
The reward model is only an imitation of human judgement, learned from a limited number of comparisons, and it has blind spots. A model optimised hard against it will find them. It might learn that longer answers score higher regardless of content, or that confident phrasing scores higher than honest hedging, or that a certain flattering phrase is always rewarded, and drift towards producing exactly those, while getting worse by the standards of the people the reward model was meant to imitate. This is reward hacking, and it is the reinforcement learning version of overfitting from 12.3.3.
The standard defence is a leash. The quantity being maximised is not the reward alone, but the reward minus a penalty for drifting away from the SFT model:
\text{objective} = r(\text{answer}) - \beta \cdot \text{KL}\big(\text{assistant} \,\|\, \text{SFT model}\big)
Read it as "reward, minus beta times how far the assistant's output probabilities have moved from the original's". KL stands for Kullback–Leibler divergence, a measure from information theory, introduced by Solomon Kullback and Richard Leibler in 1951, of how different two probability distributions are; it is closely related to the cross-entropy of 12.2.6. The weight \beta (beta) sets how short the leash is. The model can improve its answers, but only by moving in ways that stay close to the readable, sensible behaviour it started with.
3. DPO: preferences without the reward model
RLHF works, but it is complicated and fragile: three models in memory at once (the assistant, the reward model, and the frozen SFT reference), a reinforcement learning loop that is sensitive to its settings, and a reward model that can be gamed. In 2023, Rafael Rafailov and colleagues at Stanford published direct preference optimisation, DPO, with the subtitle "Your Language Model Is Secretly a Reward Model".
Their insight was mathematical. For the objective above, the best possible assistant can be written down exactly in terms of the reward, and that relationship can be turned around, so the reward can be expressed in terms of the assistant's own probabilities compared with the reference model's. Substituting that back into the Bradley-Terry rule removes the reward model entirely. What remains is an ordinary supervised loss on the comparison pairs:
L = -\ln \sigma\Big(\beta\big[(\ln \pi(y_w) - \ln \pi_{\text{ref}}(y_w)) - (\ln \pi(y_l) - \ln \pi_{\text{ref}}(y_l))\big]\Big)
That looks heavy, so read it in pieces. y_w is the preferred ("winning") answer and y_l the rejected ("losing") one. \pi(y) is the probability the model being trained gives to an answer, and \pi_{\text{ref}}(y) is the probability the frozen SFT model gives to it. Each bracket asks: how much more likely has the model made this answer, compared with where it started? The loss wants the preferred answer's increase to be bigger than the rejected answer's.
A worked example, with \beta = 0.1. Suppose the model gives the preferred answer a log-probability of -10 and the reference gave it -11, so the model has raised it by 1. For the rejected answer, the model gives -12 and the reference -11, so the model has lowered it by 1. The difference of differences is 1 - (-1) = 2, times \beta is 0.2, and the loss is -\ln \sigma(0.2) \approx -\ln 0.550 \approx 0.598. If the model had not moved at all from the reference, the inner value would be 0 and the loss -\ln 0.5 \approx 0.693; if it had moved the wrong way, raising the rejected answer, the loss would be larger, for example -\ln \sigma(-0.2) \approx 0.798. The gradient pushes the preferred answer up and the rejected one down, and the reference terms act as the leash, built in.
DPO needs only the model, a frozen copy, and the comparison data, and it trains like ordinary fine-tuning. It became the most common way to align open models, with several later variants. RLHF with PPO remains in use at large labs, which have the infrastructure for it and find advantages in some settings.
Feedback from AI, guided by principles
Human comparisons are slow and expensive, and asking people to read harmful content to label it has its own costs. In 2022, Anthropic described Constitutional AI: the model is given a written set of principles, the "constitution", and is trained to critique and revise its own answers against them; then an AI model, rather than people, chooses which of two answers better follows the principles, and those AI preferences train the model as in RLHF. The general idea, using AI judgements as preference data, is called RLAIF (reinforcement learning from AI feedback), and some mix of human and AI feedback is now standard.
4. Reasoning models: training a model to think before answering
The newest stage targets a different weakness. Asked a hard question in one go, a model must produce the answer token by token with no chance to work it out first. Researchers noticed early, in a 2022 paper by Jason Wei and colleagues at Google, that simply asking a model to "think step by step" before answering, called chain-of-thought prompting, improved accuracy on maths and logic problems: writing out intermediate steps gives the model somewhere to do the work.
Reasoning models train that behaviour in deliberately. The model is trained with reinforcement learning to produce a long internal chain of reasoning before its final answer, rewarded on problems where the answer can be checked automatically: a maths problem with a known numerical answer, a coding task with test cases that pass or fail, a puzzle with a verifiable solution. Because the reward comes from a checker rather than a learned reward model, there is much less room for reward hacking, and the model can be trained on enormous numbers of problems. Over training, models learn to try approaches, notice mistakes, backtrack and verify. OpenAI's o1, released in September 2024, was the first widely used model of this kind; DeepSeek's R1, released openly in January 2025, described the method publicly, including a simplified reinforcement learning algorithm called GRPO that compares a group of answers to the same problem with each other instead of training a separate value model.
This opened a new way to buy capability: test-time compute. Instead of only making models bigger, let them spend more tokens thinking on harder questions. Accuracy on hard problems rises with the amount of thinking allowed. The engineering consequence is direct: reasoning requests take longer and cost more, because the thinking tokens are output tokens (see 12.7.6). Many APIs let you set a thinking budget. For Cartwise, extended thinking earns its cost on a tangled refund dispute; it is wasted on "where is my order?".
5. What alignment leaves behind
Every assistant carries habits from this training that you should know and design around.
Sycophancy. Models trained to win human approval learn that agreeing with people tends to be approved. So they sometimes accept a user's wrong premise ("Since your refund policy is 60 days…" when it is 30), or change a correct answer when the user pushes back. For a support assistant, that is dangerous. The mitigation is to put the facts in the context (Chapter 12.9), instruct the model to correct false premises politely, and test for it explicitly in evals.
Confident errors. Fluent, assured prose was rewarded, and a confident wrong answer can read better than an honest "I don't know". Labs train against this, and it still happens. Ground answers in retrieved facts, and allow, even reward, "I can't find that information".
Refusals that are too eager or not eager enough. Safety training teaches models to decline harmful requests, and the boundary is never perfect: a model may refuse a harmless question that resembles a harmful one ("how do I kill a process?") or comply with a cleverly disguised harmful one. Your system prompt, and evaluation on your real traffic, tune where the boundary sits for your use, and Chapter 12.14.1 covers the attacks that target it.
A small cost in raw capability. Alignment training can slightly reduce scores on some benchmarks compared with the base model, sometimes called the alignment tax. For almost every application, the assistant's usefulness far outweighs it; base models are mainly useful for research and for some specialised fine-tuning.
The model follows instructions, including bad ones. An assistant trained to follow instructions will also try to follow instructions that arrive hidden inside a document or web page it was asked to read. That is prompt injection, and it is a direct consequence of this whole page: the more obedient the model, the more carefully you must control what text reaches it.
What the interviewer will push on
"How is ChatGPT different from GPT-3?" They want the stages: pretraining gives knowledge, SFT gives the assistant format, preference training (RLHF or DPO) gives better-judged, safer answers. The InstructGPT result, smaller but preferred, is the tell of someone who has read the source.
"Explain RLHF versus DPO." A good answer: RLHF trains a reward model from comparisons, then uses RL with a KL penalty; DPO derives a loss that optimises directly on the comparisons with the KL constraint built in, so no reward model and no RL loop. Bonus for naming reward hacking as the failure the KL term prevents.
"What is reward hacking?" The model exploiting gaps in the learned reward, such as length or confident tone, instead of genuinely improving. Mention the KL leash and checkable rewards.
"Why does a model agree with a false premise?" Sycophancy from preference training; mitigations are grounding, explicit instructions and targeted evals.
Recall
- A base model only continues text. Alignment training makes it an assistant: supervised fine-tuning on example conversations teaches the format and role (LIMA: about 1,000 good examples go a long way). A chat template with special tokens flattens roles into one sequence; the system prompt comes first. Self-hosted models must use their own template.
- Reinforcement learning from human feedback: people compare answers; a reward model learns P(A \succ B) = \sigma(r_A - r_B); PPO optimises the assistant against it, minus a KL penalty that prevents reward hacking. InstructGPT at 1.3 billion parameters beat GPT-3 at 175 billion in human preference.
- Direct preference optimisation removes the reward model and the RL loop: a supervised loss on preference pairs that raises the preferred answer's probability relative to a frozen reference more than the rejected one's. Constitutional AI and RLAIF use AI judgements guided by written principles.
- Reasoning models are trained with RL on automatically checkable problems to think before answering; test-time compute trades more output tokens for accuracy on hard questions.
- Alignment leaves sycophancy, confident errors, imperfect refusal boundaries, a small capability cost, and obedience that prompt injection exploits.
Self-test: Why are people asked to compare two answers rather than score one? In the DPO example, what is the loss if the model raised both answers by exactly 1? Why does rewarding reasoning models with automatic checkers reduce reward hacking?
Next: 12.8.1 Calling a Model — you now know what happens inside an LLM, from tokens to alignment; Chapter 12.8 turns to building with one, starting with what is actually in an API request and response, and how to call a model reliably from production code.