Appearance
12.7.5 — Pretraining and Scaling
Take the transformer of 12.7.4, fill its billions of weights with random numbers, and ask it where order 4471 is. It produces gibberish: random tokens, because random weights encode nothing. The same architecture, after training, writes code, explains tax rules and drafts a polite apology. Nothing about its structure changed. Everything that it can do came from one training task, run at a scale that is hard to picture:
read a huge amount of text, and at every position, predict the next token.
That stage is called pretraining, and the model it produces is called a base model or foundation model. This page covers how pretraining works, why such a simple task produces so much capability, what data and compute it takes, and the scaling laws: the surprisingly regular rules that say how much better a model gets as it grows, which drove every decision in the field from 2020 onwards.
1. The task: predict the next token, everywhere at once
Training uses the self-supervised idea from 12.1.2: the text labels itself. Take one sentence, "Your parcel will arrive tomorrow.", tokenized into five tokens. It contains four training examples at once:
At the first position, the model sees "Your" and must predict "parcel". At the second, it sees "Your parcel" and must predict "will". And so on. Because of the causal mask from 12.7.3, each position can only look at the tokens before it, so it cannot cheat by reading its target, and all four predictions are computed in one parallel pass. The loss is the cross-entropy from 12.2.6, -\ln p(\text{correct next token}), averaged over every position of every sequence in the batch. Backpropagation from 12.5.2 and AdamW from 12.5.3 then nudge every weight. Repeat for trillions of tokens.
This is why training is fast and generation is slow. During training, the whole correct sequence is known in advance, so every position's prediction is made at once. During generation, the next token is not known until it has been chosen, so tokens must be produced one at a time, as 12.7.1 described.
2. Why predicting the next word teaches so much
It is natural to doubt that "guess the next word" could produce a model that writes working code. The answer is that predicting text well, across all kinds of text, requires knowing what the text is about. Look at what each of these continuations demands:
- "The capital of Australia is ___": knowing a fact. Guessing "Sydney", the more famous city, gets a high loss.
- "Order 4471 was placed on 3 March and takes five working days, so it should arrive by ___": counting working days from a date.
- "def total(items): return sum(item.price * item.___": understanding what the code is doing ("quantity").
- "She thanked him, but her tone made it clear she was ___": reading people's intentions from words.
Every time the model gets one of these wrong, the loss is high and the gradient pushes its weights towards whatever internal machinery would have made the right prediction more likely. Across trillions of tokens covering science, law, code, conversations and stories, the cheapest way to keep lowering the loss is to build internal representations of grammar, facts, arithmetic, programming and human behaviour. Nobody designs those representations; they are what minimising next-token error on this much varied text ends up producing.
A useful way to say it: a model that could predict text perfectly would have to have compressed everything the text describes. Real models are far from perfect, which is why they make mistakes, but this explains why getting better at the prediction task kept producing more capability.
What a base model is, and what it is not
The output of pretraining is a model that continues text. That is not the same as an assistant. Give a base model the prompt "Customer: Where is my order 4471?\nAgent:" and it might answer helpfully. It might equally continue with a second customer complaint, or a heading from a customer-service training manual, or a list of more questions, because all of those are plausible continuations of text that looks like that on the internet. A base model has no notion of "the user" or "being helpful"; it only knows what text tends to look like.
Turning a base model into an assistant that follows instructions, answers the question asked and declines harmful requests takes further training stages, covered in Chapter 12.7.9. But nearly all the knowledge and skill comes from pretraining. The later stages mostly teach the model how to use it.
3. The data
A model can only learn what is in its training text, so pretraining data decides much of what a model knows and how well it reasons. Building the dataset is a large engineering project in its own right, usually more effort than the model code.
Sources. The bulk usually comes from web pages, most often from Common Crawl, a non-profit that has published regular crawls of billions of web pages since 2008. That is mixed with code (from public repositories), books, scientific papers, encyclopedias, question-and-answer sites, and increasingly text in many languages. The proportions matter: adding more code to the mix is widely reported to improve reasoning in general, not just programming.
Filtering. Raw web text is mostly not worth learning from: navigation menus, cookie banners, spam, machine-generated pages built to attract search traffic, broken encodings. Pipelines remove most of it with rules (pages that are too short, too repetitive, mostly symbols) and with small classifiers, often the very logistic regressions of 12.3.1 trained to tell "looks like a good encyclopedia article" from "looks like spam". Personal information and harmful content are filtered too.
Deduplication. The web is full of copies: the same news story syndicated on hundreds of sites, the same licence text in thousands of code repositories. Training on many copies wastes compute and makes the model more likely to memorise and repeat text word for word. Exact duplicates are found by hashing, and near-duplicates with a technique called MinHash, which gives similar documents similar short fingerprints, much like the locality-sensitive hashing mentioned in 12.6.3.
Scale. The numbers are large and have grown fast. Meta's Llama 2 models, released in 2023, were trained on about 2 trillion tokens; Llama 3, in 2024, on over 15 trillion. A trillion tokens is roughly the text of several million books.
Two honest problems hang over pretraining data. Much of it is copyrighted, and whether training on it is lawful is being argued in courts and legislatures in several countries. And high-quality human-written text is finite: careful estimates suggest that the largest training runs are approaching the total stock of usable public text, which is one reason the field is investing heavily in synthetic data and in other ways to improve models.
4. The compute
How much computing does pretraining take? A useful rule of thumb, from the scaling-law papers below, is that training a transformer costs about 6 floating-point operations per parameter per training token: roughly 2 for the forward pass and 4 for the backward pass. For a model with N parameters trained on D tokens:
C \approx 6 N D
Work it for Llama 2 7B: 6 \times (7 \times 10^9) \times (2 \times 10^{12}) \approx 8.4 \times 10^{22} operations. An A100 GPU can in principle do about 3 \times 10^{14} such operations per second in 16-bit arithmetic, and real training runs typically achieve about 40 percent of a chip's peak because of communication and memory waits, so about 1.25 \times 10^{14} per second. Dividing: 8.4 \times 10^{22} / 1.25 \times 10^{14} \approx 6.7 \times 10^8 GPU-seconds, about 186,000 GPU-hours. Meta's own report for that model gives 184,320 GPU-hours, which shows how good the rule of thumb is. Spread across 2,000 GPUs, it is about four days of training; the largest models take tens of thousands of GPUs for months.
That arithmetic is why pretraining large models is done by a small number of well-funded labs, and why nearly every company, Cartwise included, starts from someone else's pre-trained model.
5. Scaling laws: steady returns from size
In 2020, Jared Kaplan and colleagues at OpenAI published a paper that changed the industry: "Scaling Laws for Neural Language Models". They trained many models of different sizes on different amounts of data and found that the test loss fell in a remarkably smooth and predictable way as the model got bigger, the data got bigger, and the compute got bigger, across many orders of magnitude.
The shape of the relationship is a power law:
L(N) \approx \left(\frac{N_c}{N}\right)^{\alpha}
Read it as "loss is roughly a constant divided by the number of parameters, raised to a small power alpha". For model size they measured \alpha \approx 0.076. A power law has a distinctive property: every tenfold increase gives the same fractional improvement. Here, multiplying the parameters by 10 multiplies the loss by 10^{-0.076} \approx 0.84, about 16 percent lower, whether you go from 10 million to 100 million parameters or from 10 billion to 100 billion. On a chart with both axes on a logarithmic scale, a power law is a straight line, and that is what the paper's charts looked like.
Why did this matter so much? Because it made capability predictable from money. A lab could train a series of small, cheap models, fit the line, and forecast how good a model a hundred times bigger would be before spending tens of millions of dollars on it. Predictable returns on investment are rare in research, and this one justified the enormous spending on compute that followed.
Chinchilla: the recipe gets corrected
The 2020 paper suggested that, for a given compute budget, it was best to make models very large and train them on relatively little data. In 2022, Jordan Hoffmann and colleagues at DeepMind revisited the question with more careful experiments and reached a different answer: model size and data should grow together, and the best ratio is roughly 20 training tokens per parameter. They demonstrated it with a model called Chinchilla: 70 billion parameters trained on 1.4 trillion tokens, which outperformed DeepMind's own earlier Gopher, four times larger at 280 billion parameters but trained on only 300 billion tokens, using the same compute. Many of the large models of 2020 and 2021 had been, by this measure, much too big for the data they were trained on.
Then practice deliberately went past it
Chinchilla's 20-to-1 ratio gives the lowest loss for a fixed training budget. But a model is trained once and then used millions of times, and a smaller model is cheaper every single time it is used. So labs now deliberately train smaller models on far more data than the compute-optimal recipe says, accepting a more expensive training run to get a cheaper, faster model to serve. Llama 2 7B saw 2 trillion tokens, about 286 tokens per parameter, more than fourteen times the Chinchilla ratio. That is why today's 7-to-30-billion-parameter models are good enough to run on a single GPU or even a laptop, and the cost logic behind it reappears in Chapter 12.12.3.
Do abilities "emerge" suddenly?
A famous claim of the early 2020s was that some abilities, such as multi-step arithmetic, appear suddenly once a model passes a certain size: absent at 10 billion parameters, present at 100 billion. That is called emergence. A 2023 analysis by Rylan Schaeffer and colleagues, titled "Are Emergent Abilities of Large Language Models a Mirage?", showed that much of the suddenness came from how ability was measured. If you score multi-digit addition as all-or-nothing (the whole answer must be exactly right), steady improvement in each digit looks like a jump from zero to success. Score it digit by digit and the curve is smooth. The fair summary is that some apparent jumps are measurement artefacts and some sharpness may be real, and that "abilities appear by magic at scale" was an overstatement.
6. Context windows: how much the model can read at once
A model is trained on sequences up to a fixed length, its context window: the maximum number of tokens it can take in at once, prompt and output together. Windows grew from about 2,000 tokens in 2020 to hundreds of thousands, and in some models over a million, using the position tricks of 12.7.4 and the memory-saving attention of 12.7.8. A large window is useful, but three practical limits remain.
Cost and speed grow with what you send. Every token in the prompt is processed on every request, and attention's work grows with the square of the length. Sending 200,000 tokens of documents with every question is slow and expensive, however large the window allows.
Models do not read every part equally well. A 2023 study by Nelson Liu and colleagues, titled "Lost in the Middle", placed a needed fact at different positions in a long context and found that models answered best when the fact was near the start or the end, and noticeably worse when it was buried in the middle. Models have improved since, but the practical rule stands: put the most important material near the beginning or the end.
The usable length is shorter than the advertised one. Accuracy on tasks that need the whole context often starts to decline well before the stated maximum.
This is why retrieval, Chapter 12.9, did not become obsolete when windows grew. Finding the right 3,000 tokens and sending only those is usually cheaper, faster and more accurate than sending 300,000.
7. What this means for an AI engineer
You will almost certainly never pretrain a large model. But pretraining explains things you will deal with every week:
- The model knows what was in its data, up to a date. Anything after its training cutoff, and anything private to Cartwise, it does not know. That is a large part of why retrieval exists.
- Its strengths follow its data. Languages and domains well represented in pretraining work better; underrepresented ones, like some Indian languages or Cartwise's internal jargon, work worse.
- Scaling explains the model menu. Providers offer large, slow, capable models and small, fast, cheaper ones; choosing between them per task is a core skill, covered in 12.12.1.
- Long context is a tool, not a substitute for choosing what to send.
Recall
- Pretraining trains on next-token prediction over trillions of tokens; every position predicts its next token in one parallel pass (the causal mask prevents cheating), with cross-entropy loss. The result is a base model, which continues text but is not yet an assistant.
- Predicting varied text well requires knowledge of facts, arithmetic, code and people, so minimising next-token error builds those capabilities.
- Data: web crawls plus code, books and papers; heavy filtering and deduplication; proportions matter. Llama 2 used about 2 trillion tokens, Llama 3 over 15 trillion. Copyright and finite human text are open problems.
- Compute ≈ 6 × parameters × tokens: about 8.4 \times 10^{22} operations, roughly 186,000 A100-hours, for Llama 2 7B (Meta reported 184,320).
- Scaling laws are power laws: each 10× gives a steady fractional gain, so small runs predict big ones. Chinchilla's compute-optimal ratio is about 20 tokens per parameter; practice over-trains smaller models (Llama 2 7B at about 286) because serving is paid forever. "Emergence" is partly a measurement artefact.
- A context window is the most a model can read at once; cost, "lost in the middle" and usable-versus-advertised length keep retrieval essential.
Self-test: How many training examples does a 1,000-token document provide, and why can they be computed in one pass? Estimate the training compute for a 13-billion-parameter model on 2 trillion tokens. If loss falls by 16 percent per tenfold increase in parameters, roughly how much lower is the loss of a model 100 times larger?
Next: 12.7.6 Inference, the KV Cache and Batching — training happens once; inference happens on every request, and its two very different phases, prefill and decode, explain most of what you pay for and wait for.