Appearance
12.7.8 — Smaller and Faster Models
Cartwise's platform team compared two ways of running a capable open 70-billion-parameter model for the support assistant. In standard 16-bit format, its weights alone are 140 gigabytes, so it needs at least two of the largest data-centre GPUs before a single customer is served. The same model in a 4-bit format is 35 gigabytes, fits on one GPU with room for the KV cache, and, surprisingly, answers faster. In the support team's blind comparison of a few hundred real conversations, they could barely tell the two apart.
That is one of several techniques that make large models cheaper and faster to run, and they all follow from one fact established in 12.7.6: generating each token means reading the model's weights from memory, and that reading, not the arithmetic, is usually the bottleneck. So the levers are:
- Fewer bytes per weight: quantisation.
- Fewer weights used per token: mixture of experts.
- More tokens per expensive step: speculative decoding.
- A smaller model that learned from a bigger one: distillation.
- Less memory traffic inside attention: FlashAttention.
1. Quantisation: store each weight in fewer bits
Weights are trained as 16-bit floating-point numbers. Quantisation stores them with fewer bits, typically 8 or 4, and sometimes 3 or 2. The memory arithmetic is the whole first half of the story:
| Bits per weight | 7-billion model | 70-billion model |
|---|---|---|
| 16 (bfloat16) | 14 GB | 140 GB |
| 8 | 7 GB | 70 GB |
| 4 | 3.5 GB | 35 GB |
| 3 | 2.6 GB | 26 GB |
| 2 | 1.75 GB | 17.5 GB |
The second half is speed. Since each decode step must read every weight, halving the bytes roughly halves the time spent reading, so a quantised model is usually faster as well as smaller. That is why the 4-bit model in the opening answered faster.
How it works: a scale and some rounding
The simplest scheme gives a group of weights one shared scale factor and stores each weight as a small whole number. Take four weights: (-0.62,\ 0.11,\ 0.48,\ -0.05).
8-bit. Whole numbers from −128 to 127 are available. Choose the scale so the largest weight maps to the edge: \text{scale} = 0.62 / 127 \approx 0.00488. Divide each weight by the scale and round: (-127,\ 23,\ 98,\ -10). These four small integers, plus the one scale, are what gets stored. To use them, multiply back: (-0.620,\ 0.112,\ 0.478,\ -0.049). The largest error is about 0.002. That is why 8-bit quantisation is essentially lossless for nearly every model.
4-bit. Only whole numbers from −8 to 7 are available, so the scale is 0.62 / 7 \approx 0.0886, and the weights become (-7,\ 1,\ 5,\ -1), which multiply back to (-0.620,\ 0.089,\ 0.443,\ -0.089). The largest error is now about 0.04: every weight has been rounded to one of only 16 levels.
2-bit. Only four levels. The weights become (-1,\ 0,\ 1,\ 0) times 0.62, so 0.11 and −0.05 both collapse to zero and 0.48 becomes 0.62. The largest error is 0.14, nearly a quarter of the biggest weight.
Real methods are smarter than this in three ways. Small groups: instead of one scale for a whole matrix, each group of 32 to 128 weights gets its own scale, so one unusually large weight only spoils the precision of its own group. Protecting what matters: methods called GPTQ (2022) and AWQ (activation-aware weight quantisation, 2023) use a small sample of real text to find which weights affect the outputs most, and round those more carefully. Better level spacing: weights cluster near zero in a bell shape, so the NF4 (4-bit NormalFloat) format used in QLoRA, Chapter 12.11.2, puts its 16 levels closer together near zero, where most weights live, and further apart in the tails.
You will also meet GGUF, a file format from the open-source llama.cpp project, which runs quantised models on ordinary processors and Apple laptops. Its quantisation types have names like Q4_K_M or Q3_K_S: the number is roughly the bits per weight, and the letters name the grouping scheme and how much precision is kept for the most sensitive layers.
What quality to expect
- 8-bit: essentially no loss.
- 4-bit with a good method: a small loss, often not noticeable in ordinary use. The most common way to run open models locally.
- 3-bit: a noticeable loss for smaller models; often acceptable for very large ones, which have more redundancy to spare.
- 2-bit: large losses for most models with ordinary methods, although specialised research methods have pushed it further.
The loss is not evenly spread. Arithmetic, code and long careful reasoning usually suffer before casual conversation does. Always evaluate a quantised model on your own tasks with the methods of Chapter 12.8.6 before switching.
"2-bit to 3-bit migrations" and other terms you may hear
In conversations about local and cost-sensitive deployments you will hear people talk about "moving from a 2-bit to a 3-bit build" of a model. That simply means switching to a less aggressive quantisation of the same model, usually because the 2-bit version made too many mistakes, accepting about 50 percent more memory in exchange for quality. Labels like "T4" in such discussions almost always refer to hardware: the NVIDIA T4 is a common, inexpensive 16-gigabyte inference GPU, often the cheapest cloud option, and choosing a quantisation level that fits a model onto one T4 is a typical constraint (12.12.3 covers GPU tiers). There is no standard industry term "T3" or "B2" in quantisation; if you hear them, ask, because they are usually a team's internal names for a test configuration. One genuinely different idea worth knowing by name is BitNet b1.58, a 2024 research model trained from the start with every weight restricted to −1, 0 or +1, which needs about 1.58 bits per weight; that is training a low-bit model from scratch rather than quantising a finished one.
Quantising the KV cache as well, storing keys and values in 8 bits, is often worth doing separately, since at long contexts the cache can be larger than the weights.
2. Mixture of experts: use only part of the model for each token
A normal "dense" model uses every one of its weights for every token. A mixture of experts model, or MoE, replaces the feed-forward layer of each transformer block from 12.7.4 with many separate feed-forward networks, the experts, plus a small router that decides, for each token, which few experts to use.
A common setup has 8 experts with the top 2 chosen per token. The router is a small layer that scores every expert for the current token's vector; softmax over the top scores gives the mixing weights, here 0.6 and 0.4. Only the chosen two compute anything.
The appeal is the gap between total parameters and active parameters. Mistral AI's Mixtral 8x7B, released in 2023, has about 46.7 billion parameters in total but uses only about 12.9 billion for each token. DeepSeek-V3, released in late 2024, has 671 billion in total and about 37 billion active. Roughly speaking, what the model knows grows with its total parameters, while the compute per token grows only with its active parameters. You get much of the capability of a large model at the per-token cost of a much smaller one, which is why many frontier models are now believed to be MoE designs.
The costs are specific. Memory follows the total: every expert must be loaded, because the next token may need any of them, so Mixtral needs memory for 46.7 billion parameters even though it computes with 12.9. Routing must be balanced: left alone, the router tends to send most tokens to a few favourite experts, which then get most of the training while the others stay weak; an extra term in the training loss pushes tokens to spread evenly. And when experts live on different GPUs, routing each token to the right place becomes a communication problem. Contrary to the tempting picture, experts do not neatly specialise in topics like "finance" or "code"; analyses find their specialisation is mostly at the level of token types and patterns, not subjects.
3. Speculative decoding: guess several tokens, check them at once
Decode is slow because each step reads all the weights to produce one token, while the arithmetic sits idle. Speculative decoding uses that idle arithmetic. A small, fast draft model, perhaps a hundred times smaller, quickly proposes the next few tokens, say four. The big target model then checks all four in a single forward pass, the same way prefill processes many tokens at once. It accepts the longest beginning of the draft that matches what it would have produced, and adds one token of its own.
If the draft proposed "will arrive by Thursday" and the big model agrees with "will arrive by" but would have said "Friday", it keeps the three matching tokens and adds "Friday": four tokens for the price of one expensive step plus a few cheap ones. When the draft is wrong at the first token, you lose very little. On typical text, where many tokens are predictable ("the", the end of a common phrase, closing brackets in code), the big model accepts several tokens per step and generation is commonly two to three times faster.
The remarkable property, shown independently in 2023 by Yaniv Leviathan and colleagues at Google and by Charlie Chen and colleagues at DeepMind, is that with the right acceptance rule, a form of the rejection sampling from statistics, the output has exactly the same probability distribution as the big model alone. Speculative decoding changes the speed, not the answers. Variants build the draft from the big model's own extra prediction heads instead of a separate model.
4. Distillation: teach a small model with a big one
Sometimes the best option is a smaller model that has been trained to imitate a larger one. That is knowledge distillation, popularised by Geoffrey Hinton, Oriol Vinyals and Jeff Dean in a 2015 paper. A large teacher model produces outputs, and a small student model is trained to match them.
The key idea is what the student learns from. Ordinary training gives it the hard answer: the next token was "Paris". Distillation gives it the teacher's whole probability distribution, the soft targets: say 0.968 for "Paris", 0.020 for "Lyon", 0.010 for "Marseille", 0.002 for others. Those small numbers carry information the hard label does not: that Lyon is a much more plausible mistake than a random word, which tells the student something about how the teacher organises knowledge. The paper also softened the teacher's distribution with a temperature above 1, from 12.2.6, to make those small probabilities larger and easier to learn from: at temperature 2, the same logits give about 0.774, 0.110, 0.078 and 0.039.
In the LLM era, distillation is also done more simply: generate a large set of high-quality answers with a big model and fine-tune a small model on them. Many small open models were trained partly this way, which is one reason a current 8-billion-parameter model can outperform much larger models from a few years earlier. For Cartwise, it is a practical route to cost savings: if the big model handles email routing well, its labelled outputs can train a tiny classifier that does the same job for a fraction of the cost, the pattern Chapter 12.11.3 works through. (Note that providers' terms of service often restrict using their models' outputs to train competing models, so check before doing this with a hosted model.)
5. FlashAttention: the same attention, less memory traffic
Attention, as 12.7.3 showed, builds an n \times n table of scores. Done naively, that table is written out to the GPU's main memory, read back for the softmax, written again, and read again to multiply by the values. For long sequences, that is enormous memory traffic, and memory traffic is exactly what GPUs are short of.
FlashAttention, by Tri Dao and colleagues in 2022, reorganises the computation into tiles small enough to fit in the GPU's tiny but extremely fast on-chip memory. It computes the softmax piece by piece, using a running maximum and running total so that the pieces combine into exactly the right answer, and never writes the full table to main memory at all. It is not an approximation: the result is the same attention, to within floating-point rounding. It is much faster and uses memory proportional to n instead of n^2, and it is one of the main reasons long context windows became practical. It is built into PyTorch's attention function and every serious serving engine, so you get it without asking.
6. Choosing among the levers
| Technique | What it reduces | Changes the answers? | Typical use |
|---|---|---|---|
| 8-bit quantisation | memory, read time | barely | almost always safe |
| 4-bit quantisation | memory ×4, read time | slightly | local and single-GPU serving |
| Mixture of experts | compute per token | it is the model's design | chosen by the model builder |
| Speculative decoding | latency | no | serving engines, interactive apps |
| Distillation | everything (smaller model) | yes, a new model | high-volume narrow tasks |
| FlashAttention | attention memory, time | no | always on |
For an AI engineer using hosted APIs, most of this is done by the provider and shows up as the price and speed of the model you choose. For anyone serving open models, quantisation and a good serving engine are the first two decisions, and Chapter 12.11.4 works through when self-hosting is worth it at all.
Recall
- Decode is bound by reading weights, so the levers are fewer bytes per weight, fewer weights per token, more tokens per step, smaller models and less attention memory traffic.
- Quantisation stores weights as small integers with a scale factor: 8-bit is essentially lossless (error about 0.002 in the example), 4-bit a small loss, 2-bit collapses distinct weights. Group-wise scales, GPTQ, AWQ and NF4 improve it; GGUF names like
Q4_K_Mgive rough bits per weight. Quantised models are faster as well as smaller. Evaluate on your own tasks. - "Moving from 2-bit to 3-bit" means a less aggressive quantisation for better quality; "T4" usually means the 16 GB NVIDIA T4 GPU.
- Mixture of experts uses a router to pick a few experts per token: total parameters set knowledge, active parameters set compute (Mixtral 8x7B: 46.7 billion total, 12.9 billion active), but memory follows the total and routing needs balancing.
- Speculative decoding drafts tokens with a small model and verifies them in one big-model pass: two to three times faster with an identical output distribution. Knowledge distillation trains a small student on a teacher's soft probabilities or outputs. FlashAttention computes exact attention in tiles, using memory proportional to n.
Self-test: Quantise 0.30 to 4 bits using the scale from the worked example; what value comes back? Why does an MoE model with 12.9 billion active parameters still need memory for 46.7 billion? Why can speculative decoding never make the output worse?
Next: 12.7.9 From Predictor to Assistant — every model on these pages, however fast, is still a text predictor; the last page of the chapter covers how instruction tuning, human feedback and reasoning training turn it into an assistant you can give a job to.