12.8 — Fine-Tuning, LoRA and Alignment

"We'll fine-tune the model on our documentation."

This is the most common first instinct and usually the wrong one. Fine-tuning teaches a model how to behave; it is a poor way to teach it what is true. A model fine-tuned on your handbook will adopt its tone and will not reliably recall its contents, cannot cite a source, cannot be updated when the handbook changes, and cannot hide a document from a user who should not see it. Retrieval does all four (Chapter 12.6.2).

So start with the decision, then the mechanics.

1. When fine-tuning is the right answer

Consistent format or style. You need every output in a house voice, or in a specific structure, every time. Prompting gets you most of the way; fine-tuning gets the last part and removes the instructions from every request.

A narrow task done extremely well. Classifying support tickets into your 40 categories, extracting fields from your document type, converting your natural-language queries into your query language. A small fine-tuned model routinely beats a much larger prompted one on a narrow task, and this is the strongest business case.

Cost and latency. If a 7B fine-tune matches a frontier model on your task, the saving at volume is enormous. This is the reason most production fine-tunes exist.

Behaviour that resists prompting. Reliable tool use in a specific pattern, an unusual output format, a domain style the model keeps drifting away from.

A domain the model barely saw. A rare language, a specialised notation, an internal query language.

And when not to:

• To add knowledge. Use retrieval.
• When facts change. Retraining is not an update mechanism.
• When you need attribution. A fine-tune cannot cite.
• With fewer than a few hundred good examples. You will overfit and gain nothing.
• Before trying prompting and retrieval properly. Fine-tuning is expensive to iterate on; a prompt change takes seconds.

2. Why full fine-tuning is out of reach

Training memory is not the weights. For mixed-precision training with Adam, per parameter:

• Weights in bf16 — 2 bytes
• Gradients in bf16 — 2 bytes
• Optimiser: an fp32 master copy, plus momentum and variance — 12 bytes

Roughly 16 bytes per parameter, before activations.

Model	Weights only (bf16)	Full fine-tuning
7B	14 GB	~112 GB
70B	140 GB	~1.1 TB

A 7B model needs several high-end accelerators to fine-tune fully, and that is before the activation memory that scales with batch size and sequence length. That gap is why parameter-efficient methods exist, and they are not a compromise — they are how fine-tuning is done.

3. LoRA

LoRA (low-rank adaptation) freezes the original weights and learns a small correction.

The observation behind it: when you adapt a model to a task, the change to each weight matrix is low-rank — it does not need the full expressive freedom of the matrix. So represent the update as a product of two thin matrices:

W' = W + \Delta W = W + BA

where W is d \times k and frozen, A is r \times k, B is d \times r, and r is small — typically 8 to 64.

The arithmetic is the whole argument. For a 4096 \times 4096 matrix:

• Full: 4096^2 = 16.8 million parameters.
• LoRA at r = 8: 8 \times 4096 \times 2 = 65{,}536 — 0.4%.

Across a whole model, LoRA typically trains 0.1–1% of the parameters, and because optimiser state is only kept for trainable parameters, the memory saving is far larger than the parameter saving.

Four implementation details that matter:

A is initialised randomly and B to zero, so BA = 0 at the start and the model begins exactly as the base model — training starts from a known-good point rather than a perturbed one.

A scaling factor \alpha/r multiplies the update, so changing the rank does not require retuning the learning rate. Common practice is \alpha = 2r.

Where to apply it. Originally the attention query and value projections; applying it to all linear layers, including the feed-forward ones, generally works better and costs a little more.

It merges at inference. Compute W + BA once and you have an ordinary weight matrix — zero added latency, unlike adapter layers, which insert extra computation. Or keep it separate and serve many adapters against one base model in memory, switching per request, which is how a provider offers per-customer fine-tunes economically.

4. QLoRA

QLoRA fine-tunes a 4-bit quantised base model with LoRA adapters in higher precision, and it is what put fine-tuning on single machines.

Three pieces:

• NF4, a 4-bit data type designed for the roughly normal distribution of neural network weights, which loses less than a uniform 4-bit format.
• Double quantization — quantise the quantisation constants too, saving a further fraction of a bit per parameter.
• Paged optimisers — move optimiser state to CPU memory when a memory spike would otherwise fail the run.

The base model is frozen, so quantising it costs little — the gradients flow through it to the adapters, which stay in 16-bit. A 65B model becomes fine-tunable on a single 48 GB accelerator, and quality is close to 16-bit LoRA.

5. The LoRA family

Each variant targets one specific cost, and the differences are worth knowing precisely because they are frequently confused.

LoRA-FA — freeze A after random initialisation and train only B. Roughly halves the optimiser memory with little quality loss, because much of the adaptation is expressible through B alone.

VeRA — go further: use a single pair of random frozen matrices shared across all layers, and learn only small per-layer scaling vectors. Trainable parameters drop by another order of magnitude. Useful when you need many adapters — thousands of per-user personalisations — where storage per adapter is the constraint.

LoRA+ — a one-line change with a real effect: use a higher learning rate for B than for A (often 16×). The argument is that the two matrices sit at different scales in the update, so a single learning rate is wrong for one of them. Faster convergence and better final quality at no extra cost.

Delta-LoRA — also update the frozen W using the change in BA between steps, so the base weights move a little too. Narrows the gap to full fine-tuning without storing optimiser state for W.

DoRA — decompose each weight into a magnitude and a direction, and apply LoRA only to the direction while learning the magnitude separately. Closer to full fine-tuning behaviour, at a small extra cost.

Practical guidance: start with plain LoRA at r=16, \alpha=32, on all linear layers. Add QLoRA if it does not fit. Reach for the variants when you have a specific constraint — optimiser memory, adapter count, or a measured quality gap.

6. Data

Quality dominates quantity, and by a lot. A thousand carefully constructed examples beat a hundred thousand scraped ones. The published result that shaped practice is LIMA: 1,000 curated examples produced a competitive assistant, on the argument that instruction tuning mostly teaches format and style, and the capability is already in the base model.

Rules that decide whether a run works:

Match the target distribution. The training data should look like production inputs — same length, same messiness, same edge cases. A model trained on tidy examples fails on real ones.

Include the hard cases. Ambiguity, refusals, missing information. If you never show it what to do when the answer is not available, it will invent one.

Deduplicate. Near-duplicates cause memorisation and inflate your evaluation scores.

Balance. A dataset that is 80% one class biases the output toward it.

Hold out a test set before you start, and check for contamination — the same examples appearing in training and evaluation is the most common cause of a fine-tune that scores brilliantly and fails in production.

Format exactly as the base model expects. Chat models are trained on a specific template of role markers. A mismatched template is a silent, large quality loss with no error message, and it is one of the most common fine-tuning mistakes.

7. Running it

Learning rate: around 1\text{e-}4 to 2\text{e-}4 for LoRA — much higher than full fine-tuning's 1\text{e-}5, because you are training few parameters from a zero-initialised start.

Epochs: 1 to 3. More than three usually overfits, and the sign is unmistakable — validation loss rising while training loss falls (Chapter 12.4.2).

Batch size: as large as memory allows, with gradient accumulation to simulate more.

Watch for catastrophic forgetting. A model fine-tuned hard on one task gets worse at everything else. Always evaluate on general capability as well as your task, or you ship a model that is excellent at classification and can no longer hold a conversation. Lower learning rates, fewer epochs and mixing in some general data all reduce it.

Evaluate against the base model on your task, with the same prompt. A fine-tune that does not clearly beat a good prompt is not worth the operational weight.

8. Alignment: from a text predictor to an assistant

A pre-trained model completes text. It does not answer questions, refuse harmful requests or follow instructions. Three stages turn it into an assistant.

Stage 1 — supervised fine-tuning. Train on demonstrations: prompts with high-quality human-written responses. This teaches the format of being an assistant — that a question gets an answer rather than more questions.

Stage 2 — a reward model. Humans rank several responses to the same prompt from best to worst. A model is trained to predict those rankings. Ranking is used rather than scoring because people are far more consistent at "which is better" than at "rate this out of ten" — the same finding as Chapter 12.6.3's evaluation section.

Stage 3 — reinforcement learning. Optimise the model to maximise the reward model's score, with a penalty for drifting too far from the supervised model. That penalty is essential: without it, the model finds outputs that score highly and are degenerate — the classic reward hacking failure, where it produces text the reward model loves and humans find useless.

Together this is RLHF, and it is what turned a capable text predictor into something usable.

DPO replaced most of it. Direct preference optimisation showed that you can skip the reward model and the reinforcement learning loop entirely: train directly on preference pairs with a loss that increases the likelihood of the preferred response and decreases the other, with a term that keeps the model near its starting point. Simpler, more stable, far less compute — and it is now the common choice, with variants (IPO, KTO, ORPO) adjusting the objective or the data requirements. KTO is notable for needing only "good" or "bad" labels rather than pairs, which is much cheaper to collect.

RLAIF and constitutional methods replace human preference labels with model-generated ones, guided by a written set of principles. It scales far beyond what human labelling can, and its quality depends on the labelling model and the principles being right.

Three honest costs of alignment:

Sycophancy. Models trained on human preferences learn that agreement is preferred, so they tend to concede when pushed even when they were right. It is a direct consequence of the training signal.

The alignment tax. Alignment can reduce raw capability on some benchmarks. Modern methods have narrowed it and not eliminated it.

Preferences are not truth. A reward model learns what raters liked — which correlates with confidence, length and formatting as well as correctness. Some of what looks like a knowledge failure is a preference artefact.

9. Distillation

Train a small model to reproduce a large model's outputs. The student learns from the teacher's full probability distribution, which carries more information than a hard label — the teacher's uncertainty between two plausible answers is itself a signal.

This is now the standard way capable small models are produced, and it is why 7B models today outperform much larger models from a few years ago. For a specific task it is often the best available option: generate a few thousand high-quality outputs from a frontier model, fine-tune a small open model on them, and serve that.

Check the licence. Many providers' terms restrict using their outputs to train competing models, and this is a real constraint rather than a formality (Chapter 8.7).

10. Shipping a fine-tune

Version everything — base model, data, hyperparameters, adapter — and record which version produced which output.

Evaluate on your task set and on general capability, against the base model with a good prompt, before and after.

Keep the base model as a fallback. If the fine-tune degrades on a case class you did not anticipate, you need a switch.

Plan for the base model being deprecated. A fine-tune is tied to its base, and providers retire models. Budget for redoing it, and keep the data and pipeline reproducible so that redoing it is a job rather than a project.

Recall

• Fine-tuning teaches behaviour; retrieval teaches facts. Fine-tune for format, a narrow task, or cost and latency — not to add knowledge, not when facts change, and not with fewer than a few hundred good examples.
• Full fine-tuning costs about 16 bytes per parameter (weights, gradients, and Adam's fp32 copy plus two moments) — roughly 112 GB for a 7B model. That gap is why PEFT exists.
• LoRA: W' = W + BA with small rank r. At r=8 a 4096² matrix trains 0.4% of its parameters. B starts at zero so training begins from the base model, \alpha/r scales the update, all linear layers is better than attention-only, and merging gives zero inference latency — or keep adapters separate and serve many against one base.
• QLoRA = 4-bit NF4 base + 16-bit adapters + double quantization + paged optimisers, putting a 65B fine-tune on one 48 GB device.
• The family, each targeting one cost: LoRA-FA (freeze A, halve optimiser memory), VeRA (shared frozen random matrices, learn tiny scaling vectors — for thousands of adapters), LoRA+ (higher learning rate for B), Delta-LoRA (also nudge W), DoRA (split magnitude from direction).
• Data quality dominates quantity — 1,000 curated examples beat 100,000 scraped. Match the production distribution, include hard cases and refusals, deduplicate, hold out before starting, check contamination, and use the base model's exact chat template or you lose quality silently.
• LoRA learning rate ~1\text{e-}4, 1–3 epochs, and always evaluate general capability too or you ship catastrophic forgetting.
• Alignment: SFT → reward model from rankings → RL with a drift penalty (without which you get reward hacking). DPO removed the reward model and the RL loop and is now the common choice. Costs: sycophancy, the alignment tax, and preferences that reward confidence and length rather than truth.
• Distillation is how capable small models are made — and provider terms often restrict training on their outputs.

Self-test: Why does fine-tuning on documents fail as a knowledge strategy? · Where does 16 bytes per parameter come from? · Why is B initialised to zero? · What does LoRA-FA save, and what does VeRA save? · Why are human rankings used rather than ratings? · What does DPO remove from RLHF, and what does it keep?