12.11.2 — LoRA and Parameter-Efficient Fine-Tuning

Cartwise's ticket classifier from 12.11.1 was to be a fine-tuned open-weights model of about 7 billion parameters. Chapter 12.5.7 did the memory arithmetic for exactly this size: training with the Adam optimiser stores about 16 bytes per parameter (the weight, its gradient, and Adam's running averages), so the 6.74 billion parameters of a typical 7B model need about 108 gigabytes of training state, before counting activations. Running the same model, without training it, needs only its 16-bit weights: about 13.5 gigabytes. Full fine-tuning needs a cluster of large GPUs; inference needs one modest one.

This page is about the idea that closed that gap, so that a 7B model can be fine-tuned on a single GPU of the kind that serves it. The idea is called LoRA, and to understand why it works you need one concept from linear algebra, the rank of a matrix, which we build first.

1. Rank: how much information a matrix really holds

Every weight matrix in a transformer is a grid of numbers. One of the attention matrices in a 7B model is 4,096 rows by 4,096 columns: about 16.8 million numbers. But a grid of numbers can hold much less information than its size suggests.

Take two short lists of numbers, a column u = (1, 2, 3) and a row v = (2, 0, 1), and build a 3-by-3 matrix by multiplying every entry of u by every entry of v. This is called the outer product, written u v^\top ("u times v transposed"):

u v^\top = \begin{pmatrix} 1 \\ 2 \\ 3 \end{pmatrix} \begin{pmatrix} 2 & 0 & 1 \end{pmatrix} = \begin{pmatrix} 2 & 0 & 1 \\ 4 & 0 & 2 \\ 6 & 0 & 3 \end{pmatrix}

Row 1 is 1 \times (2, 0, 1), row 2 is 2 \times (2, 0, 1), row 3 is 3 \times (2, 0, 1). Every row is a multiple of the same row. The matrix has nine numbers, but you can rebuild all nine from six: the three in u and the three in v. A matrix that is one outer product like this is said to have rank 1.

Add two such products together, u_1 v_1^\top + u_2 v_2^\top, and you get a matrix of rank 2, rebuildable from four short lists. In general, a matrix has rank r if it can be written as a sum of r outer products and no fewer. Stack the r columns u_1, \dots, u_r side by side into a tall, thin matrix B, and the r rows v_1, \dots, v_r on top of each other into a short, wide matrix A, and the sum is exactly the product BA:

\underbrace{M}_{d \times d} = \underbrace{B}_{d \times r} \; \underbrace{A}_{r \times d}

Read it as "M, which is d by d, equals B, which is d by r, times A, which is r by d". The point is the count. M has d \times d numbers. B and A together have d \times r + r \times d = 2dr. For d = 4{,}096 and r = 16, that is 4{,}096^2 = 16{,}777{,}216 numbers against 2 \times 4{,}096 \times 16 = 131{,}072, less than 1%, and yet BA is a full-size 4,096-by-4,096 matrix. A low-rank matrix is a big matrix that can be described with very few numbers.

2. The observation that made LoRA possible

Fine-tuning changes each weight matrix W by some amount, giving W + \Delta W, where \Delta W (read "delta W", the change in W) is the same size as W. Full fine-tuning learns every number of \Delta W freely. The question that led to LoRA was: does the change really need all those numbers?

In 2020, Armen Aghajanyan, Luke Zettlemoyer and Sonal Gupta at Facebook AI measured this. They took a pretrained language model and fine-tuned it while allowing the weights to move only within a small random subspace: a few hundred numbers, mapped by a fixed random projection onto all the model's weights. On one standard task, a model with over a hundred million parameters reached 90% of its full fine-tuning performance while training only about 200 numbers. They called the smallest number of directions a task needs its intrinsic dimension, and found it was tiny for most tasks, and smaller still for larger, better-pretrained models.

The intuition is the one from 12.11.1: pretraining already built nearly everything the model needs, and fine-tuning teaches it to use what it has in a particular way. Adjusting how existing abilities are combined is a small change, even though it touches large matrices. In the language of section 1, the useful \Delta W has low rank.

3. LoRA

In 2021, Edward Hu and colleagues at Microsoft turned that observation into a method, Low-Rank Adaptation. Instead of learning \Delta W directly, LoRA learns it already in the low-rank form BA, and leaves the original weights untouched.

One LoRA layer. The input goes through the frozen original matrix as before, and also through a narrow side path: A squeezes it to 16 numbers, B expands it back. Only A and B are trained, and the side path's output is added to the original.

For a layer whose original computation was h = W_0 x, LoRA computes

h = W_0 x + \frac{\alpha}{r} B A x

Read it piece by piece. x is the layer's input, a list of 4,096 numbers. W_0 ("W nought") is the original pretrained matrix, frozen: it is used but never changed, so it needs no gradient and no optimiser state. A is a trainable r \times d matrix that squeezes the input down to r numbers, and B is a trainable d \times r matrix that expands those r numbers back to d. Their product BA is the low-rank change \Delta W of section 2. The rank r, typically 8 to 64, is a setting you choose. And \alpha / r ("alpha over r") is a scaling factor, where \alpha is another setting, explained below. The figure shows the two paths: the old one, unchanged, and the new narrow one added beside it.

Four details make it work in practice.

B starts at zero. A is filled with small random numbers and B with zeros, so at the start of training BA = 0 and the layer computes exactly W_0 x. The fine-tuned model begins identical to the pretrained one and moves away from it only as far as training pushes it. Starting both at random would add random noise to every layer on the first step, damaging the abilities you are trying to build on.

The scale \alpha / r keeps the size of the update stable when you change r. Without it, doubling the rank roughly doubles how much the side path contributes, and you would have to find a new learning rate every time you tried a different rank. A common choice is \alpha = 2r, which makes the scale 2.

Which matrices get adapters. The original paper adapted only two of the four attention matrices (the query and value projections of 12.7.3). Later practice, encouraged by the QLoRA work below, adds adapters to every linear layer, attention and feed-forward alike, which costs more parameters but usually gives better results.

Merging. After training, you can compute W_0 + \frac{\alpha}{r} BA once and save it as an ordinary matrix. The merged model has exactly the shape of the original, so it runs at exactly the original speed. Or you can keep the adapter separate, which section 5 shows is often more useful.

The numbers

For a typical 7B model (32 layers, width 4,096, feed-forward width 11,008), the Hugging Face PEFT library sets up LoRA in a few lines:

from transformers import AutoModelForCausalLM
from peft import LoraConfig, get_peft_model

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf", torch_dtype="bfloat16")

config = LoraConfig(r=16, lora_alpha=32,                        # (1) rank 16, scale 32/16 = 2
                    target_modules=["q_proj", "v_proj"],         # (2) the original paper's choice
                    lora_dropout=0.05, task_type="CAUSAL_LM")
model = get_peft_model(model, config)                            # (3) freezes everything else
model.print_trainable_parameters()
# trainable params: 8,388,608 || all params: 6,746,804,224 || trainable%: 0.1243

Line (1) sets the rank and \alpha. Line (2) names the layers that get adapters, here the query and value projections in each of the 32 layers. Line (3) wraps the model: it adds an A and a B beside each named layer and freezes every original weight. The printed line confirms the arithmetic: each adapter has 16 \times (4{,}096 + 4{,}096) = 131{,}072 numbers, two per layer, 32 layers, giving 131{,}072 \times 64 = 8{,}388{,}608, which is 0.12% of the model.

Changing line (2) to target_modules="all-linear" adds adapters to all seven linear layers in each block (four attention, three feed-forward). The count rises to 39,976,960 trainable parameters, about 0.59% of the model. Their training state at 16 bytes each is 39{,}976{,}960 \times 16 \approx 0.64 gigabytes. Add the frozen 16-bit weights, about 13.5 GB, and the activations, and training fits comfortably on a single 24 GB GPU, against roughly 108 GB of state for full fine-tuning.

4. QLoRA: a frozen base in 4 bits

LoRA removed the training state for the frozen weights, but the frozen weights themselves still had to sit in memory, 13.5 GB in 16-bit form. In May 2023, Tim Dettmers and colleagues at the University of Washington published QLoRA (quantised LoRA), which stores the frozen base in 4 bits per weight, using the quantisation of 12.7.8, and trains ordinary 16-bit LoRA adapters on top. The frozen 7B base shrinks to about 3.4 GB.

Three inventions made this work without losing quality. NF4, the "4-bit NormalFloat" format, places its 16 possible values closer together near zero, where most weights cluster, as 12.7.8 described. Double quantisation quantises the scale factors themselves, because with one scale per small group of weights the scales add up to a noticeable amount of memory. And paged optimisers move optimiser state to ordinary computer memory when a sudden spike of activation memory would otherwise crash the GPU, and bring it back afterwards. The paper's headline result was fine-tuning a 65-billion-parameter model on a single 48 GB GPU, with results matching 16-bit fine-tuning.

Loading a model for QLoRA is one extra configuration object:

import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

bnb = BitsAndBytesConfig(load_in_4bit=True,
                         bnb_4bit_quant_type="nf4",                # NormalFloat 4-bit
                         bnb_4bit_use_double_quant=True,           # quantise the scales too
                         bnb_4bit_compute_dtype=torch.bfloat16)    # do the arithmetic in 16-bit
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf", quantization_config=bnb)

The last setting is worth understanding. The weights are stored in 4 bits, but each one is converted back to 16 bits just before it is used in a multiplication, so the arithmetic itself keeps 16-bit precision. The price is a little speed: QLoRA trains somewhat slower than LoRA because of those conversions. It is the standard choice when memory is the limit.

5. Serving adapters: the system design

LoRA's small adapters change how fine-tuned models are served, not only how they are trained. A LoRA adapter for a 7B model is tens of megabytes; the base model is gigabytes. That asymmetry allows a design that full fine-tuning never could.

Many fine-tuned models on one GPU. The large frozen base is loaded once; each fine-tune is a small adapter. Requests needing different adapters share a batch, because the expensive base computation is the same for all of them.

Instead of one GPU per fine-tuned model, one GPU holds one copy of the base model and many adapters. Each request names the adapter it needs. The server batches requests together regardless of adapter, which works because of the shape of the LoRA formula: for every request, the expensive part, W_0 x, uses the same shared matrix, and only the cheap side path BAx differs. A 2023 Berkeley system called S-LoRA showed thousands of adapters served this way from one machine, and mainstream serving engines such as vLLM now accept a LoRA adapter per request. Adapters that are not in GPU memory are loaded from storage when first needed, like pages in an operating system's virtual memory.

This is how providers offer custom fine-tuning to many customers economically, and it is what lets Cartwise run its ticket classifier, a brand-voice model and a Hindi-reply model on the same GPU. The trade-off: an unmerged adapter adds a small amount of work to every layer, so a single high-volume model that never shares its GPU is better merged.

6. The LoRA family

LoRA started a family of variants, each changing one part of the recipe to save more memory, train faster, or get closer to full fine-tuning. All of them keep the core idea, a frozen base plus a small trained change.

LoRA-FA: freeze A as well

LoRA-FA ("frozen A"), from Longteng Zhang and colleagues in 2023, initialises A randomly and then never trains it; only B learns:

h = W_0 x + \frac{\alpha}{r} B A_{\text{frozen}} x

Halving the trained parameters is the obvious saving, but the bigger one is less obvious. To compute the gradient for a trained A, backpropagation (12.5.2) needs the layer's full input x, 4,096 numbers per token, kept in memory from the forward pass. With A frozen, the gradient for B needs only z = A x, the 16-number squeezed version. Keeping 16 numbers per token per layer instead of 4,096 cuts the activation memory of the adapter paths sharply, and activations are often the largest memory cost for long inputs. The random frozen A still works because a random projection to 16 dimensions preserves enough of the input's structure for B to learn a useful update from it. The authors report quality comparable to LoRA on their benchmarks.

VeRA: share one random pair across the whole model

VeRA ("vector-based random matrix adaptation"), from Dawid Kopiczko, Tijmen Blankevoort and Yuki Asano (ICLR 2024), goes much further. It uses one pair of frozen random matrices A and B, shared by every layer, and trains only two small vectors per layer that scale them:

h = W_0 x + \Lambda_b \, B \, \Lambda_d \, A \, x

Here \Lambda_d ("lambda d") is a diagonal matrix built from a trainable vector d of length r, which scales each of the r squeezed numbers, and \Lambda_b is a diagonal matrix built from a trainable vector b of length equal to the layer's output size, which scales each output. A diagonal matrix only multiplies each entry by its own number, so these are just two lists of scaling factors. Because the shared random matrices can be regenerated from a single random seed, they need not even be stored: an adapter is just the vectors.

The rank can then be large, since it costs almost nothing. With r = 256, each 4,096-wide layer trains 256 + 4{,}096 = 4{,}352 numbers, against 131,072 for a rank-16 LoRA adapter. Over all the linear layers of the 7B model, VeRA trains about 1.4 million numbers against LoRA's 40 million, roughly 28 times fewer. The use case is the multi-adapter serving of section 5 pushed to the limit: a platform storing a separate adapter for each of a million users, where each adapter must be kilobytes rather than megabytes.

LoRA+: two learning rates instead of one

LoRA+ (Soufiane Hayou, Nikhil Ghosh and Bin Yu, 2024) changes no formula at all, only the training. It gives B a larger learning rate than A:

\eta_B = \lambda \, \eta_A

where \eta ("eta") is the learning rate from 12.2.4 and \lambda ("lambda") is the ratio between them; the authors suggest about 16 as a default. Their argument comes from analysing very wide networks: A and B play different roles in the side path, one squeezing a wide input and the other expanding a narrow one, and a single step size that is right for one is too small for the other, so the side path learns more slowly than it could. Separating them gave up to twice the training speed and 1–2% better results in their experiments, at no extra cost. In PEFT it is a different optimiser construction (create_loraplus_optimizer with loraplus_lr_ratio=16), not a different model.

Delta-LoRA: let the base move too

Delta-LoRA (Bojia Zi and colleagues, 2023) tries to close the remaining gap to full fine-tuning by letting the frozen weights change after all, but without computing gradients for them. At each step, after A and B are updated, the change in their product is also added to the base:

W^{(t+1)} = W^{(t)} + \lambda \left( B^{(t+1)} A^{(t+1)} - B^{(t)} A^{(t)} \right)

Read it as: the base weights at the next step equal the base weights now, plus \lambda times how much the low-rank product just changed. The superscript (t) means "at step t". Because this update reuses BA, which is computed anyway, it needs no gradient for W and no Adam state for it, so the memory stays close to plain LoRA's while the large matrix gets to move.

It has a consequence for the system design of section 5: the base weights change, so the result is a full new model, not a small adapter on a shared base. Delta-LoRA suits a single model you will merge and serve alone, not a platform of many adapters.

DoRA: adjust size and direction separately

DoRA ("weight-decomposed low-rank adaptation", Shih-Yang Liu and colleagues at NVIDIA, 2024) starts from an observation about how full fine-tuning changes weights. Think of each column of a weight matrix as an arrow with a length (its magnitude) and a direction. Full fine-tuning tends to change the two somewhat independently; plain LoRA changes them in a tied way. DoRA separates them:

W' = m \cdot \frac{W_0 + BA}{\lVert W_0 + BA \rVert_c}

The fraction takes each column of W_0 + BA and divides it by its own length (the \lVert \cdot \rVert_c, "column-wise norm"), leaving only its direction, which LoRA's BA adjusts. Then m, a trainable vector with one number per column, sets each column's length. The extra cost is one number per column, small beside A and B, and DoRA often comes closer to full fine-tuning than LoRA at the same rank, particularly at low ranks. In PEFT it is one option: LoraConfig(..., use_dora=True). After training it can be merged like LoRA.

Before LoRA, briefly

Three earlier ideas still appear in papers and libraries. Adapters (Neil Houlsby and colleagues at Google, 2019) inserted small extra layers inside each transformer block; they work, but add delay to every step because the new layers run in sequence and cannot be merged away. Prompt tuning and prefix tuning (2021) leave the whole model frozen and train a few dozen "virtual tokens", vectors placed before the real input. They are tiny, but generally weaker than LoRA on hard tasks. IA³ (2022) trains only scaling vectors on a few internal activations, an idea VeRA later combined with random matrices.

The family side by side

Method	Trained	Adapter size	Mergeable
LoRA	A, B	MBs	yes
QLoRA	A, B (4-bit base)	MBs	yes
LoRA-FA	B only	~half of LoRA	yes
VeRA	two vectors per layer	KBs	yes
LoRA+	A, B, different rates	same as LoRA	yes
Delta-LoRA	A, B, and base nudged	whole model	n/a
DoRA	A, B, length vector	slightly over LoRA	yes

For most projects the choice is simple: LoRA, or QLoRA if memory is tight, on all linear layers, with DoRA or LoRA+ as cheap improvements worth trying against your evaluation set. VeRA is for platforms with enormous numbers of adapters, LoRA-FA for very long inputs where activation memory dominates, and Delta-LoRA for a single model where the last bit of quality matters more than adapter flexibility.

7. Choosing the settings

A few settings matter, and sensible starting points exist for each.

Rank r. 8 to 32 for most tasks. Higher ranks can learn more complex changes but need more data to do so; a narrow task such as ticket classification rarely benefits from more than 16.

\alpha. Start with \alpha = 2r (scale 2) or \alpha = r (scale 1), and leave it fixed while you vary other things. A later refinement, rank-stabilised LoRA, scales by \alpha / \sqrt{r} instead of \alpha / r, which behaves better at high ranks.

Target layers. All linear layers, unless memory forces fewer.

Learning rate. Around 10^{-4} to 2 \times 10^{-4}, roughly ten times higher than typical full fine-tuning, because only the small adapters move and they start from zero.

Epochs. One to three passes over the data. Small fine-tuning sets overfit quickly (12.3.3), so watch the validation loss and stop when it stops falling.

None of these replaces measurement: every choice is confirmed against the evaluation set, which the next page builds for a real project.

What the interviewer will push on

"What is LoRA and why does it work?" They want three things:

1. The observation. The changes fine-tuning makes have low rank, so a small number of directions is enough.
2. The method. h = W_0 x + \frac{\alpha}{r} BAx. The base weights are frozen, and only A and B, of rank r, are trained.
3. The result. Training only has to keep state for the adapters, which are under 1% of the parameters.

Doing the 2dr versus d^2 count out loud is a strong signal.

"Why is B initialised to zero?" So that BA = 0 and the model starts exactly at the pretrained weights. Random initialisation of both would inject noise into every layer on the first step.

"What is the difference between LoRA and QLoRA?" QLoRA stores the frozen base in 4-bit NF4 with double quantisation and paged optimisers, while training 16-bit adapters; it trades a little speed for a base that is four times smaller.

"How would you serve 500 fine-tuned customer models?" One base model per GPU with many LoRA adapters loaded on demand, mixed in the same batches, because the base computation is shared and only the small side paths differ. Mention that Delta-LoRA would break this, because it changes the base.

"Compare LoRA-FA, VeRA, LoRA+, Delta-LoRA and DoRA." Give one line each, saying what changes:

• LoRA-FA freezes A. That saves activation memory.
• VeRA shares one pair of frozen random matrices and trains only small scaling vectors. The adapters become tiny.
• LoRA+ gives B a higher learning rate than A. Training gets faster.
• Delta-LoRA also nudges the base weights by the change in BA. It gets closer to full fine-tuning, but it is no longer a separate adapter.
• DoRA trains the size and the direction of the weight change separately. It gets closer to full fine-tuning and can still be merged.

Something to volunteer: the serving consequence is often worth more than the training saving. LoRA turned "one GPU per fine-tuned model" into "hundreds of fine-tuned models per GPU", which is what made per-customer fine-tuning a viable product.

Recall

• Full fine-tuning of a 7B model needs about 108 GB of training state; inference needs about 13.5 GB.
• A matrix of rank r can be written as BA with B of size d \times r and A of size r \times d: 2dr numbers instead of d^2 (131,072 against 16.8 million for d = 4{,}096, r = 16).
• Fine-tuning changes have low intrinsic dimension (Aghajanyan et al., 2020). LoRA (Hu et al., Microsoft, 2021): h = W_0 x + \frac{\alpha}{r} BAx, W_0 frozen, B initialised to zero, scale \alpha/r, adapters on all linear layers, mergeable after training. A 7B model with rank-16 adapters on all linear layers trains about 40 million parameters (0.59%).
• QLoRA (Dettmers et al., 2023): 4-bit NF4 base, double quantisation, paged optimisers; 65B fine-tuned on one 48 GB GPU.
• Serving: one base plus many adapters per GPU, mixed in batches (S-LoRA, vLLM).
• The family:
  • LoRA-FA freezes A, which saves activation memory.
  • VeRA shares frozen random A and B and trains only scaling vectors, about 28× fewer parameters here.
  • LoRA+ uses \eta_B = \lambda \eta_A with \lambda \approx 16.
  • Delta-LoRA adds the change in BA to the base. It is closer to full fine-tuning, but no longer an adapter.
  • DoRA trains the size of the change separately from its direction.
• Defaults: r 8–32, \alpha = 2r, all linear layers, learning rate about 10^{-4} to 2 \times 10^{-4}, one to three epochs, confirmed on the evaluation set.

Self-test: How many numbers does a rank-8 adapter for a 4,096-by-11,008 matrix have, and what fraction of the full matrix is that? Why does freezing A reduce activation memory, not only parameter count? Why can a VeRA adapter be stored as kilobytes?

Next: 12.11.3 A Fine-Tuning Project, End to End — Cartwise's ticket classifier from data to deployment: building the baseline, collecting and checking the examples, training with QLoRA, proving it beats the prompted model, and serving it.