12.11.1 — Prompt, RAG or Fine-Tune?

Three requests reached Cartwise's AI team in the same month, and all three were phrased the same way: "Can we train the model on our data?"

• The support team wanted "Ask Cartwise" to know the new returns policy, which had changed again.
• The brand team wanted product descriptions to sound like Cartwise: warm, plain, never boastful.
• The operations team wanted every one of the two million support tickets a month sorted into one of 40 categories, with the order number, product and urgency pulled out, cheaply and fast enough to route each ticket within a second of it arriving.

"Train the model on our data" means changing the model's weights, which is called fine-tuning. It is the right answer to exactly one of those three requests. Knowing which one, and why, is the most important skill in this chapter, because fine-tuning is the most expensive of the ways to change what a model does, and the most commonly misused.

1. Three levers, in the order to pull them

When a model does not do what you want, there are three levers, and each changes something different.

The prompt changes what the model is told: instructions, examples, the format wanted, the context of 12.8.2. It takes minutes to try, costs nothing to change, and can be undone instantly.

Retrieval changes what the model can look at: the RAG of Chapter 12.9 puts the relevant documents in front of it for each question. It needs a pipeline, but updating what the model knows is as simple as updating a document.

Fine-tuning changes what the model is: it continues training on your examples, so the model's weights, and therefore its habits, shift. It needs training data, a training run, evaluation and a way to serve the new model, and every change means doing it all again.

The order matters because the cost and the time to change your mind rise steeply from the first lever to the third. The working rule is: try the prompt first, add retrieval when the problem is missing knowledge, and fine-tune only when the problem is behaviour that the first two cannot fix, or cost and speed at a scale where a smaller specialised model pays.

2. Fine-tuning teaches behaviour, not facts

The single most important fact about fine-tuning is this: fine-tuning is good at teaching a model how to respond, and poor at teaching it what is true.

That sounds surprising, since pretraining is how models learned their facts in the first place (12.7.5). But pretraining saw each fact many times, in many phrasings, across trillions of tokens. A fine-tuning set shows the model a few thousand examples a few times each. The model picks up what is consistent across those examples, such as the format, the tone, the kind of answer, very quickly. Specific facts that appear once or twice are absorbed weakly and unreliably, and research has found two specific problems with trying:

• A 2023 Microsoft study by Oded Ovadia and colleagues compared continued training on a set of documents with retrieving from the same documents, as ways of teaching models new facts, and found retrieval consistently better, both for facts the model had partly seen before and for entirely new ones.
• A 2024 Google study by Zorik Gekhman and colleagues found that models learn genuinely new facts from fine-tuning much more slowly than they learn the examples' style, and that as they do, they become more likely to invent answers in general. The model learns, in effect, "answer confidently even when you did not know this before".

Beyond accuracy, fine-tuned facts have practical problems that retrieval does not. They cannot be cited, because the model cannot point to where it learned something. They cannot be updated without another training run. And they cannot be restricted to the people allowed to see them, because whatever is in the weights is available to everyone who uses the model.

So the support team's request, "know the new returns policy", is a retrieval problem. The policy changes, answers must cite it, and a fine-tuned model would keep the old policy in its weights until the next training run. Section 4 deals with the other two requests.

3. What fine-tuning is good for

Fine-tuning earns its cost in five situations.

A consistent style or voice that prompting does not hold. Instructions and a few examples get most of the way, but over thousands of outputs a prompted model drifts back towards its default voice. A model fine-tuned on a few thousand approved examples holds the style without being reminded.

A narrow, high-volume task done by a much smaller model. This is the most common reason in industry. A large model prompted to classify tickets works well but is slow and expensive at two million tickets a month. A small model fine-tuned on that one task can match the large model's accuracy on it, at a fraction of the cost and latency. The small model becomes worse at everything else, which does not matter, because everything else is not its job.

A precise output format or an unusual language. Following a complicated schema exactly, writing a company's own query language, or producing code in an internal framework, where the pattern is hard to describe but easy to show.

Teaching a skill that examples show better than words. Judging which customer emails need a manager, in the way experienced staff do, is hard to write down as rules and easy to demonstrate with thousands of past decisions.

Shortening prompts. A model fine-tuned to follow a long set of instructions no longer needs them in every request, saving input tokens on every call. At high volume, this alone can pay for the fine-tuning.

4. Kinds of fine-tuning

"Fine-tuning" covers several different techniques, all of which continue training a model but on different kinds of data.

Supervised fine-tuning (SFT) trains on pairs of input and desired output: this ticket, this classification. The model learns to produce outputs like the examples. It is the same stage that turned base models into assistants in 12.7.9, applied to your task, and it is what most fine-tuning projects use.

Preference fine-tuning trains on pairs of a better and a worse response to the same input, using a method such as DPO from 12.7.9. It suits qualities that are easier to compare than to write: "this reply is warmer than that one".

Distillation, from 12.7.8, is SFT where a large model's outputs are the training examples for a small one. It is how the "small model for a narrow task" pattern usually gets its data: run the large prompted model on many real inputs, check and correct its outputs, and train the small model on them.

Reinforcement fine-tuning gives the model tasks and a grader, code or a model that scores each attempt, and reinforces the attempts that score well, the reasoning-model training of 12.7.9 applied to your task. It suits tasks with checkable answers where you have few worked examples but can score attempts.

Each of these can update all of the model's weights, called full fine-tuning, or only a small added set, called parameter-efficient fine-tuning. The second is how almost all fine-tuning is done today, because it needs a fraction of the memory and produces a small file instead of a whole new model. 12.11.2 explains how it works.

And there are two ways to do it. Provider fine-tuning means uploading your examples to a model provider's fine-tuning service, which trains and hosts a custom version of one of its models. Open-weights fine-tuning means downloading a model whose weights are published, training it yourself, and serving it yourself or through a hosting service, the subject of 12.11.4.

5. What fine-tuning really costs

The training run is usually the cheapest part. The full cost includes:

• Data. Thousands of good examples, checked by people who know what good looks like. This is usually the largest cost and the one that decides the result, as Chapter 12.4 explained for every kind of model.
• Evaluation. An evaluation set and a comparison with the prompted baseline, before and after every training run (12.8.6).
• Serving. A fine-tuned model must be hosted somewhere, which for open-weights models means GPUs and the engineering to run them.
• Keeping it current. When the task changes, say a 41st ticket category, the model must be retrained. And when a better base model is released, a prompted system upgrades by changing one string, while a fine-tuned one must be trained again on the new base, or it falls behind.
• Lost generality. A model trained hard on one task loses some of its other abilities, an effect called catastrophic forgetting. Usually this does not matter for a single-purpose model, but it means a fine-tuned model should not be used for anything it was not trained and evaluated on.

That last pair of costs explains a pattern seen in many companies: a fine-tuned model that beat the prompted large model when it was built is overtaken a year later by a newer, cheaper prompted model, and the team that built it is left maintaining it.

6. Making the call

Here are the three requests again, worked through the decision.

Request	Problem type	Lever
Know the new returns policy	missing, changing facts	retrieval
Sound like Cartwise	style	prompt first; fine-tune if it drifts
Classify 2M tickets a month fast	narrow, high-volume task	fine-tune a small model

The returns policy is knowledge that changes, must be cited and must be current. Retrieval, already built in Chapter 12.9.

The brand voice starts with the prompt: a short style guide and five approved descriptions as examples. Cartwise measured it with the evaluator loop of 12.10.6 and a model judge, and found 94% of descriptions passed the brand rules on the first attempt, with the evaluator catching most of the rest. That was good enough, so no fine-tuning. Had it been 70%, with the same problems recurring, a fine-tune on approved descriptions would have been the next step.

The ticket classification is the one that fine-tuning wins. The prompted large model classified accurately, but at two million tickets a month the cost was significant and each call took over a second. A small open-weights model, fine-tuned on examples labelled by the large model and checked by support staff, could do this one job at a fraction of the cost and within the one-second target. 12.11.3 builds it from start to finish.

When the answer is not obvious, three questions settle it:

1. Is the problem what the model knows, or how it behaves? Knowledge points to retrieval; behaviour points to the prompt and then fine-tuning.
2. Have you pushed the prompt as far as it goes, and measured it? Fine-tuning to fix a problem a better prompt would have fixed is the most common waste in this area. Measure the prompted version on a proper evaluation set first; that measurement is also the baseline the fine-tuned model must beat.
3. Will the volume and the stability of the task pay for the upkeep? A task done ten thousand times a month rarely justifies a fine-tune; one done two million times, unchanged for a year, often does.

The levers also combine. A fine-tuned model can be given a prompt and retrieved documents like any other: a model fine-tuned to answer in Cartwise's support format, with retrieval supplying the current policy, uses each lever for what it is good at.

What the interviewer will push on

"When would you fine-tune versus use RAG?" This is one of the most common LLM interview questions. The answer they want: RAG for knowledge, especially knowledge that changes, must be cited or must be access-controlled; fine-tuning for behaviour (style, format, a narrow skill) and for making a small model do one high-volume task cheaply. The tell of real understanding is saying why fine-tuning is poor for facts: few examples teach style fast and facts slowly, fine-tuned facts cannot be cited, updated or restricted, and research found fine-tuning on new facts increases hallucination.

"A team wants to fine-tune on its internal wiki so the chatbot knows it. What do you say?" Build retrieval over the wiki instead, for the reasons above. If they also want a particular answer style, consider fine-tuning for the style, with retrieval supplying the content.

"What does fine-tuning cost beyond the training run?" Data preparation and checking (usually the biggest), evaluation, serving, retraining when the task changes, re-training on each new base model, and lost general ability.

"What kinds of fine-tuning are there?" SFT on input-output pairs, preference fine-tuning such as DPO, distillation from a larger model, and reinforcement fine-tuning with a grader; each either full or parameter-efficient; each through a provider or on open weights.

Something to volunteer: fine-tuned models age. A prompted system upgrades to a better base model by changing a string; a fine-tuned one must be rebuilt, so a fine-tune that wins today should be re-evaluated against the newest prompted models every few months.

Recall

• Three levers, cheapest first: the prompt changes what the model is told; retrieval changes what it can look at; fine-tuning changes the model's weights.
• Fine-tuning teaches behaviour, not facts: few examples teach style and format quickly and facts slowly; fine-tuned facts cannot be cited, updated or access-controlled; fine-tuning on new facts increases invented answers (Gekhman et al., 2024). Knowledge belongs in retrieval.
• Fine-tune for: a style prompting cannot hold, a narrow high-volume task done by a smaller model, a precise format or unusual language, a skill easier to show than describe, or shorter prompts at scale.
• Kinds: supervised (SFT), preference (DPO), distillation, reinforcement with a grader; full or parameter-efficient; through a provider or on open weights.
• Real costs: data and checking, evaluation, serving, retraining as the task changes, rebuilding on new base models, lost generality.
• Decide by asking: knowledge or behaviour? prompt pushed and measured? volume and stability enough to pay for upkeep? Combine levers freely.

Self-test: Why would a model fine-tuned on Cartwise's returns policy still give the old window after the policy changes? What measurement must exist before any fine-tuning project starts, and why? Give one reason a fine-tuned model that beats a prompted large model today might lose to one next year.

Next: 12.11.2 LoRA and Parameter-Efficient Fine-Tuning — fine-tuning a 7-billion-parameter model in full needs over 100 GB of memory; the next page explains the idea that cut this to a single ordinary GPU, the mathematics behind it, and the family of variants it started.