12.11.3 — A Fine-Tuning Project, End to End

Every support ticket that reaches Cartwise, two million a month by email, chat and the app, has to go somewhere: the refunds queue, the courier escalations team, the payments team, the fraud desk, and 36 other destinations. A ticket sent to the wrong team waits, is read by someone who cannot help, and is passed on. Cartwise measured the cost of each misrouted ticket at about $0.30 of staff time, plus a slower answer for the customer. Routing is done by a classifier that reads each ticket and returns its category and three extracted fields.

12.11.1 identified this as the request where fine-tuning might win. This page runs the whole project, in the order a real one runs: define the target, measure the baseline, build the data, train, evaluate, deploy, and keep it working. The numbers are Cartwise's own measurements on its own tickets; your task will give different ones, and the method of getting them is the point.

1. The task and the bar to clear

The classifier receives one ticket's text and returns:

{"category": "delivery_not_received", "order_id": "4471", "product": "wireless earphones", "urgency": "high"}

category is one of 40 fixed values; order_id and product are copied from the ticket when present and null otherwise; urgency is low, normal or high. Tickets arrive in English, Hindi, and the mixture of the two that many customers write in, often called Hinglish.

Before anything is trained, the team wrote down what success means, as numbers:

• Accuracy on category at least as good as the best prompted model, measured also as macro-F1 (the average F1 across all 40 categories, from 12.3.2), so that rare categories such as suspected_fraud count as much as common ones.
• Every output valid JSON with allowed values, every time.
• p95 latency under 500 milliseconds, because routing happens while the customer waits for a "we've received your message" reply that names the right team.
• Total monthly cost, model plus misrouting, lower than the best alternative.

That last line is the economics of 12.10.10 again: what matters is the cost of the outcome, which here includes every misrouted ticket.

2. The baseline: what prompting already achieves

The team first built the evaluation set, because nothing else can be judged without it. Two senior support staff independently labelled 1,000 tickets from October, sampled so that every category had at least 15 examples, and resolved their disagreements together, the labelling practice of 12.4.1. This gold set is never used for training.

Then they ran the best prompted models on it, with a prompt containing the 40 category definitions and structured output from 12.8.3. The category definitions, about 700 tokens, are identical for every ticket, so they were cached (12.10.10).

Model, prompted	Accuracy	Macro-F1	p95 latency	Model cost / month
Claude Opus 5.5	93.1%	0.89	1.3 s	$3,480
Claude Haiku 5.5	86.4%	0.79	0.6 s	$220

The cost column comes from token counts. Each ticket sends about 700 cached tokens of definitions, 200 tokens of ticket, and receives about 40 tokens of JSON. For Opus that is 700 \times \$0.20 + 200 \times \$4 + 40 \times \$20 per million tokens, about $0.0017 a ticket, or $3,480 for two million. For Haiku, $220.

Now add misrouting. Opus misroutes 6.9% of two million tickets, about 138,000, costing about $41,400 a month. Haiku misroutes 13.6%, about 272,000, costing about $81,600. Haiku's model bill is sixteen times smaller and its total cost is nearly twice as large. Opus is the real baseline, and it fails the latency target. The question for fine-tuning is now precise: can a small model match Opus's accuracy, inside 500 milliseconds, for less money?

3. The data

Where the examples came from

The team had two sources, each with a weakness.

Historical corrections. Eighteen months of tickets that support staff had moved to a different queue: 140,000 tickets whose final queue is a human judgement. But they are noisy, since staff sometimes moved tickets for reasons other than category, such as workload, and they are skewed towards the cases the old keyword router got wrong.

Labels from the large model. Opus classified 20,000 recent tickets with the baseline prompt. Its labels are consistent and well explained, but they carry Opus's own 7% of errors.

Combining them, with people checking where they disagree, gives better data than either alone. This is distillation from 12.7.8, with human review in the loop:

1. Where Opus and the human correction agree, keep the example.
2. Where they disagree, about 9% of cases, a support lead decides. These are the most valuable examples in the set, because they are exactly the hard cases.
3. For rare categories with fewer than 150 examples, search the archive for likely candidates (by keyword, and by embedding similarity to known examples from 12.6.2), have Opus label them, and have staff check every one.
4. Cap common categories at 600 examples each, so that where_is_my_order, a quarter of all tickets, does not drown out the rest. This is the imbalance handling of 12.4.3.

The result was 12,000 training examples and 1,500 validation examples, with every category represented.

Cleaning

Personal details were replaced with placeholders: phone numbers became <PHONE>, email addresses <EMAIL>, card fragments <CARD>. Order numbers were kept, because extracting them is part of the task; a model trained on placeholders would learn to output placeholders. Near-duplicates, such as the same complaint pasted into three channels, were removed so that the validation set could not contain copies of training examples. The split is by time: training tickets from January to August, validation from September, and the gold set from October. Splitting randomly would let the model see next month's promotions and problems during training and score unrealistically well, the leakage of 12.4.2.

Formatting

Each example becomes a short conversation in the chat format the model was trained on (12.7.9):

import json

SYSTEM = "Classify this Cartwise support ticket. Reply with JSON only."           # (1)

def to_example(ticket_text: str, label: dict) -> dict:
    return {"messages": [
        {"role": "system", "content": SYSTEM},
        {"role": "user", "content": ticket_text},
        {"role": "assistant", "content": json.dumps(label, ensure_ascii=False)},   # (2)
    ]}

with open("train.jsonl", "w", encoding="utf-8") as f:
    for ticket, label in training_pairs:
        f.write(json.dumps(to_example(ticket, label), ensure_ascii=False) + "\n")  # (3)

Line (1) is the whole system prompt: one sentence. The 40 category definitions that the prompted models needed are gone, because the fine-tuned model learns the categories from twelve thousand examples instead of reading definitions every time. This is the prompt shortening of 12.11.1, and it cuts each request from about 900 tokens to about 250. Line (2) is the answer the model should learn to produce, exactly in the output format. ensure_ascii=False keeps Hindi text readable rather than turning it into escape codes. Line (3) writes one example per line, the JSONL format training tools expect.

4. Training

The base model is an open-weights, instruction-tuned model of about 7 billion parameters with good Hindi support and a licence that allows commercial use (here Qwen2.5-7B-Instruct, published under the Apache 2.0 licence). Training uses QLoRA from 12.11.2, with Hugging Face's TRL library running the training loop:

import torch
from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig
from trl import SFTConfig, SFTTrainer

BASE = "Qwen/Qwen2.5-7B-Instruct"
model = AutoModelForCausalLM.from_pretrained(BASE, quantization_config=BitsAndBytesConfig(
    load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16))                                          # (1)
tokenizer = AutoTokenizer.from_pretrained(BASE)

data = load_dataset("json", data_files={"train": "train.jsonl", "validation": "val.jsonl"})

trainer = SFTTrainer(
    model=model, processing_class=tokenizer,
    train_dataset=data["train"], eval_dataset=data["validation"],
    peft_config=LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05,
                           target_modules="all-linear", task_type="CAUSAL_LM"),      # (2)
    args=SFTConfig(output_dir="ticket-classifier", num_train_epochs=2,               # (3)
                   per_device_train_batch_size=8, gradient_accumulation_steps=4,     # (4)
                   learning_rate=2e-4, lr_scheduler_type="cosine", warmup_ratio=0.03, # (5)
                   eval_strategy="steps", eval_steps=100, logging_steps=20,           # (6)
                   bf16=True))
trainer.train()
trainer.save_model("ticket-classifier/adapter")                                       # (7)

Line (1) loads the frozen base in 4-bit NF4, about 4 to 5 GB for a 7B model of this shape. Line (2) adds rank-16 adapters to every linear layer, the defaults recommended in 12.11.2. Line (3) trains for two passes over the data, because a 12,000-example set begins to overfit soon after. Line (4) uses batches of 8 examples and accumulates gradients over 4 batches before each update, giving an effective batch of 32 without the memory of 32 at once, the gradient accumulation of 12.5.7.

Line (5) sets the learning rate that suits LoRA, with a short warm-up and a cosine decay, the schedule of 12.5.4. Line (6) measures the loss on the validation set every 100 steps, which is how overfitting is spotted. Line (7) saves only the adapter, about 80 MB.

Two settings deserve a word because their defaults vary between library versions. The loss should be computed only on the assistant's answer, not on the system prompt and ticket, otherwise the model is partly graded on reproducing ticket text it was given. Training libraries provide an option for this, and it is worth checking that it is on. And the chat template must be the base model's own, which the tokenizer supplies, so that the format at training time matches the format at serving time exactly.

The run processed about 3.6 million tokens per epoch and took under two hours on a single 48 GB GPU rented by the hour, a few dollars of compute. The validation loss fell steeply for the first epoch, flattened during the second, and was still falling slightly at the end, so two epochs was the right length: a third would most likely have started overfitting.

5. Evaluation

The trained model was run on the 1,000-ticket gold set, with exactly the same scoring as the baseline.

Model	Accuracy	Macro-F1	JSON valid	p95 latency
Opus 5.5, prompted	93.1%	0.89	100%	1.3 s
Haiku 5.5, prompted	86.4%	0.79	100%	0.6 s
Fine-tuned 7B, round 1	92.2%	0.86	99.6%	0.42 s

Round 1 was close but not good enough, and the numbers say where to look. Accuracy was slightly below Opus. Macro-F1 was further below, which means rare categories were the weak point. And four answers in a thousand were not valid JSON.

Error analysis, sorting the 78 mistakes by cause as in 12.8.6, found three groups:

• 31 mistakes confused delivery_delayed with delivery_not_received. Reading them showed the labels themselves were inconsistent: staff had used the two interchangeably for parcels several days late. The team wrote a clear rule (not received means the tracking says delivered, or the expected date passed by more than five days) and relabelled 600 training examples to follow it.
• 22 mistakes were in five rare categories with under 200 examples each. More examples were gathered for those, by the search-and-check method of section 3.
• The 4 invalid outputs were fixed at serving time rather than by training, as section 6 shows.

Round 2, trained on the corrected data, scored 93.6% accuracy and 0.90 macro-F1, slightly above Opus on both, at the same latency. On the Hinglish subset of the gold set, where the prompted Haiku had struggled most, it matched Opus. That is the model that shipped.

The round-1-to-round-2 story is typical. The biggest improvements in a fine-tuning project usually come from fixing the data, here inconsistent labels and thin categories, rather than from changing training settings.

6. Deployment

Packaging the model

The adapter was merged into the base weights, using a 16-bit copy of the base, because merging into the 4-bit training copy would add rounding error. The merged model was then quantised to 4 bits for serving (the AWQ method of 12.7.8), which halves the bytes read for every generated token compared with 8 bits and so roughly halves the time per token, the arithmetic 12.12.3 works through. The gold set was run again on the quantised version to confirm nothing was lost: accuracy moved by less than a tenth of a point.

import torch
from transformers import AutoModelForCausalLM
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=torch.bfloat16)   # 16-bit, not 4-bit
merged = PeftModel.from_pretrained(base, "ticket-classifier/adapter").merge_and_unload()
merged.save_pretrained("ticket-classifier/merged")

Merging made sense here because this model has one job and a GPU of its own. Had Cartwise wanted several fine-tunes on the same GPU, it would have kept the adapters separate, the multi-adapter design of 12.11.2.

Serving

The model runs on vLLM (12.7.6), which exposes an HTTP endpoint and supports structured outputs: given a JSON Schema, it constrains decoding so that only tokens that keep the output valid can be chosen, the constrained decoding of 12.8.3. That fixed round 1's four invalid answers permanently, with no retraining. The classifier service calls it like this:

import math, requests

def classify(ticket_text: str) -> dict:
    r = requests.post("http://classifier.internal:8000/v1/chat/completions", timeout=2, json={
        "model": "ticket-classifier",
        "messages": [{"role": "system", "content": SYSTEM},
                     {"role": "user", "content": ticket_text}],
        "response_format": {"type": "json_schema",
                            "json_schema": {"name": "ticket", "schema": TICKET_SCHEMA}},  # (1)
        "logprobs": True, "max_tokens": 80}).json()
    choice = r["choices"][0]
    label = json.loads(choice["message"]["content"])
    confidence = category_probability(choice["logprobs"]["content"])                       # (2)
    if confidence < 0.80:
        label = classify_with_opus(ticket_text)                                            # (3)
    return label

Line (1) passes the output schema, so every reply is valid JSON with one of the 40 allowed categories. Line (2) reads the model's own confidence: with logprobs turned on, the response includes the probability of each generated token, and the helper multiplies the probabilities of the tokens that spell out the category (adding their logarithms and taking the exponent, the softmax and log arithmetic of 12.2.6). Line (3) sends the uncertain tickets, about 4% of them, to the prompted Opus model instead.

That fallback is a cascade, from 12.10.6, and it does two jobs. It sends the hardest 4% to the strongest model, which raised overall accuracy to 94.1%. And it is also the plan for when the classifier is unavailable: if the GPU servers fail, every ticket goes to Opus, slower and dearer but still correctly routed.

The service runs on two GPUs in Cartwise's own cloud region, so that one can fail or be updated without stopping routing. It was released gradually: 5% of tickets for a week with its decisions compared against Opus's on the same tickets, then 50%, then all.

7. Was it worth it?

Monthly	Opus, prompted	Fine-tuned + 4% to Opus
Model cost	$3,480	$1,460 GPUs + $140 Opus
Misrouted tickets	138,000	118,000
Misrouting cost	$41,400	$35,400
Total	$44,880	$37,000
p95 latency	1.3 s	0.42 s (0.44 s with fallback)

The fine-tuned system saves about $7,900 a month and meets the latency target that Opus missed. Against that sit the costs of 12.11.1 that do not appear in this table: about five weeks of an engineer's time to build it, a few days a quarter to retrain it, and the operation of two GPU servers.

The honest conclusion is that the saving was real but modest, and latency was the deciding reason: Opus could not route tickets inside half a second, and the fine-tuned model could. A team that did not need the latency would have been right to stay with prompted Opus and spend the five weeks elsewhere. And the comparison must be repeated as new models arrive. If a future small prompted model reaches 93% accuracy at low latency, the fine-tune loses its reason to exist.

8. Keeping it working

A deployed fine-tune is a model in production, with the life cycle of 12.1.4.

• Watch for drift. The distribution of predicted categories is compared week by week with the one before. A sudden rise in payment_failed might be a real payments outage, or a new kind of ticket the model is pushing into the nearest category it knows. Either way, someone looks. 12.12.5 covers this in depth.
• Collect corrections. Every ticket staff move to a different queue is a new label, checked and added to the next training set.
• Retrain on a schedule, quarterly, and whenever the categories change. When Cartwise added a 41st category for its new grocery service, the model could not produce it at all until retrained, because a fine-tuned classifier knows only the categories it was trained on. The prompted fallback handled the new category from the day it was added to its prompt, which bought time for the retraining.
• Re-run the baseline comparison whenever a new model generation is released, as section 7 concluded.

What the interviewer will push on

"Walk me through a fine-tuning project." They want the order and the reasons:

1. Define success as numbers.
2. Build a gold evaluation set first.
3. Measure how well plain prompting does. That is the baseline to beat.
4. Build the training data: pick sources, have people check the cases where labels disagree, balance the classes, clean it, and split it by time.
5. Train with LoRA or QLoRA.
6. Evaluate on the same gold set.
7. Study the errors and fix the data.
8. Deploy with a fixed output format, a fallback when the model is unsure, and a gradual release.
9. Monitor, and retrain when it drifts.

The tell is starting with the evaluation set and the baseline.

"How do you get training data?" Combine human labels and a large model's labels, and send their disagreements to people; gather rare categories deliberately; cap common ones. Mention that the disagreement cases are the most valuable examples.

"Your fine-tuned model is 1% below the prompted large model. What next?" Sort the errors by cause before touching any settings. Most gains come from fixing inconsistent labels and thin categories, as round 2 showed.

"How do you make a fine-tuned model's output reliable in production?" Constrained decoding against a schema, confidence from token probabilities with a fallback to a stronger model for uncertain cases, two or more servers, and a gradual release compared against the old system.

"Was it worth it?" Total cost including errors, not just model cost, plus engineering and upkeep; be willing to conclude that the saving alone would not justify it, and name the real deciding factor.

Something to volunteer: a fine-tuned classifier cannot produce a category it never saw, so adding a category means retraining. A prompted fallback that can learn a new category from its prompt the same day is what makes the fine-tune safe to depend on.

Recall

• Start with success as numbers (accuracy, macro-F1, valid output, latency, total cost including errors) and a gold evaluation set labelled by experts, never trained on.
• Measure the prompted baseline first. Count error costs: Haiku's bill was 16× smaller than Opus's, but its total cost was nearly twice as large.
• Data: combine human corrections with large-model labels, have people resolve disagreements (the most valuable examples), gather rare categories deliberately, cap common ones, remove personal details but keep the fields to extract, deduplicate, split by time. A one-line system prompt replaces the long definitions.
• Train with QLoRA on all linear layers for one or two epochs, with loss only on the answer and the base model's own chat template; watch validation loss.
• Evaluate on the same gold set; sort errors by cause; fix the data (round 2 beat the baseline after relabelling and more rare examples).
• Deploy: merge into a 16-bit base, quantise and re-check, serve with constrained JSON output, send low-confidence cases to the strong model, run two servers, release gradually. Then monitor drift, collect corrections, retrain on a schedule, and keep comparing against new prompted models.

Self-test: Why was the training set split by month rather than at random? Which single serving change fixed the invalid JSON without retraining? Using the section 7 table, how large would the monthly saving be if latency did not matter, and would you still build it?

Next: 12.11.4 Self-Hosting versus APIs — the classifier runs on Cartwise's own GPUs; the next page works out, in general, when running models yourself beats paying a provider per token, and when it only looks as if it does.