Appearance
12.13.1 — Vision and Vision-Language Models
Two image problems arrived at Cartwise in the same quarter.
The refund agent of Chapter 12.10 asks customers with damaged items for a photo. Support leads were looking at 4,000 of them a day, and most were easy: a cracked screen, a crushed box. Could a model look first, describe the damage, check that the photo shows the product that was ordered, and pass the support lead a short summary?
Meanwhile, sellers upload about 30,000 product photos a day, and quite a few break the listing rules. Some are blurry. Some carry another website's logo. Some show the wrong product. And some contain text, such as phone numbers, that sellers are not allowed to put in images. A team of reviewers checked a sample by hand.
Both are problems of seeing. This page explains how models turn pixels into something they can reason about, from the convolutional networks of 12.5.5 to the transformers that now read images the way they read text, and then how to use vision-language models, the language models that accept images, in a real product, including where they cannot be trusted.
1. An image is a grid of numbers
A colour photo is a grid of pixels, and each pixel is three numbers, the brightness of red, green and blue, usually from 0 to 255. A photo 1,000 pixels wide and 800 high is therefore 1{,}000 \times 800 \times 3 = 2.4 million numbers. Everything a vision model does starts from that grid.
The difficulty is that the numbers say almost nothing directly. The same earphones photographed in a darker room, from a different angle, or a little further away produce a completely different grid. A model must learn what stays the same, the shapes and parts, while ignoring what does not, the lighting, position and scale.
Chapter 12.5.5 showed the first successful answer, the convolutional network: small filters slide across the image detecting edges, then combinations of edges such as corners and textures, then parts such as a wheel or an eye, building up to whole objects. Convolutional networks are built to assume two things about images: nearby pixels belong together, and a pattern means the same thing wherever it appears. Those built-in assumptions are what let them learn from relatively little data, and they dominated computer vision from 2012 to around 2020.
2. The Vision Transformer: images as sequences of patches
In 2020, Alexey Dosovitskiy and colleagues at Google published a paper whose title states its idea: An Image is Worth 16×16 Words. They asked whether the transformer of Chapter 12.7, which had transformed language, could read images if images were made to look like text. Their Vision Transformer, or ViT, does this in three steps.
First, the image is cut into patches. A 224-by-224-pixel image cut into 16-by-16 squares gives 14 \times 14 = 196 patches. Each patch holds 16 \times 16 \times 3 = 768 numbers, which are laid out in a row.
Second, each row of 768 numbers is multiplied by a learned matrix to produce a patch embedding, exactly as a token ID is turned into an embedding in a language model, and a position embedding is added, because otherwise the model would not know which patch came from which part of the image (12.7.4). An extra learned token is placed at the front of the sequence to gather a summary of the whole image.
Third, the sequence of 197 vectors passes through ordinary transformer blocks. Self-attention, from 12.7.3, lets every patch look at every other: the patch showing a crack in an earphone case can relate directly to the patch showing the brand logo on the other side, which a convolutional network would only connect after many layers. At the end, the summary token's vector represents the image, and a classifier or another model can use it.
The ViT gave up the convolutional network's built-in assumptions about images, and paid for it: trained on a million images it did worse than a good convolutional network, because it had to learn from data what the convolutional network assumed from the start. Trained on hundreds of millions of images, it did better, and it scaled further, as transformers do. Since then, transformers have become the standard image encoders in large systems, with convolutional networks still common where data is limited or models must be small and fast, such as on a phone.
3. Matching images to words: CLIP and zero-shot classification
Chapter 12.6.2 introduced CLIP, OpenAI's 2021 model trained on 400 million image-caption pairs from the internet so that an image and a description of it get nearby vectors. The same property gives a very useful ability for the listing problem: classifying images into categories nobody trained it on, called zero-shot classification.
To check whether a seller's photo shows what the listing says, embed the image, embed a short description of each possible category, and pick the description whose vector is closest:
python
import torch
from PIL import Image
from transformers import CLIPModel, CLIPProcessor
model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")
labels = ["a product photo of wireless earphones",
"a product photo of a phone case",
"a product photo of a kitchen knife set",
"a screenshot of a website"] # (1)
image = Image.open("listing_88231.jpg")
inputs = processor(text=labels, images=image, return_tensors="pt", padding=True) # (2)
with torch.no_grad():
logits = model(**inputs).logits_per_image # (3)
probs = logits.softmax(dim=-1)[0] # (4)
for label, p in sorted(zip(labels, probs.tolist()), key=lambda x: -x[1]):
print(f"{p:.2f} {label}")Line (1) writes the categories as short phrases. Phrasing matters: "a product photo of wireless earphones" works better than the bare word "earphones", because CLIP learned from captions, and captions look like phrases. The last label catches a common rule-break, a screenshot of another website instead of a real photo. Line (2) prepares the image (resized and normalised) and tokenises the four phrases. Line (3) computes the similarity between the image and each phrase, scaled by a temperature CLIP learned in training. Line (4) turns the similarities into probabilities with softmax (12.2.6). For a clear photo of earphones, the printout puts the earphones phrase first with most of the probability; when a seller lists earphones with a photo of a phone case, the phone-case phrase wins, and the listing is flagged.
Zero-shot classification needs no training data at all, which makes it ideal for starting. Once some examples exist, two cheap improvements usually help: averaging the image's similarity over several phrasings of each label, and training a small classifier on CLIP's image vectors using a few hundred labelled examples, a linear probe.
4. More than one label per image
Classification gives one answer per image. Many real tasks need more, and each has its own kind of model.
| Task | Output | Cartwise use |
|---|---|---|
| Classification | one label per image | is this a real product photo? |
| Detection | boxes around objects, with labels | find a watermark, find the product |
| Segmentation | which pixels belong to which object | cut the product out of its background |
| OCR (reading text) | the text in the image, with positions | phone numbers in listing photos |
| Embedding | a vector for search | "search by photo" |
Detection draws a box around each object and labels it. The YOLO family ("you only look once"), started by Joseph Redmon in 2015, made it fast enough to run on video in real time by predicting all boxes in one pass of the network. Segmentation goes further and marks exactly which pixels belong to each object; Meta's Segment Anything model of 2023 can outline almost any object in any image when pointed at it with a click or a box, which makes cutting products out of cluttered backgrounds easy. OCR, optical character recognition, reads text in images, the first step of the document pipelines of 12.15.4. And image embeddings power visual search: a customer photographs a lamp they like, the photo is embedded, and the vector index of 12.6.3 returns the most similar products in the catalogue.
5. Vision-language models: language models that can see
A vision-language model (VLM) accepts images as well as text and answers in text: "describe the damage in this photo", "what does the label on this box say?", "does this photo show the same earphones as this product listing?". It can do things none of the specialised models above can, because it combines seeing with the reasoning and instruction-following of a language model.
Most VLMs are built the same way, a design popularised by the open LLaVA model in 2023:
A vision encoder, usually a ViT trained CLIP-style, turns the image into one vector per patch. A small projector network converts those vectors into the language model's own embedding space, so they become, in effect, extra tokens. The language model reads the image tokens and the text tokens in one sequence, with the attention of 12.7.3 connecting words in the question to regions of the image. Training first teaches the projector to connect the two halves on image-caption pairs, then teaches the whole model to follow instructions about images with examples of questions and answers, the same supervised fine-tuning as 12.7.9. The largest commercial models are trained on images and text together from the start, but the idea, images becoming tokens the model attends over, is the same.
Images cost tokens
Because images become tokens, they are billed and counted like tokens. A rough guide for Claude models is one token per 750 pixels of image area; current models accept images up to 2,576 pixels on the long edge, and a full-resolution image can use up to about 4,800 tokens. On a large model at $4 per million input tokens, that is about two cents an image, and on a small model at $0.10 per million, about a twentieth of a cent. Images are a large share of the input of any vision feature, so sending them at the resolution the task actually needs, and no higher, is the first cost lever.
What VLMs are good and bad at
VLMs are very good at describing a scene, reading clear text including handwriting, interpreting charts and screenshots, comparing two images, and answering questions that need both seeing and knowing ("is this a European plug?").
They are unreliable at counting many similar objects, measuring sizes and distances precisely, small details in large images (a hairline crack in a phone screen photographed from a metre away), exact positions, and anything that needs certainty about what is not in an image. And they invent details, the visual form of the hallucinations of 12.7.9: asked to describe the damage on an undamaged item, a model may describe some anyway.
Two practices help with the small-detail problem. Send the image at a resolution where the detail is visible, and give the model a crop tool: a tool that returns an enlarged region of the image, so that the model can look closer at the part that matters, as a person would. Current models use such a tool well, and it improves accuracy on dense or detailed images more than any prompt does.
6. The damaged-item check, built
The refund agent's photo check uses a VLM with structured output, and it is designed around what VLMs are bad at:
python
import base64
from typing import Literal
from pydantic import BaseModel
class PhotoCheck(BaseModel):
shows_ordered_product: Literal["yes", "no", "unclear"]
damage_visible: Literal["yes", "no", "unclear"]
damage_description: str # what is visible, in one or two sentences
photo_quality: Literal["good", "blurry", "too_dark", "too_far"]
def check_damage_photo(photo_bytes: bytes, product_title: str, customer_claim: str) -> PhotoCheck:
image = {"type": "image", "source": {"type": "base64", "media_type": "image/jpeg",
"data": base64.standard_b64encode(photo_bytes).decode()}} # (1)
r = client.messages.parse(
model="claude-sonnet-5-5", max_tokens=600,
messages=[{"role": "user", "content": [image, {"type": "text", "text":
f"A customer says this photo shows damage to '{product_title}'. "
f"Their description: \"{customer_claim}\". Describe only what you can see. "
"If you cannot tell whether something is damaged, say unclear."}]}], # (2)
output_format=PhotoCheck) # (3)
return r.parsed_outputLine (1) sends the photo as an image block, before the text, which is the recommended order. Line (2) gives the model the product and the customer's claim so it knows what to look for, and explicitly asks it to describe only what it sees and to say "unclear" when it cannot tell, the visual version of the permitted "I don't know" from 12.8.2. Line (3) returns the four fields as a typed object (12.8.3).
The most important design decision is not in the code: the check informs a person; it does not decide the refund. Its output goes to the support lead's approval screen from 12.10.8, next to the photo, so the lead can confirm at a glance. A photo_quality of blurry or too_far triggers an automatic request for a better photo. Only for low-value orders with a clear "yes, damage visible, shows the product" does the agent proceed without a person, within the limits of its policy code.
Two kinds of abuse needed separate defences, because a VLM is not the right tool for either. Reused photos, the same damaged-item picture submitted for several orders or copied from the internet, are caught with a perceptual hash, a short fingerprint of an image that stays the same when the image is resized or recompressed, compared against every earlier submission. Generated or edited photos are a growing problem, and detecting them reliably is not yet possible; the defence is the overall design, in which a photo is one piece of evidence among the order history, the customer's record and the limits in code.
7. Choosing the right tool for the volume
For the 30,000 daily listing photos, the team compared approaches on cost and accuracy:
| Approach | Cost per day | Strength |
|---|---|---|
| CLIP zero-shot + linear probe, self-hosted | a few dollars of GPU time | fast, cheap, good for clear-cut rules |
| OCR model for text in images | similar | exact text, positions |
| Small VLM on every image | about $5 (30,000 × ~1,500 tokens at $0.10/M) | flexible, reads rules in words |
| Large VLM on every image | about $180 at $4/M | best judgement on hard cases |
Each estimate is the token arithmetic of 12.10.10 applied to images. The design Cartwise chose is the cascade of 12.10.6 again: CLIP and OCR screen every photo for the clear-cut problems (wrong category, screenshots, phone numbers), a small VLM checks the photos they are unsure about against the written listing rules, and the hardest few hundred go to the large model or a reviewer. The reviewers, instead of sampling at random, now see the cases the models disagree on, which is where their judgement is worth the most.
What the interviewer will push on
"How does a Vision Transformer work?" Cut the image into patches (16×16 gives 196 for a 224×224 image), turn each into an embedding with a learned matrix plus a position embedding, add a summary token, and run ordinary transformer blocks. It lacks a convolutional network's built-in assumptions, so it needs more data but scales better.
"How does zero-shot image classification work?" CLIP embeds images and text in one space; embed the image and a phrase per category, pick the closest. Mention phrasing labels as captions and improving with a linear probe once you have examples.
"How are vision-language models built?" A vision encoder produces patch vectors, a projector maps them into the language model's embedding space as image tokens, and the language model reads them with the text; trained first on image-caption alignment, then on visual instructions. Images cost tokens, roughly by pixel area.
"Where would you not trust a VLM?" Counting many objects, precise measurement, tiny details at low resolution, exact positions, proving absence, and invented details. Defences: right resolution, a crop tool, "unclear" as an allowed answer, and a person or code making consequential decisions.
"How would you moderate 30,000 product photos a day?" A cascade: cheap specialised models (CLIP classification, OCR, perceptual hashing) on everything, a small VLM on uncertain cases with the rules in words, a large model or people on the hardest, with costs worked out per image.
Something to volunteer: detecting AI-generated or edited photos is not reliable today, so any process that accepts photos as evidence should treat them as one signal among several, with limits enforced in code, rather than as proof.
Recall
- An image is a grid of pixels, three numbers each. Convolutional networks build features from local filters with built-in assumptions; they learn from less data.
- The Vision Transformer (Dosovitskiy et al., Google, 2020): 16×16 patches (196 for 224×224), each embedded with a learned matrix plus position, plus a summary token, read by ordinary transformer blocks. Needs more data, scales better.
- CLIP puts images and text in one space; zero-shot classification picks the closest caption-like label; a linear probe improves it with a few hundred examples.
- Other tasks: detection (boxes; YOLO), segmentation (pixel masks; Segment Anything), OCR (text), embeddings (visual search).
- A vision-language model = vision encoder → projector → image tokens read by a language model. Images cost tokens, roughly one per 750 pixels, up to about 4,800 at full resolution on current Claude models.
- VLMs describe, read, interpret charts and compare well; they count, measure, see tiny details and prove absence poorly, and can invent details. Use the right resolution, a crop tool, an "unclear" option, and keep consequential decisions with people or code.
- Use cascades: cheap specialised models on everything, VLMs on uncertain cases, people on the hardest; perceptual hashes catch reused photos.
Self-test: How many patches does a 384×384 image give with 16×16 patches, and how many numbers is each patch before embedding? Why does the damage check allow the answer "unclear"? Roughly how many tokens is a 1,500×1,000 photo, and what does it cost on a model charging $2 per million input tokens?
Next: 12.13.2 Speech Recognition and Synthesis — from seeing to hearing: how sound becomes numbers, why accents and dialects make speech recognition hard, and how synthetic voices learned to sound human, including the markup that controls them.