Appearance
12.13.3 — Image Generation and Diffusion
Cartwise's sellers upload product photos taken on kitchen tables, against bedsheets, under yellow bulbs. The marketing team wanted those products shown in clean, attractive settings, a lamp on a bedside table, a backpack on a mountain trail, without a photo shoot for each of 400,000 products. And the design team wanted a quick way to draft banner images for sales campaigns.
Both are jobs for image generation: models that create new images from a text description, or change parts of existing ones. Since 2022, these models have gone from research curiosities to tools used by millions, and almost all of them are built on one idea, diffusion. This page explains that idea from the beginning, with the mathematics worked by hand, then how text steers it, how it is made fast enough to use, and what a retailer may and may not responsibly do with it. The last question matters as much as the technology: a generated image of a product that changes the product is a false advertisement.
1. The idea: learn to remove noise
Generating an image from nothing seems impossibly hard. Diffusion makes it tractable by turning it into a task that is merely hard: removing a little noise from an image.
Take any photo and add a small amount of random noise: tiny random changes to every pixel, like the grain of an old photograph. The photo is still recognisable. Add a little more, and a little more, hundreds of times, and eventually nothing is left but random static, with no trace of the original. That is easy to do and needs no learning at all. Call it the forward process.
Now train a neural network to do the reverse of one small step: given a slightly noisy image, predict the noise that was added, so it can be subtracted. If a network can do that well at every noise level, from barely noisy to almost pure static, then it can generate images. Start from pure random static, remove a little noise, then a little more, hundreds of times, and an image appears out of the static that was never in any training set. That is the reverse process.
The idea was first proposed in 2015 by Jascha Sohl-Dickstein and colleagues, who borrowed it from the physics of diffusion, the way a drop of ink spreads through water until it is evenly mixed. The reverse, un-mixing the ink, is what the model learns. It became practical in 2020, when Jonathan Ho, Ajay Jain and Pieter Abbeel at Berkeley published denoising diffusion probabilistic models (DDPM) and showed that a simple version of the training produced images of a quality that rivalled the best methods of the time. Before diffusion, the leading approach was the generative adversarial network (GAN, Ian Goodfellow and colleagues, 2014), in which one network generates images and another tries to tell them from real ones. GANs could produce sharp images but were notoriously unstable to train and tended to produce a narrow range of outputs; diffusion is steadier and more varied, which is largely why it took over.
2. The mathematics, worked by hand
The forward process in one formula
Adding noise step by step has a convenient property: you can jump straight to any step t in one calculation instead of adding noise t times. Write x_0 for the original image (all its pixel values), x_t for the image after t steps, and \epsilon ("epsilon") for a sample of random noise drawn from the standard bell curve, the normal distribution with average 0 and spread 1. Then
x_t = \sqrt{\bar\alpha_t}\, x_0 + \sqrt{1 - \bar\alpha_t}\, \epsilon
The symbol \bar\alpha_t (read "alpha-bar t") is a number between 0 and 1, fixed in advance for every step by a noise schedule. It says how much of the original image survives at step t: close to 1 at the start, close to 0 at the end. The formula mixes the image and the noise in proportions set by \bar\alpha_t, and the square roots keep the overall size of the pixel values steady as one is traded for the other.
Follow one pixel through it. Its original value is x_0 = 0.8. At a step where \bar\alpha_t = 0.25, so a quarter of the image's strength remains, and with a noise sample of \epsilon = -0.4:
x_t = \sqrt{0.25} \times 0.8 + \sqrt{0.75} \times (-0.4) = 0.5 \times 0.8 + 0.866 \times (-0.4) = 0.400 - 0.346 = 0.054
The pixel that was 0.8 is now 0.054, mostly noise.
What the network learns
The network, written \epsilon_\theta (its learned weights are \theta, "theta"), is given the noisy image x_t and the step number t, and asked to predict the noise \epsilon that was mixed in. Training is simple: take a training image, pick a random step, draw random noise, build x_t with the formula, and measure how far the network's prediction is from the true noise with the squared error of 12.3.1:
\text{loss} = \lVert \epsilon - \epsilon_\theta(x_t, t) \rVert^2
Repeat that over hundreds of millions of images and every noise level, adjusting the weights by gradient descent (12.2.4), and the network becomes very good at seeing which part of a noisy image is noise.
Why is predicting the noise enough? Rearrange the forward formula: if you know x_t and the noise, you can solve for the original image.
\hat x_0 = \frac{x_t - \sqrt{1 - \bar\alpha_t}\, \hat\epsilon}{\sqrt{\bar\alpha_t}}
For our pixel, a perfect prediction \hat\epsilon = -0.4 gives back (0.054 + 0.346) / 0.5 = 0.8, the original value. A slightly wrong prediction, \hat\epsilon = -0.3, gives (0.054 + 0.260) / 0.5 \approx 0.63: close, but not exact. That is why generation does not jump from static to image in one step. Each step uses the current estimate to move only a little way towards a cleaner image, adds back a small amount of fresh noise to stay on track, and lets the next step correct the remaining error.
Generating an image
Put together, generation is a loop:
python
import torch
def generate(denoiser, schedule, shape, text_embedding, steps=50):
x = torch.randn(shape) # (1) pure random static
for t in schedule.timesteps(steps): # (2) from noisiest to cleanest
eps = denoiser(x, t, text_embedding) # (3) predict the noise
x = schedule.step(eps, t, x) # (4) move a little towards clean
return x # an image that never existedLine (1) starts from noise drawn from the normal distribution, the same shape as an image. Line (2) runs through the steps from the noisiest to the cleanest. Line (3) asks the trained network for its estimate of the noise in the current image, given the step and the text (section 3). Line (4) applies the update rule of the chosen sampler, which uses the estimate to move part of the way towards a cleaner image. The original method used about 1,000 steps; improved samplers reach similar quality in 20 to 50, and recent distilled models, trained to imitate many steps in a few, need as few as 1 to 4, which is what made generation fast enough for interactive use.
3. Steering with text
An unconditioned diffusion model generates some image that looks like its training data. To generate the image described by a prompt, the text must steer every denoising step.
Encoding the text. The prompt is turned into vectors by a text encoder: CLIP's text half, from 12.6.2, or a large language-model encoder. Larger, better text encoders follow long, detailed prompts more faithfully.
Feeding it in. Inside the denoising network, cross-attention layers let each region of the image attend to the words of the prompt, the attention of 12.7.3 with the image as queries and the text as keys and values. The region becoming the lamp attends to "brass lamp"; the region becoming the table attends to "wooden bedside table".
Pushing harder: classifier-free guidance. Left alone, a conditioned model often follows the prompt only loosely. A technique by Jonathan Ho and Tim Salimans, called classifier-free guidance, strengthens it. At each step, the network predicts the noise twice: once with the prompt, \epsilon_{\text{cond}}, and once with an empty prompt, \epsilon_{\text{uncond}}. The difference between the two is the direction the prompt is pushing in. Guidance exaggerates that direction by a factor w, the guidance scale:
\hat\epsilon = \epsilon_{\text{uncond}} + w \,(\epsilon_{\text{cond}} - \epsilon_{\text{uncond}})
With w = 1, this is just the conditioned prediction. With larger w, the image is pushed further towards the prompt. For a single number, if the unconditioned prediction is 0.10 and the conditioned one 0.30, then w = 1 gives 0.30, w = 3 gives 0.10 + 3 \times 0.20 = 0.70, and w = 7.5 gives 1.60. Typical values are around 5 to 8: too low, and the image ignores parts of the prompt; too high, and colours become garish and details unnatural. It is a dial between following the prompt and looking natural.
4. Making it practical: working in a smaller space
Running hundreds of denoising steps on a full-resolution image is expensive: a 1,024-by-1,024 image has over three million numbers. The decisive improvement was latent diffusion, by Robin Rombach and colleagues at LMU Munich and the company Runway, published in 2022 and released as the open model Stable Diffusion that August.
The idea is to run diffusion not on pixels but on a compressed version of the image. A separate network, an autoencoder, is first trained to squeeze an image into a much smaller grid of numbers, its latent representation, and to expand that grid back into the image with little loss. For Stable Diffusion, a 512-by-512 image becomes a 64-by-64 latent grid, about 48 times fewer numbers. Diffusion happens entirely in that small space, and only at the end is the result expanded back to pixels. Generation became fast and cheap enough to run on a consumer graphics card, which, together with the open release, set off the wave of image tools of the following years.
Two further developments shape today's models. The denoising network, originally a convolutional design called a U-Net, has increasingly been replaced by a transformer working on patches of the latent, the diffusion transformer (William Peebles and Saining Xie, 2023), which scales better in the familiar transformer way. And video models apply the same ideas across space and time together, denoising a block of frames at once so that motion stays consistent; their main difficulties are keeping objects and backgrounds stable over many frames and making motion obey physics.
5. Changing part of an image
For a retailer, generating images from nothing is less useful than editing real ones, and diffusion handles that naturally.
Image-to-image. Instead of starting from pure noise, add a moderate amount of noise to an existing image and denoise it with a new prompt. The result keeps the original's overall layout and changes its style or details; a strength setting chooses how much noise is added, and so how much changes.
Inpainting. Mark part of an image with a mask, and regenerate only that part, so that it blends with the rest. This is what Cartwise uses for backgrounds.
Structural control. A method called ControlNet (Lvmin Zhang and colleagues, 2023) adds a second input that fixes the structure of the result, such as an outline, a pose or a depth map, so the generated image follows a given layout exactly.
Cartwise's background tool combines inpainting with the segmentation of 12.13.1, arranged so that the product itself is never generated:
python
import torch
from diffusers import AutoPipelineForInpainting
pipe = AutoPipelineForInpainting.from_pretrained(
"diffusers/stable-diffusion-xl-1.0-inpainting-0.1", torch_dtype=torch.float16).to("cuda")
product_mask = segment_product(photo) # (1) pixels that are the product
background_mask = invert(product_mask) # (2) everything else may change
result = pipe(prompt="on a wooden bedside table in a softly lit bedroom, photograph",
negative_prompt="text, logo, watermark, extra objects",
image=photo, mask_image=background_mask,
guidance_scale=6.0, num_inference_steps=30, strength=0.99).images[0] # (3)
final = paste_original_pixels(result, photo, product_mask) # (4)Line (1) finds exactly which pixels belong to the product, using a segmentation model. Line (2) inverts that mask, so only the background is regenerated. Line (3) generates a new background from the prompt, with a negative prompt listing things the image must not contain, a moderate guidance scale, and a strength near 1 so the old background is fully replaced. Line (4) is the safeguard that matters most: it pastes the original product pixels back over the result, so that even if the model subtly altered the product's edges, colour or shape, the customer sees the product exactly as photographed.
6. What a retailer must not do
Image generation makes it very easy to show customers something that is not true, and in e-commerce that is a legal and ethical line, not a matter of taste. Cartwise's rules:
- Never generate or alter the product. Its colour, shape, size, texture, labels and condition come from real photos. The pasting step above enforces this in code, and a reviewer compares before and after for every new template.
- Never imply what is not included. A generated scene must not show accessories, quantities or settings that suggest the customer gets more than they do.
- Label generated imagery where customers might otherwise assume a real photograph, and keep the original photo available.
- Keep records of provenance. Standards exist for attaching a record of how an image was made, a record that anyone can check. The C2PA standard comes from the Coalition for Content Provenance and Authenticity, a group founded in 2021 by companies including Adobe and Microsoft. It attaches signed "content credentials" to the image file. Separately, invisible watermarks such as Google's SynthID mark generated images in a way that survives resizing. Cartwise attaches credentials to every generated asset.
- Respect rights. Generated images must not imitate a living artist's distinctive style on request, reproduce trademarks or real people, or copy recognisable copyrighted work. Legal disputes over whether training on copyrighted images was lawful are ongoing; a company using these models should follow its lawyers' guidance on which models and uses are acceptable.
The same models are used to make convincing fake images of real people and events, which is why provenance standards matter beyond retail. For Cartwise's own processes, it is also why a customer's photo is never treated as proof on its own, the point made in 12.13.1.
What the interviewer will push on
"How does a diffusion model work?" Training adds noise to images at random levels using x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon and teaches a network to predict the noise, with a squared-error loss. Generation starts from random noise and removes predicted noise step by step. Explain why predicting noise is enough: the formula can be solved for x_0.
"How does the text prompt control the image?" A text encoder produces vectors that the denoiser attends to through cross-attention; classifier-free guidance compares predictions with and without the prompt and exaggerates the difference by the guidance scale, trading prompt-following against naturalness.
"What is latent diffusion and why did it matter?" Diffusion in the compressed space of an autoencoder (64×64 instead of 512×512 for Stable Diffusion), roughly 48 times fewer numbers, making generation cheap enough for ordinary GPUs; its open release in 2022 drove widespread use.
"Why did diffusion replace GANs?" More stable training and a wider variety of outputs; GANs train two networks against each other, which is fragile and tends to collapse to a narrow range of images.
"How would you use image generation for product photos?" Segment the product, regenerate only the background by inpainting, paste the original product pixels back, label generated imagery and attach provenance. Never let the model alter the product.
Something to volunteer: fewer steps is where much of the recent progress lies. The original method needed about a thousand denoising steps; distilled models now produce good images in one to four, which is what turned image generation from a batch job into something interactive.
Recall
- Diffusion: the forward process adds noise until only static remains (no learning); a network learns to predict the noise at every level; generation starts from static and removes predicted noise step by step. Sohl-Dickstein et al. (2015); DDPM (Ho, Jain, Abbeel, 2020). It replaced GANs through stabler training and more varied outputs.
- Forward in one step: x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1 - \bar\alpha_t}\,\epsilon; training loss \lVert \epsilon - \epsilon_\theta(x_t, t)\rVert^2; recovering \hat x_0 = (x_t - \sqrt{1-\bar\alpha_t}\,\hat\epsilon)/\sqrt{\bar\alpha_t} explains why predicting noise suffices. Samplers need 20–50 steps; distilled models 1–4.
- Text steers through a text encoder and cross-attention; classifier-free guidance \hat\epsilon = \epsilon_{\text{uncond}} + w(\epsilon_{\text{cond}} - \epsilon_{\text{uncond}}), typically w \approx 5–8.
- Latent diffusion (Rombach et al., 2022; Stable Diffusion) works in an autoencoder's compressed space (64×64 for 512×512). Diffusion transformers scale further; video denoises frames together.
- Editing: image-to-image (strength), inpainting (mask), ControlNet (structure).
- For products: segment, regenerate only the background, paste original product pixels back, label, attach C2PA credentials or watermarks, respect rights. Never alter the product.
Self-test: With x_0 = 0.5, \bar\alpha_t = 0.64 and \epsilon = 0.5, what is x_t? What happens to an image as the guidance scale rises from 1 to 15? Which line of the background tool guarantees the product is shown exactly as photographed, and why is a negative prompt not enough on its own?
Next: 12.13.4 Machine Translation — the last multimodal task is the oldest problem in the field: how computers learned to translate, from hand-written rules in the 1950s through statistics and neural networks to language models, and how Cartwise translates its catalogue into eleven languages.