Guide

Diffusion models explained

While transformers dominate text generation, diffusion models power most modern image and video synthesis — Stable Diffusion, DALL-E 3, Midjourney-style pipelines, and open-weight checkpoints on Hugging Face. Instead of predicting the next token, a diffusion model learns to remove noise from a corrupted image, one small step at a time, until a sharp picture emerges. That reverse process sounds slow, but clever math (latent spaces, fast samplers, distilled models) made consumer GPUs viable. This guide explains the forward/reverse intuition, how text prompts steer generation, and what trade-offs engineers face when shipping image AI in production.

The core idea: add noise, then learn to undo it

Imagine taking a photograph and sprinkling static over it repeatedly until the image is pure random noise. That is the forward diffusion process: over T timesteps (often hundreds or thousands during training), Gaussian noise is added according to a fixed noise schedule until the original signal is unrecognizable.

The model's job is the opposite — the reverse process. Starting from random noise, it predicts how to remove a little bit of noise at each step, gradually reconstructing structure. After enough steps, you get an image. The breakthrough of denoising diffusion probabilistic models (DDPMs), popularized around 2020, was showing that training a neural network to predict noise (or the slightly less noisy image) at each timestep is stable and produces high-quality samples — better than many earlier generative adversarial networks (GANs) for diverse, mode-covering outputs.

At inference time you do not need as many steps as training used. Modern samplers (DDIM, Euler, DPM-Solver++) skip timesteps intelligently, generating a 512×512 image in 20–30 steps instead of 1,000. Quality versus speed is one of the first knobs practitioners tune.

What the network actually predicts

A DDPM trains a denoiser — historically a U-Net, a convolutional architecture with skip connections that preserves spatial detail. Given a noisy image x_t at timestep t, the network outputs either:

the noise that was added (epsilon prediction), or
the denoised image one step earlier (x0 or v-prediction variants).

Training pairs are cheap to generate: take a real image, sample a random timestep, add the scheduled noise, and supervise the network to recover the target. No adversarial training loop, no mode collapse from a competing discriminator — one reason diffusion overtook GANs for research and product teams.

The U-Net also receives the timestep t as an embedding (sinusoidal or learned) so it knows how much noise remains. Without that signal, the same weights could not distinguish nearly-clean images from snowstorms of static.

Latent diffusion: why Stable Diffusion runs on a GPU you own

Denoising full-resolution RGB tensors is expensive. Latent diffusion (the architecture behind Stable Diffusion) compresses images first with a variational autoencoder (VAE) into a smaller latent tensor — typically 64×64×4 for a 512×512 output. The diffusion U-Net operates in that compressed space, then the VAE decoder upsamples back to pixels.

Most compute shifts from pixel space to latent space, cutting memory and time by an order of magnitude. That is why open-weight SD 1.5 and SDXL checkpoints run on 8–12 GB consumer cards while full-pixel diffusion models often demand datacenter GPUs.

The VAE is frozen during diffusion training; artifacts (blurry faces, bad text rendering) often trace back to VAE limitations rather than the U-Net itself. Newer models sometimes ship improved VAEs or train end-to-end refinements for those failure modes.

Text-to-image: CLIP embeddings and cross-attention

Unconditional diffusion generates random images. Product use cases need text conditioning. Stable Diffusion encodes prompts with a frozen CLIP text encoder, producing a sequence of embedding vectors. Those vectors feed into the U-Net through cross-attention layers — the same attention mechanism described in our transformer guide, but here image features attend to text features so "a red bicycle on a beach" steers denoising toward bicycle-and-sand statistics rather than arbitrary noise patterns.

Classifier-free guidance (CFG) amplifies prompt adherence at inference. The model runs twice per step: once with the text embedding, once with an empty or null prompt. The final noise prediction blends both, pushing the sample toward what the text version prefers and away from the unconditional average. Higher CFG (7–12 is common) increases contrast and literalism but can cause oversaturated colors or artifact halos; lower CFG looks more natural but ignores nuanced prompt details.

Writing prompts is its own skill — overlapping with prompt engineering for LLMs, but with vocabulary tuned to what CLIP saw during training ("cinematic lighting" works; hyper-specific brand names may not). Negative prompts ("blurry, watermark, extra fingers") are a second conditioning channel implemented as separate embeddings in many UIs.

Sampling schedulers and step count

The trained model defines a score — which direction in image space reduces noise. Samplers turn that score into a discrete trajectory from t = T down to t = 0:

Euler / Heun — simple ordinary-differential-equation solvers; fast, good defaults for SDXL.
DDIM — deterministic sampling; useful for reproducible seeds and img2img editing.
DPM-Solver++ — fewer steps with competitive quality; common in production APIs.
Distilled models — student networks trained to denoise in 4–8 steps (SDXL Turbo, LCM-LoRA); sacrifice some detail for real-time previews.

Step count interacts with CFG and resolution. Doubling steps past a saturation point rarely helps; halving steps often costs texture. Benchmark with your exact checkpoint and hardware — the same sampler name in ComfyUI and an API endpoint may use different implementations.

Beyond text-to-image: img2img, inpainting, and video

Diffusion is a framework, not a single product feature. Common variants:

Image-to-image — start denoising from a noised version of a source image instead of pure noise; strength controls how much structure survives (style transfer, upscaling prep).
Inpainting — mask regions; only masked areas receive full noise while context pixels anchor composition (object removal, fill).
ControlNet / adapters — inject edge maps, depth, poses as extra conditioning without retraining the base U-Net.
Video diffusion — extend the U-Net with temporal attention or generate keyframes plus interpolation; compute scales roughly linearly with frames.

Game and media pipelines increasingly use these for concept art, texture variants, and marketing assets — often with human review because hands, text, and brand consistency remain failure modes.

Inference cost, quantization, and deployment

Serving diffusion differs from LLM serving. There is no KV cache growing with output length; cost is dominated by repeated U-Net forward passes (one per sampling step) across large activation maps. A 30-step 1024×1024 SDXL job can exceed the wall-clock time of a short chat completion on the same GPU.

Optimization paths mirror language models in spirit:

FP16 / BF16 — default on NVIDIA datacenter and consumer cards.
INT8 / INT4 weight quantization — reduces VRAM; watch for banding in smooth gradients (see our quantization guide for shared concepts).
TensorRT / ONNX Runtime — kernel fusion and graph optimization for fixed resolutions.
Batching — amortize overhead when generating thumbnails; less useful for interactive single-user edits.

Product teams often route "draft" requests to few-step distilled models and "final" requests to full step counts — a tiered latency strategy similar to small-vs-large LLM routing.

Diffusion vs transformers: when each wins

Transformers excel at discrete sequences with autoregressive factorization — natural for text, code, and tokenized audio. Diffusion excels at high-dimensional continuous signals where iterative refinement beats one-shot prediction — images, audio waveforms, 3D fields, and some molecular designs.

Hybrid systems are common: CLIP (transformer) understands the prompt; the U-Net (convolutional + attention) paints pixels. Newer multimodal transformers generate images as discrete visual tokens (part of some frontier models), blurring the boundary — but latent diffusion remains the workhorse for self-hosted and fine-tuned image stacks in 2026.

Key takeaways

Diffusion trains a denoiser to reverse a fixed noise schedule — stable training, diverse outputs, slower inference than GANs unless optimized.
Latent diffusion (Stable Diffusion) operates in a VAE-compressed space so consumer GPUs can generate 512–1024 px images.
Text conditioning flows through CLIP embeddings and cross-attention; CFG trades naturalness for prompt literalism.
Sampler and step count dominate latency — distilled models and DPM-Solver++ cut steps without retraining the base weights.
Production means tiered quality, quantization awareness, and human review for known failure modes — not raw step maximization.