Diffusion Models 101

Imagine dropping ink into water. The forward direction is easy: the ink spreads until the original pattern disappears. Reconstructing the exact drop is impossible, but a model can learn the statistical direction toward images that look plausible.

A diffusion model does not uncover a photograph hidden inside a noise field. It follows a learned probability landscape, nudging an arbitrary point toward regions where training-like data is likely to live.

Start with a clean sample x₀. At every timestep, add a small amount of Gaussian noise according to a variance schedule. After enough steps, the result xₜ becomes almost indistinguishable from standard normal noise.

During training, we can skip simulating every earlier step and directly mix the clean sample with fresh noise:

Here, ε is sampled noise and ᾱₜ tells us how much signal survives at timestep t. Early steps retain the scene; late steps retain almost none of it.

A neural network—commonly a U-Net or diffusion transformer—receives the noisy sample and its timestep. In the standard DDPM formulation, it is trained to predict the noise that was added.

The objective is surprisingly plain: compare the true noise with the predicted noise using mean squared error. Repeating this across images and noise levels teaches the network what structure looks like at every stage of corruption.

The optional c is conditioning information such as text, a class label, depth, pose, or another image.

Generation starts at xₜ ∼ N(0, I). A sampler asks the network for a denoising direction, takes one step, and repeats. Early steps determine broad composition; later steps resolve edges, texture, and local detail.

DDPM sampling is stochastic and historically required many sequential steps. DDIM showed that the same training objective could support a non-Markovian, often deterministic path with far fewer evaluations. Modern schedulers continue the same trade-off: fewer steps improve speed, while more careful trajectories can improve fidelity or diversity.

Text-to-image systems encode a prompt into vectors and expose them to the denoiser, usually through attention. The model then predicts noise in a way that is consistent with both the current latent and the prompt.

Classifier-free guidance evaluates conditional and unconditional predictions, then amplifies their difference. Higher guidance usually makes the prompt more explicit, but too much can flatten color, reduce diversity, and create brittle artifacts. Guidance is a control knob, not a free quality upgrade.

Running every step over full-resolution pixels is expensive. Latent diffusion first compresses an image with an autoencoder, performs diffusion in that smaller representation, and decodes the final latent back to pixels.

Compression removes some imperceptible detail while retaining semantics and spatial structure. That makes training and sampling much cheaper, but the decoder also becomes part of the quality ceiling: details discarded by the representation cannot be recovered perfectly later.

A diffusion model is a learned denoising field. Training teaches local directions at many noise levels. Sampling chains those local directions into a global journey from randomness to data.

Once that clicks, noise schedules, samplers, guidance, latent spaces, and control signals become choices about where the journey happens and how strongly we steer it.