AI Foundations

Diffusion Model

A diffusion model is a generative model that learns to create data by reversing a gradual noising process. It is trained to remove noise step by step, turning pure random noise into a coherent sample such as an image.

What a Diffusion Model Is

A diffusion model is a class of generative model that produces new data by learning to undo a corruption process. Training defines a fixed forward process that slowly adds Gaussian noise to a real sample over many steps until almost nothing of the original remains. A neural network then learns the reverse process: given a noisy sample at any step, predict how to make it slightly less noisy. To generate, the model starts from pure noise and applies that learned denoising repeatedly until a clean sample emerges.

The modern formulation was set out by Jonathan Ho, Ajay Jain, and Pieter Abbeel in the 2020 paper Denoising Diffusion Probabilistic Models (DDPM). They framed the method as a latent variable model inspired by nonequilibrium thermodynamics, and showed it could match or beat earlier generative approaches on image benchmarks such as CIFAR-10. The same noising and denoising structure underpins later text-to-image systems.

Diffusion models are most associated with image synthesis, but the framework is general. The same idea has been applied to audio, video, molecular structures, and other continuous data, because the only requirement is a sensible way to add and remove noise on the data type in question.

Generation is iterative: it walks from noise to data over many small denoising steps.
The forward noising process is fixed and has no learned parameters.
Only the reverse denoising process is learned by a neural network.
The 2020 DDPM paper by Ho, Jain, and Abbeel is the standard reference formulation.

The Forward Noising Process

The forward process is a Markov chain that gradually adds Gaussian noise across T steps. At each step it scales the previous sample down slightly and injects a fixed amount of noise governed by a variance schedule beta_t. Because each step depends only on the one before it, the whole chain is fully specified once the schedule is chosen, with no training required.

A useful property is that the result of many noising steps has a closed form. By defining alpha_t as 1 minus beta_t and alpha_bar_t as the running product of those alpha values, a noisy sample at step t can be drawn directly from the clean sample x_0 in a single shot. This means training never has to simulate the full chain; it can jump straight to any step t.

By the final step T, the variance schedule is set so the sample is essentially indistinguishable from standard Gaussian noise. That endpoint is what generation starts from, which is why the same simple noise prior can seed every new sample the model produces.

q(x_t | x_{t-1}) = N(x_t; √(1 - β_t) x_{t-1}, β_t I)

One forward noising step: scale the previous sample by √(1 - β_t) and add Gaussian noise with variance β_t.

q(x_t | x_0) = N(x_t; √(ᾱ_t) x_0, (1 - ᾱ_t) I),  where ᾱ_t = ∏ α_i and α_i = 1 - β_i

Closed form for any step t, so a noisy sample can be drawn directly from x_0 without running the full chain.

The forward process adds Gaussian noise step by step following a schedule beta_t.
alpha_t equals 1 minus beta_t, and alpha_bar_t is the cumulative product of alpha values.
A closed form lets you sample any noised step directly from the clean input.
After T steps the data has become close to pure standard Gaussian noise.

Training: Predicting the Noise

The network is trained to reverse the forward process, but DDPM does this in a specific way: instead of predicting the clean sample or the reverse mean directly, it predicts the noise that was added. A training step picks a random data point x_0, a random step t, and a random Gaussian noise vector epsilon, builds the noised sample using the closed form, and asks the network to recover that epsilon. This is called noise prediction or epsilon-prediction.

Ho and colleagues found that a simplified objective worked better in practice than the full variational bound. The simplified loss, called L_simple, is just the mean squared error between the true noise epsilon and the network's prediction epsilon_theta, averaged over random steps and inputs. Dropping the theoretical weighting terms made training more stable and produced higher quality samples.

The network itself is typically a U-Net, a convolutional architecture with skip connections, conditioned on the step index t so it knows how much noise to expect. Because every training example reduces to predicting one Gaussian noise vector, the objective is simple to implement and scales well with data and compute.

L_simple = E_{t, x_0, ε} [ ‖ ε − ε_θ(x_t, t) ‖² ],  with x_t = √(ᾱ_t) x_0 + √(1 − ᾱ_t) ε

The DDPM simplified objective: predict the noise added to the sample at a randomly chosen step.

python

import torch

def training_loss(model, x0, alpha_bar):
    # x0: clean batch; alpha_bar: tensor of cumulative-product values per step
    B = x0.shape[0]
    t = torch.randint(0, len(alpha_bar), (B,))      # random step per sample
    a_bar = alpha_bar[t].view(B, 1, 1, 1)
    eps = torch.randn_like(x0)                       # true noise
    x_t = a_bar.sqrt() * x0 + (1 - a_bar).sqrt() * eps  # closed-form noising
    eps_pred = model(x_t, t)                         # network predicts the noise
    return ((eps - eps_pred) ** 2).mean()            # L_simple

Minimal DDPM training step: noise the input in closed form, then regress the network onto the noise.

The network predicts the added noise epsilon rather than the clean sample directly.
L_simple is the mean squared error between true and predicted noise.
The simplified loss drops the variational weighting and trains more stably.
A U-Net conditioned on the step index t is the usual backbone.

Sampling: DDPM, DDIM, and the Score View

Generation runs the learned reverse process. Starting from pure Gaussian noise at step T, the model uses its noise prediction to estimate a slightly cleaner sample, optionally adds a small amount of fresh noise, and repeats down to step 0. Standard DDPM sampling is stochastic and uses the same number of steps as training, often several hundred to a thousand, which makes it slow.

Denoising Diffusion Implicit Models (DDIM), introduced by Jiaming Song, Chenlin Meng, and Stefano Ermon in 2020, reformulate sampling as a non-Markovian process with the same training objective. DDIM can produce high quality samples in far fewer steps, reported at 10 to 50 times faster than DDPM, and its deterministic variant maps each starting noise to a single output, which enables smooth interpolation between samples.

A parallel line of work by Yang Song and colleagues, published as Score-Based Generative Modeling through Stochastic Differential Equations in 2021, recast diffusion as a continuous process. In that view the network estimates the score, the gradient of the log data density, and sampling solves a reverse-time stochastic differential equation. DDPM and score-based models are two faces of the same underlying mathematics.

python

import torch

@torch.no_grad()
def ddpm_sample(model, shape, betas, alphas, alpha_bar):
    x = torch.randn(shape)                    # start from pure noise
    for t in reversed(range(len(betas))):
        a, ab, b = alphas[t], alpha_bar[t], betas[t]
        eps = model(x, torch.tensor([t]))     # predicted noise
        mean = (x - b / (1 - ab).sqrt() * eps) / a.sqrt()
        x = mean if t == 0 else mean + b.sqrt() * torch.randn_like(x)
    return x

Minimal DDPM reverse loop: walk from noise to a sample, denoising one step at a time.

DDPM sampling is stochastic and typically uses hundreds to a thousand steps.
DDIM reaches comparable quality 10 to 50 times faster and can be deterministic.
Fewer sampling steps trade some fidelity for much lower generation cost.
The score-based SDE view by Song et al. unifies diffusion and score matching.

Conditioning and Latent Diffusion

To control what a diffusion model generates, the network is conditioned on extra information such as a text prompt or a class label. Classifier-free guidance, introduced by Jonathan Ho and Tim Salimans in 2022, is the standard technique. The model is jointly trained with and without the condition, and at sampling time the conditional and unconditional noise predictions are combined with a guidance scale that trades sample diversity for closer adherence to the prompt, without needing a separate classifier.

Running diffusion directly on high resolution pixels is expensive. Latent diffusion models, proposed by Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer in 2022, address this by first compressing images into a smaller latent space with a pretrained autoencoder, then running the entire diffusion process there. Cross-attention layers inject the text condition into the denoising network.

This latent diffusion design is the technical basis of Stable Diffusion. Operating in a compact latent space cut the compute needed for high resolution synthesis enough to make text-to-image generation practical on consumer hardware, which is a large part of why diffusion-based image models became widely accessible.

Conditioning feeds a text prompt or class label into the denoising network.
Classifier-free guidance mixes conditional and unconditional predictions, no classifier needed.
Latent diffusion runs the process in a compressed autoencoder space to save compute.
Latent diffusion (Rombach et al., 2022) is the foundation of Stable Diffusion.

Diffusion Versus GANs and Autoregressive Models

Before diffusion took over image generation, generative adversarial networks (GANs) were dominant. GANs pit a generator against a discriminator and produce samples in a single forward pass, which is fast, but training can be unstable and prone to mode collapse, where the generator covers only part of the data distribution. Diffusion models trade speed for stability: training is a straightforward regression on noise, and they tend to cover the data distribution more fully, though sampling takes many steps.

Autoregressive image models generate one pixel or one token at a time, conditioning each output on those already produced. They model the data likelihood directly and can be very high quality, but generation is inherently sequential and slow at high resolution. Diffusion models instead refine the whole image in parallel at each step, which suits the spatial structure of images.

There is no single winner. Diffusion models currently lead much of high quality image and video synthesis, GANs remain useful where single-pass speed matters, and autoregressive transformers dominate language and increasingly compete in multimodal generation. The boundaries continue to shift as methods such as few-step distillation narrow diffusion's sampling cost.

GANs generate in one pass but can be unstable and suffer mode collapse.
Diffusion training is a stable regression and covers the data distribution well.
Autoregressive image models are high quality but generate sequentially and slowly.
Distillation methods are steadily shrinking diffusion's many-step sampling cost.

Key takeaways

A diffusion model generates data by reversing a fixed Gaussian noising process step by step.
DDPM trains a network to predict the noise added at a random step using the L_simple loss.
A closed form lets training noise any step directly from the clean sample.
DDIM and step reduction make sampling far faster than the original DDPM loop.
Classifier-free guidance steers generation toward a prompt without a separate classifier.
Latent diffusion runs the process in a compressed space and underpins Stable Diffusion.

Frequently asked questions

It is a model that learns to turn random noise into realistic data, such as an image. During training it watches data being corrupted with noise and learns to reverse that corruption. To generate, it starts from pure noise and cleans it up step by step until a coherent sample appears.

Generation begins with a tensor of pure Gaussian noise. The network repeatedly predicts the noise present and removes a portion of it, moving from a fully noisy image toward a clean one over many steps. After the final step the result is a new image that resembles the training data.

DDPM is the original sampling method; it is stochastic and uses the same large number of steps as training, which is slow. DDIM reformulates sampling as a non-Markovian process with the same training objective, reaching similar quality 10 to 50 times faster and offering a deterministic variant. Both can use a network trained the same way.

A GAN generates a sample in a single forward pass by training a generator against a discriminator, which is fast but can be unstable and miss parts of the data distribution. A diffusion model generates over many denoising steps using a stable regression objective and tends to cover the distribution more fully. The tradeoff is that diffusion sampling is slower.

Latent diffusion runs the noising and denoising process inside a compressed latent space produced by a pretrained autoencoder, rather than on full-resolution pixels, which sharply reduces compute. It was introduced by Rombach and colleagues in 2022 and is the technical basis of Stable Diffusion. This efficiency is what made high quality text-to-image generation feasible on consumer hardware.

It controls how strongly a diffusion model follows its conditioning, such as a text prompt. The model is trained both with and without the condition, and at sampling time the two predictions are combined with a guidance scale. Higher guidance pushes samples to match the prompt more closely at the cost of diversity, and no separate classifier is required.

Put the idea into practice

MemX is an AI memory agent built on these ideas: store anything, skip the folders, and find it again by asking in plain English.

Try MemX Free