Models & Evaluation

Stable Diffusion

Stable Diffusion is an open-weights text-to-image latent diffusion model, first released in August 2022, that generates images by denoising a compressed latent representation conditioned on a text prompt. Operating in a low-dimensional latent space rather than on raw pixels lets it run on consumer GPUs.

What Stable Diffusion is

Stable Diffusion is a text-to-image generative model that turns a written prompt into an image by gradually removing noise from a random starting point. It belongs to the family of latent diffusion models introduced in the paper High-Resolution Image Synthesis with Latent Diffusion Models by Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer, first posted to arXiv in December 2021 (arXiv:2112.10752) and presented at CVPR 2022.

The defining idea is that diffusion runs in the latent space of a pretrained autoencoder rather than directly on pixels. A typical Stable Diffusion 1.x model compresses a 512 by 512 pixel image into a 64 by 64 latent grid, an 8x reduction per spatial dimension. Because each denoising step then operates on far fewer values, training and image generation become cheap enough to run on a single consumer graphics card with a few gigabytes of memory.

The first public weights were released on August 22, 2022 by Stability AI together with the CompVis group at Ludwig Maximilian University of Munich and Runway, distributed under the CreativeML Open RAIL-M license. Open weights meant anyone could download, run, and fine-tune the model locally, which set Stable Diffusion apart from contemporaries that were available only through hosted APIs.

A latent text-to-image diffusion model: it denoises a compressed image representation guided by text.
Built on the latent diffusion architecture from Rombach et al., arXiv:2112.10752, CVPR 2022.
First public weights released August 22, 2022 under the CreativeML Open RAIL-M license.
Designed to run on consumer GPUs because diffusion happens in a small latent space, not on pixels.

The three components

Stable Diffusion is assembled from three trained networks that handle distinct jobs. The first is a variational autoencoder (VAE). Its encoder maps an input image into a compact latent representation, and its decoder maps a latent back into a full-resolution image. The VAE is what makes the latent-space trick possible: the diffusion process never touches pixels directly.

The second component is a U-Net, the denoising network at the heart of the model. During generation it takes a noisy latent and predicts the noise to subtract, repeated over many steps until a clean latent remains. The U-Net receives the text condition through cross-attention layers, the mechanism the latent diffusion paper introduced to inject conditioning signals such as text or layout into the denoiser.

The third component is a text encoder, which converts the prompt into a sequence of embedding vectors that the cross-attention layers attend to. Stable Diffusion 1.x uses OpenAI's CLIP ViT-L/14 text encoder. Later versions expanded this: SDXL uses two CLIP text encoders, and Stable Diffusion 3 adds a large T5 text encoder alongside two CLIP encoders for stronger prompt understanding.

L_LDM = E_{z, c, ε, t} [ || ε − ε_θ(z_t, t, τ(c)) ||² ]

The latent diffusion training objective: the U-Net ε_θ learns to predict the noise ε added to a noised latent z_t at timestep t, conditioned on the encoded prompt τ(c).

VAE: encodes images into latents and decodes latents back into pixels.
U-Net: the denoiser that iteratively predicts and removes noise from the latent.
Text encoder: turns the prompt into embeddings; SD 1.x uses CLIP ViT-L/14.
Cross-attention layers feed the text condition into the U-Net at each step.

How generation works step by step

Generation starts from pure Gaussian noise in latent space. At each of a fixed number of steps, the U-Net predicts the noise present in the current latent, and a sampler (for example DDIM, Euler, or DPM-Solver) uses that prediction to produce a slightly less noisy latent. After the final step, the VAE decoder converts the cleaned latent into the output image.

The text prompt steers this process at every step through the cross-attention layers. The closer the model is told to follow the prompt, the more the predicted denoising direction is pulled toward content that matches the text. This trade-off between prompt fidelity and sample diversity is controlled by classifier-free guidance.

Classifier-free guidance, introduced by Jonathan Ho and Tim Salimans in 2022 (arXiv:2207.12598), works by running the U-Net twice per step: once with the prompt and once with an empty prompt. The two predictions are combined to amplify the direction the prompt adds. A guidance scale near 7 to 8 is a common default for Stable Diffusion; higher values follow the prompt more strictly but can introduce artifacts.

ε̂(z_t, c) = ε_θ(z_t, ∅) + w · ( ε_θ(z_t, c) − ε_θ(z_t, ∅) )

Classifier-free guidance: the unconditional prediction ε_θ(z_t, ∅) is pushed along the difference toward the prompt-conditioned prediction ε_θ(z_t, c), scaled by guidance weight w.

Start from random latent noise, then denoise over a fixed number of sampler steps.
The U-Net predicts noise each step; a sampler updates the latent toward a clean image.
The VAE decoder turns the final latent into the visible image.
Classifier-free guidance trades prompt fidelity against diversity via the guidance scale.

Versions and lineage

Stable Diffusion 1.x, including the widely used 1.5 checkpoint released in October 2022, generates 512 by 512 images using a single CLIP ViT-L/14 text encoder. Stable Diffusion 2.x, released in November 2022, swapped in an OpenCLIP text encoder and trained at higher base resolution, but produced different prompt behavior that some users found harder to control.

Stable Diffusion XL (SDXL) followed in July 2023, with the paper SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis (arXiv:2307.01952) and the 1.0 weights released on July 26, 2023. SDXL roughly tripled the U-Net size, added a second text encoder, trained natively at 1024 by 1024, and introduced an optional refinement model for a second image-to-image pass.

Stable Diffusion 3, described in Scaling Rectified Flow Transformers for High-Resolution Image Synthesis (arXiv:2403.03206), moved away from the U-Net to a Multimodal Diffusion Transformer (MMDiT) trained with a rectified-flow objective, and uses three text encoders (two CLIP plus a T5). It marked a shift from the original convolutional U-Net design toward transformer-based diffusion.

SD 1.x (incl. 1.5, Oct 2022): 512x512, single CLIP ViT-L/14 text encoder.
SD 2.x (Nov 2022): OpenCLIP text encoder, higher base resolution.
SDXL (1.0 on July 26, 2023): larger U-Net, two text encoders, 1024x1024, optional refiner.
SD3 (arXiv:2403.03206): transformer-based MMDiT with rectified flow and three text encoders.

Using Stable Diffusion in code

The most common way to run Stable Diffusion is the open-source diffusers library from Hugging Face, which wraps the VAE, U-Net, text encoder, and sampler into a single pipeline object. A model is loaded from a checkpoint identifier, and calling the pipeline with a prompt returns generated images.

The guidance_scale argument exposes classifier-free guidance directly, and a torch generator with a fixed seed makes outputs reproducible. The same pipeline interface works across Stable Diffusion versions, so swapping the checkpoint identifier is usually enough to move between SD 1.5, SDXL, and SD3.

Beyond plain text-to-image, the library supports image-to-image, inpainting, and ControlNet through dedicated pipeline classes that reuse the same underlying weights.

python

import torch
from diffusers import AutoPipelineForText2Image

pipeline = AutoPipelineForText2Image.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16,
    variant="fp16",
).to("cuda")

generator = torch.Generator("cuda").manual_seed(31)
image = pipeline(
    "a lighthouse at dusk, cold color palette, detailed, 8k",
    guidance_scale=7.5,
    generator=generator,
).images[0]
image.save("out.png")

A minimal diffusers text-to-image call. The guidance_scale argument controls classifier-free guidance, and the seeded generator makes the result reproducible.

The diffusers library bundles all three components plus the sampler into one pipeline.
guidance_scale sets classifier-free guidance; a fixed seed makes results reproducible.
The same API loads SD 1.5, SDXL, or SD3 by changing the checkpoint identifier.
Separate pipeline classes cover image-to-image, inpainting, and ControlNet.

The ecosystem and common uses

Because the weights are open, Stable Diffusion became the base for a large ecosystem of tools and adaptations. Image-to-image starts denoising from an existing image instead of pure noise, letting a prompt restyle or alter a photo. Inpainting regenerates only a masked region while preserving the rest, useful for object removal or local edits.

Fine-tuning techniques extend the base model to new styles and subjects without retraining it fully. Low-Rank Adaptation (LoRA) trains small adapter matrices that inject a new style or character into the U-Net while keeping the original weights frozen, producing files of a few megabytes that are easy to share. DreamBooth and textual inversion are related personalization methods.

ControlNet adds a parallel network that conditions generation on a structural input such as a depth map, edge map, or human pose, giving precise spatial control over the output. Together these tools turned Stable Diffusion into a general platform for image creation, editing, and controllable synthesis rather than a single fixed model.

Image-to-image and inpainting reuse the same weights for restyling and local edits.
LoRA fine-tunes add styles or subjects as small adapter files with frozen base weights.
ControlNet conditions generation on depth maps, edges, or poses for spatial control.
Open weights enabled a broad ecosystem of fine-tunes, interfaces, and extensions.

Key takeaways

Stable Diffusion is an open-weights latent text-to-image diffusion model first released in August 2022.
It runs diffusion in a compressed latent space, making it usable on consumer GPUs.
Three parts power it: a VAE, a U-Net denoiser, and a CLIP text encoder feeding cross-attention.
Classifier-free guidance, set by the guidance scale, trades prompt fidelity against image diversity.
Versions progressed from SD 1.x to SD 2.x, SDXL (2023), and the transformer-based SD3 (2024).
Open weights spawned an ecosystem of img2img, inpainting, LoRA fine-tunes, and ControlNet.

Frequently asked questions

Stable Diffusion is a model that creates an image from a text description. It starts with random noise and repeatedly cleans it up, guided by the prompt, until a matching image appears. It does this work on a compressed version of the image, which is why it can run on an ordinary gaming GPU.

Its main distinction is that the weights are open and downloadable under the CreativeML Open RAIL-M license, so anyone can run and fine-tune it locally. Technically, it runs diffusion in a low-dimensional latent space rather than on raw pixels, which keeps compute requirements low. Many hosted competitors at its 2022 launch offered only API access.

A variational autoencoder (VAE) that compresses images to and from a latent space, a U-Net that denoises the latent step by step, and a text encoder (CLIP in version 1.x) that turns the prompt into embeddings. The U-Net reads those embeddings through cross-attention so the prompt steers every denoising step.

The guidance scale controls classifier-free guidance, which combines a prompt-conditioned and an unconditioned noise prediction to decide how strongly the prompt is followed. Low values give looser, more varied images; high values follow the prompt closely but can add artifacts. Values around 7 to 8 are common defaults for the original models.

SDXL, released in 2023, kept the U-Net architecture but enlarged it, added a second text encoder, trained at 1024 by 1024, and offered a refinement model. Stable Diffusion 3, from 2024, replaced the U-Net with a Multimodal Diffusion Transformer trained using rectified flow and uses three text encoders for better prompt comprehension.

LoRA (Low-Rank Adaptation) trains tiny adapter weights to add a new style or subject while leaving the base model frozen, producing small shareable files. ControlNet adds a parallel network that conditions generation on a structural input like a pose, depth map, or edge map, giving precise control over the layout of the output.

Put the idea into practice

MemX is an AI memory agent built on these ideas: store anything, skip the folders, and find it again by asking in plain English.

Try MemX Free