Classifier-Free Guidance (CFG)

What is Classifier-Free Guidance?

Classifier-free guidance (CFG) is a sampling technique that makes a conditional diffusion model follow its prompt more strongly without using a separate classifier. Introduced by Jonathan Ho and Tim Salimans in 2022, it replaces the earlier classifier guidance, which required training an extra noise-aware classifier whose gradients pushed samples toward the target class.

Instead, CFG trains a single diffusion model that can run both conditionally (given a prompt) and unconditionally (with the prompt dropped). At each denoising step it computes both noise predictions and extrapolates away from the unconditional prediction toward the conditional one. A guidance scale sets how far to push.

• No separate classifier is required, unlike earlier classifier guidance.
• One model produces both conditional and unconditional noise estimates.
• A guidance scale controls how strongly the prompt is followed.

The guidance formula

During training, the conditioning is randomly dropped a fraction of the time (commonly around 10 to 20 percent), so the same network learns both the conditional noise estimate and an unconditional one. At sampling time, the two estimates are combined.

The guided noise prediction starts from the unconditional estimate and moves in the direction of the conditional estimate, scaled by the guidance weight. A scale of 0 ignores the prompt entirely, a scale of 1 recovers the plain conditional model, and larger scales exaggerate prompt adherence.

• One network is trained to be both conditional and unconditional via prompt dropout.
• Guidance extrapolates from the unconditional toward the conditional prediction.
• No separate classifier or its gradients are needed at any point.

The guidance scale: adherence vs diversity

The guidance scale (often labeled CFG scale, with typical values around 7 to 8 for text-to-image) is the central knob. Raising it makes outputs match the prompt more closely and look more saturated and contrast-heavy, but reduces variety across samples and can introduce artifacts or oversaturation when pushed too high.

Lowering the scale produces more diverse, sometimes more natural outputs that may drift from the prompt. The optimal value is task and model dependent: photorealistic models often prefer moderate scales, while some distilled or newer samplers work best at low scales or with no guidance at all.

• Higher scale: stronger prompt adherence, lower diversity, risk of oversaturation.
• Lower scale: more diverse and natural output, weaker prompt match.
• Typical text-to-image values cluster around 7 to 8, but vary by model.
• Each guided step costs two forward passes (conditional and unconditional).

Using CFG in Diffusers

In the Hugging Face Diffusers library, classifier-free guidance is exposed through the guidance_scale argument of a pipeline call. Internally the pipeline batches the conditional and unconditional prompts together, runs one forward pass, and combines the two predictions with the formula above at every step.

from diffusers import StableDiffusionPipeline
import torch

pipe = StableDiffusionPipeline.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-v1-5",
    torch_dtype=torch.float16,
).to("cuda")

image = pipe(
    prompt="a watercolor painting of a lighthouse at dawn",
    negative_prompt="blurry, low quality",  # unconditional branch
    guidance_scale=7.5,                      # CFG strength
    num_inference_steps=30,
).images[0]
image.save("out.png")

• guidance_scale=1.0 disables guidance (plain conditional sampling).
• The negative prompt supplies the unconditional branch in many pipelines.
• Because each step needs two predictions, guidance roughly doubles step cost.

Why CFG replaced classifier guidance

Classifier guidance required a separate classifier trained on noisy images, adding complexity and a second model to maintain, and its quality was bounded by the classifier. CFG removes that dependency: the diffusion model itself supplies both terms, so guidance works for arbitrary conditioning such as free-form text where training a clean classifier would be impractical.

The cost is that each guided step runs the model twice, once with and once without the condition, roughly doubling inference compute per step. This tradeoff is widely accepted because the quality and controllability gains are large, which is why CFG is standard in modern text-to-image systems including Stable Diffusion.

• No extra classifier to train or maintain, unlike classifier guidance.
• Works naturally with free-form text conditioning.
• Costs a second forward pass per denoising step.