AI Foundations

Autoencoders and VAEs

An autoencoder is a neural network that compresses input into a low-dimensional bottleneck and reconstructs it, learning features without labels. A Variational Autoencoder (VAE) makes that bottleneck a probability distribution, turning the model into a true generator.

What an autoencoder is

An autoencoder is a neural network trained to copy its input to its output. It is built from two parts: an encoder that maps the input x to a compressed internal code z, and a decoder that maps that code back to a reconstruction of x. The code lives in a layer called the bottleneck, which is usually much smaller than the input. Because the network must squeeze the data through this narrow layer and still rebuild it, it is forced to discover an efficient representation rather than memorize raw pixels or tokens.

Training uses no labels. The target is the input itself, so an autoencoder is a form of self-supervised or unsupervised learning. The objective is a reconstruction loss that measures how far the output is from the original. For continuous data such as images, mean squared error between input and reconstruction is common. For binary or normalized data, a binary cross-entropy loss is often used instead. Gradient descent adjusts encoder and decoder weights to drive this loss down.

The width of the bottleneck defines the type of autoencoder. An undercomplete autoencoder has a bottleneck smaller than the input, which prevents the trivial identity solution and pushes the model to capture only the most informative structure. An overcomplete autoencoder has a code larger than the input and needs an extra constraint, such as sparsity or noise, to avoid simply copying values straight through.

Encoder, bottleneck, and decoder are the three structural pieces of every autoencoder.
The training signal is reconstruction error between the input and the rebuilt output, so no labels are required.
Undercomplete models compress; overcomplete models need an added constraint to learn anything useful.
The learned bottleneck code is a compact feature representation that can be reused for other tasks.

Denoising and sparse variants

A denoising autoencoder corrupts the input before feeding it to the encoder, then asks the decoder to reconstruct the original clean version. Vincent, Larochelle, Bengio, and Manzagol introduced this idea in their 2008 paper "Extracting and Composing Robust Features with Denoising Autoencoders" at the 25th International Conference on Machine Learning. Forcing the network to undo corruption stops it from learning the identity function and pushes it toward features that capture the underlying structure of the data rather than surface noise.

A sparse autoencoder takes a different route. Instead of shrinking the bottleneck, it allows a wide code but adds a penalty that keeps most code units near zero for any given input. Only a small subset of units may activate at once, so each unit learns to specialize on a distinct pattern. Sparsity is typically encouraged with an L1 penalty on the activations or a Kullback-Leibler term that nudges the average activation toward a small target value.

These constraints share a goal: prevent the network from cheating its way to perfect reconstruction without learning meaning. Denoising adds corruption, sparsity adds a budget on active units, and both produce representations that generalize better than a naive copy. Sparse autoencoders in particular have returned to prominence in mechanistic interpretability, where they decompose a model's internal activations into many human-readable features.

Denoising autoencoders reconstruct a clean input from a deliberately corrupted copy (Vincent et al., 2008).
Sparse autoencoders keep a wide code but force most units to stay inactive, so each specializes.
Both variants block the identity shortcut and yield more informative features.
Sparsity is enforced with an L1 activation penalty or a KL target on mean activation.

Why a plain autoencoder is not generative

A standard autoencoder learns to encode and decode the specific examples it sees, but it places no structure on the space of codes. After training, the encoder maps real inputs to scattered points, and the regions between those points may decode to nothing meaningful. There is no guarantee that the latent space is continuous or that any chosen point corresponds to a plausible sample.

This breaks the obvious recipe for generation. To create a new sample one would like to draw a random code and run it through the decoder. With a plain autoencoder there is no distribution to draw from. Picking a random vector usually lands in a dead zone of the latent space, and the decoder produces blurry or incoherent output. The model can reconstruct, but it cannot reliably sample.

The missing ingredient is a known, well-shaped distribution over the latent space. If the encoder could be trained so that codes are spread according to a simple prior, such as a standard Gaussian, then sampling from that prior and decoding would produce new data. Achieving this is exactly what the Variational Autoencoder adds.

A plain autoencoder optimizes only reconstruction, so its latent space has no enforced shape.
Random points between encoded examples often decode to meaningless output.
Without a known latent distribution there is nothing principled to sample from.
Reliable generation requires the latent codes to follow a simple, samplable prior.

The Variational Autoencoder and the ELBO

The Variational Autoencoder, introduced by Diederik Kingma and Max Welling in the 2013 paper "Auto-Encoding Variational Bayes" (arXiv:1312.6114, submitted December 2013), reframes the autoencoder as a probabilistic model. Rather than mapping an input to a single code, the encoder outputs the parameters of a distribution over codes, typically a mean vector and a variance vector for a diagonal Gaussian. The decoder then defines a distribution over data given a sampled code. A prior, usually a standard normal, is placed over the latent variable.

Training maximizes the evidence lower bound, or ELBO, a tractable lower bound on the log-likelihood of the data. The ELBO has two terms. The first is an expected reconstruction term that rewards codes which decode back to the input, playing the same role as ordinary reconstruction loss. The second is a Kullback-Leibler divergence that measures how far the encoder's per-input distribution sits from the prior, and it is subtracted as a penalty. Maximizing the ELBO is equivalent to minimizing reconstruction error plus this KL regularizer.

The KL term is what makes a VAE generative. By pulling every input's latent distribution toward the same standard normal prior, it packs the codes into a continuous, well-covered region. After training, sampling a vector from the standard normal prior and passing it through the decoder produces a fresh, coherent sample, which a plain autoencoder cannot do.

log p(x) ≥ ELBO = E_{q(z|x)}[ log p(x|z) ] − KL( q(z|x) ∥ p(z) )

The evidence lower bound: an expected reconstruction term minus the KL divergence from the encoder's posterior q(z|x) to the prior p(z), usually a standard normal.

A VAE encoder outputs a distribution (mean and variance), not a single point.
The ELBO is a lower bound on the data log-likelihood and the quantity actually optimized.
Its two terms are a reconstruction reward minus a KL divergence to the prior.
The KL term shapes the latent space toward the prior, enabling sampling and generation.

The reparameterization trick

There is a practical obstacle to training a VAE. The encoder produces a distribution, and a code must be sampled from it to feed the decoder. Sampling is a random operation, and gradients cannot flow back through a raw random draw, so the encoder weights could not be updated by backpropagation. Kingma and Welling solved this with the reparameterization trick.

Instead of sampling z directly from the Gaussian with mean mu and standard deviation sigma, the trick draws a noise sample epsilon from a fixed standard normal and computes z = mu + sigma multiplied by epsilon. The randomness now lives entirely in epsilon, which has no learnable parameters, while mu and sigma are deterministic functions of the input. Gradients pass cleanly through mu and sigma, so the whole network trains end to end with standard stochastic gradient descent.

This single change is what makes VAEs trainable at scale. It turns an awkward stochastic node into a deterministic transformation of fixed noise, letting the reconstruction and KL terms of the ELBO be optimized jointly by ordinary autograd. The same pattern reappears in many later latent-variable models.

z = μ + σ ⊙ ε,   ε ~ N(0, I)

The reparameterization trick: sample fixed noise epsilon, then scale and shift it by the encoder's outputs so gradients flow through mu and sigma.

python

import torch
import torch.nn as nn
import torch.nn.functional as F

class VAEEncoder(nn.Module):
    def __init__(self, in_dim=784, hidden=400, latent=20):
        super().__init__()
        self.fc = nn.Linear(in_dim, hidden)
        self.fc_mu = nn.Linear(hidden, latent)
        self.fc_logvar = nn.Linear(hidden, latent)

    def forward(self, x):
        h = F.relu(self.fc(x))
        return self.fc_mu(h), self.fc_logvar(h)

def reparameterize(mu, logvar):
    std = torch.exp(0.5 * logvar)
    eps = torch.randn_like(std)  # epsilon ~ N(0, I)
    return mu + std * eps         # z = mu + sigma * epsilon

def vae_loss(recon, x, mu, logvar):
    # reconstruction term (Bernoulli decoder) + KL to standard normal
    recon_loss = F.binary_cross_entropy(recon, x, reduction='sum')
    kl = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())
    return recon_loss + kl       # negative ELBO, to be minimized

A minimal VAE encoder, the reparameterization step, and the negative-ELBO loss (reconstruction plus closed-form Gaussian KL) in PyTorch.

Gradients cannot flow through a direct random sample, blocking backpropagation.
The trick rewrites the sample as z = mu + sigma · epsilon with epsilon from a fixed standard normal.
Randomness is isolated in epsilon while mu and sigma stay differentiable.
This makes the VAE trainable end to end with ordinary gradient descent.

VAEs versus GANs and diffusion, and where they live today

VAEs sit among several families of generative models. A Generative Adversarial Network pits a generator against a discriminator and tends to produce sharper images, but it can be unstable to train and may collapse to a narrow set of outputs. A VAE trains stably by maximizing a likelihood bound and gives an explicit, structured latent space, but its samples are often blurrier because the reconstruction objective and the KL pressure smooth fine detail. Diffusion models generate by gradually denoising random noise over many steps and now lead on raw image quality, at the cost of slower, multi-step sampling.

Rather than competing head to head, these methods increasingly combine. In latent diffusion models, described by Rombach and colleagues in the 2022 CVPR paper "High-Resolution Image Synthesis with Latent Diffusion Models," a VAE first compresses an image from pixel space into a much smaller latent grid. The diffusion process then runs entirely in that compact latent space, and the VAE decoder converts the result back to pixels. This is the architecture behind Stable Diffusion, and the VAE is what makes high-resolution synthesis feasible on modest hardware by removing imperceptible detail before the expensive diffusion steps.

Autoencoders also drive a separate line of work in interpretability. Sparse autoencoders trained on the internal activations of large language models decompose those activations into thousands of sparse, often human-readable features, as in Anthropic's 2023 report "Towards Monosemanticity" and its 2024 follow-up scaling the method to Claude 3 Sonnet. Compression for generation and decomposition for understanding keep the autoencoder idea central in modern systems.

GANs give sharp samples but train unstably; VAEs train stably with a structured latent space but blurrier output.
Diffusion models currently lead on image quality through slow, iterative denoising.
Stable Diffusion uses a VAE to compress images into latents so diffusion runs cheaply in that space.
Sparse autoencoders now decompose LLM activations into interpretable features for mechanistic interpretability.

Key takeaways

An autoencoder compresses input through a bottleneck and reconstructs it, learning features with no labels using a reconstruction loss.
Denoising and sparsity constraints stop the network from learning the trivial identity copy and yield more meaningful representations.
A plain autoencoder is not generative because its latent space has no enforced shape, so random codes usually decode to nothing.
A VAE encodes inputs as distributions and trains by maximizing the ELBO, a reconstruction term minus a KL divergence to a prior.
The reparameterization trick z = mu + sigma * epsilon moves randomness into fixed noise so the network trains by backpropagation.
VAEs power Stable Diffusion by compressing images into latents, and sparse autoencoders now help interpret large language models.

Frequently asked questions

An autoencoder maps each input to a single fixed code and only learns to reconstruct, so its latent space has no enforced structure. A VAE maps each input to a probability distribution over codes and adds a KL penalty that pulls those distributions toward a standard normal prior. This makes the VAE generative: you can sample from the prior and decode to produce new data, which a plain autoencoder cannot do reliably.

A VAE must sample a code from the encoder's distribution before decoding, but gradients cannot flow back through a raw random draw, which would block backpropagation. The reparameterization trick rewrites the sample as z = mu + sigma * epsilon, where epsilon is drawn from a fixed standard normal. The randomness now sits in epsilon while mu and sigma stay differentiable, so the encoder can be trained with ordinary gradient descent.

The ELBO, or evidence lower bound, is a tractable lower bound on the log-likelihood of the data that a VAE maximizes during training. It has two terms: an expected reconstruction term that rewards codes which decode back to the input, and a KL divergence that penalizes how far the encoder's posterior sits from the prior. Maximizing the ELBO is equivalent to minimizing reconstruction error plus the KL regularizer.

Yes. The most prominent role is inside latent diffusion systems such as Stable Diffusion, where a VAE compresses images into a small latent space so the diffusion model can run efficiently and the decoder restores pixels. Autoencoders are also central to mechanistic interpretability, where sparse autoencoders decompose the internal activations of large language models into interpretable features.

GANs typically produce sharper images but can be unstable to train and may suffer mode collapse, where they generate only a narrow variety of outputs. VAEs train more stably by optimizing a likelihood bound and provide an explicit, structured latent space, but their samples are often blurrier because of the smoothing effect of the reconstruction and KL objectives. Many modern systems combine ideas rather than picking one approach.

A denoising autoencoder corrupts the input, for example by adding noise or zeroing some values, and then trains the network to reconstruct the original clean input. This was introduced by Vincent, Larochelle, Bengio, and Manzagol in 2008. Forcing the model to undo corruption prevents it from learning a trivial identity copy and pushes it toward features that capture the underlying structure of the data.

Put the idea into practice

MemX is an AI memory agent built on these ideas: store anything, skip the folders, and find it again by asking in plain English.

Try MemX Free