AI Foundations

Generative Adversarial Network (GAN)

A Generative Adversarial Network (GAN) is a generative model in which two neural networks, a generator and a discriminator, train against each other in a minimax game: the generator learns to produce realistic synthetic data while the discriminator learns to tell real data from generated data. Introduced by Ian Goodfellow and colleagues in 2014, GANs powered a wave of photorealistic image synthesis before diffusion models overtook them for high-fidelity generation around 2021.

What a GAN Is

A Generative Adversarial Network is a framework for training a generative model through an adversarial process. It was introduced in the 2014 paper "Generative Adversarial Networks" by Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio (arXiv:1406.2661). The setup pits two neural networks against each other so that improvement in one forces improvement in the other.

The first network is the generator G. It takes a random noise vector z, usually sampled from a simple prior such as a Gaussian or uniform distribution, and maps it to a sample in the data space, for example an image. The second network is the discriminator D. It receives a sample and outputs a single probability estimating whether that sample came from the real training data or from the generator.

Training is a competition. The generator tries to fool the discriminator into accepting its outputs as real, while the discriminator tries to correctly separate real samples from fake ones. As both networks improve, the generator is pushed toward producing samples that match the distribution of the real data. At the theoretical equilibrium, the generator reproduces the data distribution and the discriminator outputs 1/2 everywhere because it can no longer tell the two apart.

Introduced by Goodfellow et al. in 2014 (arXiv:1406.2661).
Two networks: a generator G and a discriminator D, trained jointly.
G maps random noise z to synthetic samples; D scores real versus fake.
At equilibrium G matches the data distribution and D outputs 1/2.

The Minimax Objective

GAN training is framed as a two-player minimax game with a single value function V(D, G). The discriminator maximizes this value by assigning high probability to real samples and low probability to generated ones, while the generator minimizes it by producing samples that the discriminator scores as real.

In the value function below, x is a real sample drawn from the data distribution p_data, and z is a noise vector drawn from the prior p_z. The term D(x) is the discriminator's probability that a real sample is real, which it wants near 1. The term D(G(z)) is its probability that a generated sample is real, which the discriminator wants near 0 and the generator wants near 1.

Goodfellow et al. note a practical issue with this exact formulation. Early in training, when the generator is weak, D(G(z)) is close to 0 and the generator's term log(1 - D(G(z))) saturates, giving very small gradients. The standard fix is the "non-saturating" objective, where the generator instead maximizes log D(G(z)). This provides stronger gradients early on while keeping the same fixed point.

min_G max_D V(D, G) = E_{x~p_data}[log D(x)] + E_{z~p_z}[log(1 - D(G(z)))]

The GAN minimax value function. D maximizes it by scoring real x near 1 and fakes G(z) near 0; G minimizes it by making D(G(z)) near 1.

Training optimizes one shared value function V(D, G) in opposite directions.
D maximizes V; G minimizes it, hence the min over G and max over D.
The original generator loss saturates when D easily wins early on.
The non-saturating variant has G maximize log D(G(z)) for better gradients.

How Training Works

GANs are trained by alternating gradient updates. In each step, the discriminator is updated to better separate a batch of real samples from a batch of generated samples, then the generator is updated to produce samples that the current discriminator is more likely to label as real. The two networks share no weights and are optimized with standard gradient descent based methods such as stochastic gradient descent or Adam.

There is no explicit likelihood being maximized and no separate loss target for individual samples. The generator never sees real data directly. Its only learning signal is the gradient that flows back through the discriminator, which tells it how to change its outputs to look more real to D at that moment. Because D itself keeps changing, the target the generator is chasing is a moving one.

The pseudocode below shows one minimal alternating step. In practice the discriminator and generator are often updated once each per iteration, though some recipes update the discriminator multiple times per generator update.

python

# One alternating GAN training step (minimal pseudocode)
for real_batch in data_loader:
    # 1) Train the discriminator
    z = sample_noise(batch_size)
    fake = G(z).detach()                # stop gradient into G
    d_loss = -(log(D(real_batch)) + log(1 - D(fake))).mean()
    d_loss.backward()
    opt_D.step(); opt_D.zero_grad()

    # 2) Train the generator (non-saturating objective)
    z = sample_noise(batch_size)
    g_loss = -log(D(G(z))).mean()       # G wants D(G(z)) -> 1
    g_loss.backward()
    opt_G.step(); opt_G.zero_grad()

A single alternating update: the discriminator learns to separate real from fake, then the generator learns to fool the current discriminator.

Each iteration alternates a discriminator update and a generator update.
The generator learns only through gradients passed back from the discriminator.
Common optimizers are Adam or stochastic gradient descent.
The discriminator may be updated more often than the generator in some recipes.

Why GANs Are Hard to Train

GAN training is famously unstable. Because the generator and discriminator optimize a shared objective in opposite directions, the process is a dynamic game rather than a simple loss to descend. The system can oscillate, stall, or diverge instead of settling at the desired equilibrium, a problem usually called non-convergence.

A second failure mode is mode collapse, where the generator maps many different noise vectors to a small set of near-identical outputs. It finds a few samples that reliably fool the current discriminator and stops covering the full diversity of the real data. The result is a model that produces sharp but repetitive samples.

A third issue is vanishing gradients. When the discriminator becomes too accurate, it confidently rejects every fake, the loss surface flattens, and almost no useful gradient reaches the generator. Balancing the two networks so neither overpowers the other is one of the core practical challenges of training GANs.

Non-convergence: the adversarial game can oscillate or diverge instead of stabilizing.
Mode collapse: the generator produces a narrow set of outputs, ignoring data diversity.
Vanishing gradients: a too-strong discriminator starves the generator of learning signal.
Stable training requires carefully balancing the capacities of G and D.

Key Variants

Many architectures built on the original GAN to improve stability or target specific tasks. DCGAN (Deep Convolutional GAN, Radford, Metz, and Chintala, 2015, arXiv:1511.06434) established convolutional architecture guidelines that made image GANs train far more reliably. Conditional GAN (Mirza and Osindero, 2014, arXiv:1411.1784) feeds a label or other condition y to both networks so generation can be steered, for example producing a specific digit class.

WGAN (Wasserstein GAN, Arjovsky, Chintala, and Bottou, 2017, arXiv:1701.07875) replaces the original objective with one based on the Wasserstein distance, which yields more meaningful training curves and reduces mode collapse. StyleGAN (Karras, Laine, and Aila, 2018, arXiv:1812.04948) introduced a style-based generator that gives scale-specific control over generated images and produced highly realistic faces.

Other variants target image translation. Pix2Pix (Isola et al., 2016, arXiv:1611.07004) uses conditional GANs with paired training images to map one image type to another, such as edge maps to photos. CycleGAN (Zhu et al., 2017, arXiv:1703.10593) removes the need for paired data by enforcing cycle consistency, so translating an image to another domain and back recovers the original.

DCGAN and StyleGAN improved architecture and image quality; StyleGAN added style control.
Conditional GAN steers generation with a label or other input fed to both networks.
WGAN uses the Wasserstein distance for more stable training and fewer collapses.
Pix2Pix (paired) and CycleGAN (unpaired) perform image-to-image translation.

Applications and the Shift to Diffusion

GANs drove major progress in image generation through the late 2010s. Common applications include photorealistic image synthesis, single-image super-resolution that upscales low-resolution images (for example SRGAN, Ledig et al., 2016, arXiv:1609.04802), and image-to-image translation tasks such as colorization, style transfer, and turning sketches into photos.

Around 2021 to 2022 the dominant approach for high-fidelity image generation shifted from GANs to diffusion models. The 2021 paper "Diffusion Models Beat GANs on Image Synthesis" by Prafulla Dhariwal and Alex Nichol (arXiv:2105.05233) showed that a guided diffusion model could surpass the strong BigGAN-deep baseline on ImageNet, reporting Frechet Inception Distance scores such as 2.97 at 128x128 resolution. Diffusion models also tend to cover the data distribution more fully, avoiding the mode collapse that plagues GANs.

Diffusion models then became the backbone of widely used text-to-image systems. GANs remain useful where fast single-pass generation matters, since a GAN produces a sample in one forward pass while standard diffusion sampling requires many iterative steps, but for state-of-the-art image fidelity diffusion largely took over.

Core uses: image synthesis, super-resolution, and image-to-image translation.
Dhariwal and Nichol (2021) showed diffusion models beating BigGAN on ImageNet.
Diffusion models avoid mode collapse and cover the data distribution more fully.
GANs still win on speed, generating a sample in a single forward pass.

Key takeaways

A GAN trains a generator and a discriminator against each other in a minimax game.
The generator turns random noise into samples; the discriminator scores real versus fake.
Its objective is min_G max_D V(D, G) = E[log D(x)] + E[log(1 - D(G(z)))].
Training is unstable, with mode collapse, non-convergence, and vanishing gradients.
Variants like DCGAN, WGAN, StyleGAN, Pix2Pix, and CycleGAN improved stability and reach.
Diffusion models overtook GANs for high-fidelity image generation around 2021 to 2022.

Frequently asked questions

A GAN is a pair of neural networks that learn by competing. One network, the generator, creates fake data such as images, and the other, the discriminator, tries to spot which samples are fake. As they train against each other, the generator gets better at producing realistic data until the discriminator can no longer reliably tell real from fake.

GANs were introduced by Ian Goodfellow and colleagues at the University of Montreal in the 2014 paper "Generative Adversarial Networks" (arXiv:1406.2661). The co-authors include Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.

Mode collapse is a failure mode where the generator produces only a small variety of outputs instead of covering the full diversity of the training data. It happens when the generator finds a few samples that reliably fool the current discriminator and keeps producing those. Methods like Wasserstein GAN were designed in part to reduce mode collapse.

A GAN generates a sample in a single forward pass of the generator and is trained adversarially against a discriminator, which makes training unstable. A diffusion model generates by iteratively denoising random noise over many steps and is trained on a more stable denoising objective. Diffusion models surpassed GANs on high-fidelity image synthesis around 2021, while GANs remain faster at sampling.

GAN training is a two-player game rather than a simple loss to minimize, so it can oscillate or diverge instead of converging. It also suffers from mode collapse, where the generator produces repetitive outputs, and vanishing gradients, where a too-accurate discriminator stops sending useful learning signal to the generator. Keeping the two networks balanced is the central difficulty.

GANs have been used for photorealistic image synthesis, super-resolution that upscales low-resolution images, and image-to-image translation tasks like colorization, style transfer, and turning sketches into photos. Variants such as StyleGAN became known for generating realistic human faces. For many high-fidelity generation tasks, diffusion models have since become the more common choice.

Put the idea into practice

MemX is an AI memory agent built on these ideas: store anything, skip the folders, and find it again by asking in plain English.

Try MemX Free