GAN (Generative Adversarial Network)

Definition: A generative model made of two competing neural networks — a generator that creates fake data and a discriminator that tries to distinguish fake from real — trained together in an adversarial game.

How It Works

The generator maps random noise z (sampled from a simple distribution, e.g., Gaussian) to synthetic samples in the data space (e.g., images) via G(z)
The discriminator D takes a sample (real or generated) and outputs the probability it’s real
Both networks improve through competition: the generator learns to fool the discriminator by producing more realistic outputs, the discriminator learns to catch better fakes
Training alternates: update D to better separate real from fake, then update G to better fool the current D, repeating until (ideally) the generator’s output distribution matches the real data distribution and D can’t do better than random guessing

This adversarial loop — not any single network on its own — is the defining idea of a GAN. Every training iteration runs the same exchange:

Neither network trains against a fixed target the way ordinary supervised learning does — the discriminator’s “correct answer” for a given fake sample changes every time the generator updates, and vice versa. That moving-target dynamic is what makes GAN training qualitatively different from minimizing an ordinary Loss Function, and it’s the root cause of most of the instability discussed under Common Pitfalls below.

Under the Hood

The original minimax objective (Goodfellow et al., 2014): min_G max_D E[log D(x)] + E[log(1 - D(G(z)))] — D is trained to maximize this, G is trained to minimize it
In practice, the generator’s gradient from log(1 - D(G(z))) vanishes early in training when D easily rejects G’s poor early outputs, so implementations typically use the “non-saturating” loss instead: maximize log D(G(z)) for the generator, which provides stronger gradients when G is still weak
At the theoretical optimum, D(x) = 0.5 everywhere — the discriminator can no longer tell real from fake, meaning the generator’s distribution exactly matches the real data distribution
Training is a two-player zero-sum game with no guaranteed convergence — unlike standard gradient descent minimizing a single loss, alternating minimax optimization can oscillate or diverge instead of settling into an equilibrium
Mode collapse: the generator finds a small set of outputs that reliably fool the current discriminator and stops exploring, producing low-diversity samples (e.g., generating the same few faces repeatedly) — a direct consequence of G optimizing against a single, momentarily-fixed D rather than the true data distribution
Common stabilization tricks: label smoothing (train D on soft targets like 0.9 instead of 1.0), feature matching (match statistics of intermediate discriminator layers instead of raw output), minibatch discrimination (let D compare samples within a batch to catch low-diversity generator output), spectral normalization (constrains D’s Lipschitz constant for more stable gradients), and Wasserstein loss with gradient penalty (WGAN-GP, replaces the original JS-divergence-based loss with an Earth Mover’s distance that provides smoother gradients even when real and fake distributions barely overlap)
What the minimax objective is actually optimizing: for a fixed G, the optimal discriminator is D*(x) = p_data(x) / (p_data(x) + p_g(x)). Substituting D* back into the value function shows that training G against an optimal D is equivalent to minimizing the Jensen-Shannon divergence between the real data distribution p_data and the generator’s distribution p_g — a formal way of saying the generator is being pushed toward matching the real distribution exactly, not just toward fooling whatever D happens to be at the moment
That JS-divergence connection also explains a known failure mode: JS divergence is locally flat (near-zero gradient) whenever p_data and p_g have little or no overlapping support, which is the generic case early in training when the generator is still far from realistic — a formal, not just empirical, reason why vanishing gradients showed up so often before non-saturating loss and Wasserstein-based alternatives became standard

The Adversarial Loop — A Worked Numeric Illustration

Consider a heavily simplified toy setup: real data always equals exactly 5, and the generator currently outputs values clustered around 3.

The discriminator has learned D(5) = 0.9 (confidently real) and D(3) = 0.2 (confidently fake)
The generator’s non-saturating loss gradient is driven by log D(G(z)) = log(0.2) — a large-magnitude, strongly corrective signal
Training nudges the generator’s output toward 4.5; the discriminator adapts to D(4.5) = 0.55
The generator’s gradient signal shrinks accordingly — log(0.55) is far closer to zero than log(0.2) was, because there’s less obviously wrong left to fix
At the theoretical fixed point, the generator’s distribution matches the real distribution exactly, D(x) = 0.5 everywhere regardless of input, and the generator’s gradient from D vanishes entirely — there’s nothing left for D to distinguish

A real GAN runs this same dynamic — a confident D gives large corrective gradients, a fooled D gives almost none — over thousands of iterations across an entire distribution rather than one point, but the underlying mechanism is identical.

Variants

DCGAN (2015): established convolutional architecture conventions for stable GAN training (strided convolutions instead of pooling, batch norm in both networks, ReLU/LeakyReLU activations) — became the default starting architecture for image GANs
Conditional GAN (cGAN): feeds a class label or condition into both G and D, allowing controlled generation (e.g., “generate a digit 7” instead of a random digit)
CycleGAN: learns image-to-image translation between two unpaired domains (e.g., horses to zebras, photos to paintings) using a cycle-consistency loss instead of requiring matched training pairs
Pix2Pix: conditional GAN for paired image-to-image translation (e.g., sketches to photos), trained with matched input/output pairs
WGAN / WGAN-GP: replaces the original loss with a Wasserstein distance estimate, addressing training instability and mode collapse with a more theoretically grounded loss and clearer correlation between loss value and sample quality
LSGAN (Least Squares GAN): replaces the sigmoid cross-entropy loss with a least-squares loss, penalizing fake samples based on how far D’s score is from the “real” decision boundary rather than just whether it crosses it — pushes the generator toward samples closer to the real data manifold and reduces vanishing-gradient problems relative to the original formulation
StyleGAN (2018) / StyleGAN2 / StyleGAN3: introduced a style-based generator that injects noise and learned “style” vectors at multiple resolutions, giving fine-grained control over generated image attributes and producing the highly realistic synthetic faces behind sites like thispersondoesnotexist.com
BigGAN: scaled GAN training to very large batch sizes and model capacity, substantially improving image generation fidelity and diversity on complex, multi-class datasets like ImageNet
Progressive GAN (ProGAN, 2017): trains by starting at low resolution (e.g., 4x4) and progressively adding layers to double resolution as training proceeds, stabilizing high-resolution image synthesis and directly setting up the architecture StyleGAN later built on
Self-Attention GAN (SAGAN): adds self-attention layers to both G and D so the model can relate spatially distant regions of an image directly, instead of relying only on the local receptive field of convolutions
InfoGAN: augments the generator’s input with extra latent codes and adds a mutual-information objective encouraging those codes to correspond to disentangled, interpretable factors of variation (e.g., rotation, stroke width) discovered without supervision

Training Dynamics Over Time

The same training loop, run for thousands of iterations, moves through recognizable stages — the loss values alone rarely tell this story clearly, but the visible sample quality does:

Stage 1 — the generator’s weights are randomly initialized, so G(z) is essentially structured noise; the discriminator separates real from fake almost perfectly and provides a strong, easy gradient signal
Stage 2 — the generator starts exploiting the crudest statistical regularities of the real data (average color, rough shape); outputs look like blurry blobs, and D’s job gets slightly harder
Stage 3 — the generator captures coarse structure (a face-like arrangement of features, a digit-like stroke pattern) while D shifts its attention to finer detail (texture, edge sharpness, lighting consistency) to keep winning
Stage 4 — at or near the theoretical equilibrium, the generator’s output distribution overlaps the real data distribution closely enough that D cannot reliably do better than chance; in practice, training is usually stopped once samples look good by eye or by FID score, well before any exact equilibrium is measured

This staged progression is also, in miniature, why GAN training is so often described as unstable: nothing forces the generator and discriminator to move through these stages at compatible speeds, and a discriminator that races ahead to stage-4-level discrimination while the generator is still at stage 1 gives the generator almost no useful gradient to learn from at all.

Comparison

	GAN	Diffusion Model	Autoencoder (vanilla/VAE)
Training stability	Notoriously unstable (adversarial game)	Stable (simple denoising loss)	Stable (reconstruction loss)
Sample quality (images)	High, sharp	Highest currently, very sharp	Blurrier (VAE), N/A for vanilla AE
Sampling speed	Fast (single forward pass)	Slow (many denoising steps, though distillation narrows this)	Fast (single forward pass)
Explicit likelihood	No	Approximate	Yes (VAE, via ELBO)
Mode coverage	Prone to mode collapse	Generally good coverage	Good coverage, but blurry outputs (VAE)
Latent space interpolation	Often smooth and semantically meaningful	Not directly applicable (iterative noise process)	Smooth (VAE), unreliable (vanilla AE)
Typical failure mode	Mode collapse, non-convergence	Slow sampling, high compute cost	Blurry outputs (VAE), trivial identity mapping if misconfigured

See Autoencoder for the encoder-decoder alternative to adversarial generation.

Code Example

import torch
import torch.nn as nn

# Minimal training step sketch — alternating D and G updates
def train_step(G, D, real_batch, opt_G, opt_D, z_dim, device):
    batch_size = real_batch.size(0)
    criterion = nn.BCEWithLogitsLoss()

    # --- Train Discriminator ---
    z = torch.randn(batch_size, z_dim, device=device)
    fake_batch = G(z).detach()  # detach: don't backprop into G here
    d_real = D(real_batch)
    d_fake = D(fake_batch)
    loss_D = criterion(d_real, torch.ones_like(d_real)) + \
             criterion(d_fake, torch.zeros_like(d_fake))
    opt_D.zero_grad(); loss_D.backward(); opt_D.step()

    # --- Train Generator (non-saturating loss) ---
    z = torch.randn(batch_size, z_dim, device=device)
    fake_batch = G(z)
    d_fake = D(fake_batch)
    loss_G = criterion(d_fake, torch.ones_like(d_fake))  # G wants D fooled
    opt_G.zero_grad(); loss_G.backward(); opt_G.step()

    return loss_D.item(), loss_G.item()

Interactive Example — Simulating the Adversarial Dynamic

Real GAN training needs image tensors and thousands of iterations to see anything meaningful, but the core min-max dynamic — one score improving while pushing another down — can be simulated numerically with two scalars. This tracks a generator “quality” score and a discriminator “accuracy” score across a few simulated training rounds:

Watch the two numbers move in opposite directions and both flatten out as they approach the toy equilibrium (generator quality near 1, discriminator accuracy near 0.5) — a deliberately simplified stand-in for the same push-and-pull relationship that drives real generator and discriminator loss curves, without needing any actual image data to demonstrate it.

History

Ian Goodfellow and colleagues introduced GANs in a 2014 paper, reportedly conceived after a bar debate about whether generative models could be trained via a two-network competitive game rather than explicit density estimation
The origin story has more specific detail than just “a bar debate”: Goodfellow has recounted the conversation happening at a bar in Montreal during his PhD at the Université de Montréal, and implementing the first working version of what became the GAN that same night; the resulting paper, “Generative Adversarial Networks,” was published at NeurIPS (then still called NIPS) in December 2014, with Yoshua Bengio among the co-authors
DCGAN (2015) made GANs practically trainable for images at a time when results were often noisy and unstable
Progressive GAN (2017, Karras et al. at NVIDIA) introduced training that starts at low resolution and progressively adds layers to double resolution over the course of training — the direct architectural predecessor to StyleGAN’s approach to stable high-resolution synthesis
2016-2018 saw rapid architectural experimentation (conditional GANs, CycleGAN, Pix2Pix, WGAN) addressing training stability and enabling new applications like unpaired image translation
CycleGAN (Zhu, Park, Isola, and Efros, 2017) and Pix2Pix (Isola, Zhu, Zhou, and Efros, 2017), both out of Berkeley AI Research, became widely reused reference implementations for the image-to-image translation research that followed
StyleGAN (2018-2021) pushed photorealistic face generation to the point of being difficult for humans to distinguish from real photos, raising early public awareness of synthetic media/deepfake concerns
StyleGAN’s creators followed the original 2018 paper with StyleGAN2 (2019, fixing characteristic texture artifacts) and StyleGAN3 (2021, addressing “texture sticking” under motion); the site thispersondoesnotexist.com, launched in 2019 using StyleGAN2, became many people’s first direct encounter with GAN-generated faces indistinguishable from real photographs
From roughly 2021 onward, diffusion models (DALL-E 2, Stable Diffusion, Midjourney) overtook GANs as the dominant approach for high-fidelity image generation, offering more stable training and better mode coverage, though GANs remain relevant where fast single-pass sampling matters more than absolute fidelity
The diffusion models that eclipsed GANs trace back to Sohl-Dickstein et al.’s 2015 theoretical framing and Ho, Jain, and Abbeel’s 2020 “Denoising Diffusion Probabilistic Models” (DDPM) paper, which made diffusion practically competitive; GAN research and deployment continued rather than stopped, with fast single-pass sampling and mature tooling keeping GANs in production use for latency-sensitive or resource-constrained applications

Why It Matters

Was the leading approach for realistic image/video/audio generation before diffusion models became dominant for top-end fidelity
Introduced the influential “adversarial training” concept used elsewhere in ML (e.g., adversarial robustness testing, domain adaptation via adversarial feature alignment)
Still practically relevant where fast sampling matters — a trained GAN generates a sample in a single forward pass, while diffusion models need many iterative denoising steps (though distilled/few-step diffusion variants are closing that gap)
Popularized image-to-image translation tasks (style transfer, super-resolution, colorization, data augmentation for scarce classes) that remain useful production techniques independent of pure “creative” generation
Pushed the field to take generative model evaluation seriously — FID and Inception Score exist largely because GAN loss values are uninformative about sample quality on their own, and both metrics are now used well beyond GAN research
Demonstrated that two simple, jointly-trained networks with no explicit density model could outperform more mathematically principled likelihood-based generative approaches on perceptual sample quality, reshaping what the field treated as a promising research direction for years afterward

Common Pitfalls

Training instability — GANs are notoriously hard to train, prone to mode collapse (generator produces limited variety) or one network overpowering the other (if D becomes too strong too fast, G’s gradients vanish and it stops improving)
Assuming GANs are still state-of-the-art for image generation — diffusion models have largely surpassed them on fidelity and diversity benchmarks in recent years
Judging GAN training solely by the loss curves — unlike standard supervised training, G and D losses don’t monotonically decrease toward some clean optimum, and a “good-looking” loss curve doesn’t guarantee good samples; visual/metric-based inspection (FID score, Inception Score) is necessary
Using an unbalanced learning rate or update frequency between G and D without monitoring, letting one network dominate early and stall the other’s learning signal
Forgetting detach() (or the equivalent) when generating fake samples for the discriminator update, which would otherwise needlessly backpropagate through the generator during the D-only update step
Discriminator overfitting on small training sets — with too little real data, D can memorize training examples instead of learning generalizable “real versus fake” features, leaving G with a poor and eventually meaningless training signal (NVIDIA’s adaptive discriminator augmentation, ADA, was built specifically to address this for small-dataset StyleGAN training)
Trusting a single “best” checkpoint chosen automatically by loss value — because loss curves aren’t monotonic, most practitioners periodically snapshot the generator and evaluate samples visually or via FID instead
Cyclical, non-convergent training — G and D can chase each other through a loop of similar failure modes indefinitely rather than settling near equilibrium, a documented dynamic distinct from simple divergence and not always obvious from the loss curves alone
Underestimating architecture and hyperparameter sensitivity — small changes in learning rate, normalization choice, or network depth can be the difference between stable training and total collapse, far more so than in typical supervised training
Treating a GAN’s discriminator output as a calibrated probability, or the generator as providing any explicit likelihood — D’s score is shaped by whatever the generator currently happens to produce, not a well-calibrated real-world probability, and G provides no density estimate at all

Best Practices

Use the non-saturating generator loss (maximize log D(G(z))) instead of the original minimax formulation to avoid vanishing gradients early in training
Track Fréchet Inception Distance (FID) alongside loss curves — it’s a far more reliable proxy for sample quality and diversity than either network’s loss value
Apply label smoothing and/or instance noise to the discriminator’s real-sample labels to prevent it from becoming overconfident and stalling generator learning
Start from an established, stability-tested architecture (DCGAN conventions, StyleGAN) rather than designing G/D architectures from scratch
Use a two-time-scale update rule (TTUR) — different learning rates for G and D, tuned per problem — since Heusel et al. (2017) showed this improves convergence behavior over using identical learning rates for both networks
Maintain an exponential moving average (EMA) of the generator’s weights during training and sample from the EMA weights rather than the raw training weights — this measurably smooths sample quality, since the raw generator’s weights can oscillate step to step even while the overall trend is improving
Apply spectral normalization or a gradient penalty to the discriminator when instability shows up, rather than only tuning learning rates — both directly constrain D’s function class in ways that improve gradient behavior
Save and evaluate checkpoints at regular intervals rather than relying on a single final checkpoint — the “best” point in GAN training is a sample-quality judgment call, not something the loss curve identifies on its own

Real-World Example

StyleGAN and synthetic faces. NVIDIA’s StyleGAN line (2018-2021) is the most widely recognized GAN application: a style-based generator injects learned “style” vectors and noise at multiple resolutions, giving fine-grained, disentangled control over attributes like pose, hair, and lighting, and producing faces realistic enough that thispersondoesnotexist.com became a widely-shared demonstration of what GANs could do. The same architecture family has been repurposed for generating synthetic artwork, textures, and game assets.

Deepfakes — a capability and a concern in the same technology. GAN-based (and, more recently, diffusion-based) face-swapping and voice-synthesis tools power legitimate film and dubbing work — de-aging actors, replacing stunt performers’ faces, localizing dialogue believably into other languages — using exactly the same generative mechanics that also enable non-consensual synthetic imagery and political disinformation. This dual-use tension drove a parallel research field in deepfake detection, an ongoing arms race where detectors trained on one generation of GANs regularly lose accuracy against the next.

Data augmentation for scarce or imbalanced classes. A GAN trained on a rare category — a manufacturing defect type, an uncommon condition in medical imaging data — can generate additional realistic synthetic examples of that class, helping balance a training set for a downstream classifier without the cost or delay of collecting more real examples. This is one of the more mundane, and more consistently production-relevant, GAN use cases outside of pure image generation for its own sake, and one where GANs remain competitive with newer generative approaches given their fast single-pass sampling.

FAQ

Why is GAN training considered harder than standard supervised training? Standard training minimizes a single, well-behaved loss; GAN training is a simultaneous two-player game where each network’s “loss landscape” shifts as the other network updates, so there’s no guarantee of converging to a stable equilibrium.
What is mode collapse, concretely? The generator learns to produce a narrow subset of plausible outputs (e.g., always the same face, or a handful of digit styles) because that subset already fools the current discriminator — reducing diversity even as individual samples look realistic.
Are GANs and autoencoders solving the same problem? Both are generative, but a GAN has no explicit reconstruction objective or encoder — it only ever sees random noise as generator input, whereas an Autoencoder/VAE explicitly encodes real data into a latent space and decodes it back, giving it a notion of “reconstruction error” a GAN doesn’t have.
Why does the discriminator need to be strong, but not too strong? A weak D gives G no useful signal — almost anything fools it. A D that’s too strong too early drives G’s gradient toward zero from the opposite direction, the vanishing-gradient dynamic described under Under the Hood. The practical goal is keeping the two roughly matched throughout training, not maximizing either network individually.
Can a GAN’s discriminator be reused after training finishes? Rarely directly — D learns to distinguish real data from this specific generator’s evolving output, not to solve a general-purpose classification task, so its usefulness mostly ends with training. Some techniques (feature matching, semi-supervised GANs) do repurpose intermediate discriminator activations as learned features, but that’s a deliberate design choice, not a free byproduct.
Do G and D need identical learning rates and schedules? No, and they often shouldn’t — the two-time-scale update rule (TTUR) deliberately uses different learning rates for each, and Heusel et al. (2017) showed this can improve convergence behavior over treating the two networks’ optimization symmetrically.
Why have diffusion models largely displaced GANs for state-of-the-art image generation? Diffusion models tend to train more stably (no adversarial min-max game to balance) and cover the data distribution more faithfully, at the cost of much slower sampling (many denoising steps vs. a GAN’s single forward pass) — GANs remain competitive wherever fast single-pass sampling matters more than absolute best-in-class fidelity.

Common Interview Questions

Why is the non-saturating loss used instead of the original minimax generator loss in practice? Because the original “minimize log(1 - D(G(z)))” provides very weak gradients whenever D easily rejects G’s early, obviously-fake outputs, while “maximize log D(G(z))” provides strong gradients in exactly that regime.
What causes mode collapse, and name one mitigation. G converges on a narrow set of outputs that reliably fool the current D rather than covering the full data distribution; minibatch discrimination (letting D compare samples within a batch to spot low diversity) is one standard mitigation, alongside Wasserstein-based losses and unrolled GAN training.
Walk through what happens if you forget detach() on the generator’s output during the discriminator update step. Gradients from the discriminator’s loss would needlessly backpropagate through the generator as well, wasting compute and, depending on the framework and optimizer step ordering, potentially corrupting the generator’s parameters with gradients computed against the wrong objective.
How would you detect mode collapse without eyeballing every generated sample? Track diversity metrics across generated batches (e.g., pairwise sample distances, or the diversity component of a metric like FID) rather than only average sample quality — a generator can have great-looking individual samples and still be collapsed onto a narrow subset of them.
Why do GANs not provide an explicit likelihood, and why does that matter? The generator is a deterministic function of noise with no tractable density function over its output space, unlike a VAE’s explicit (approximate, via ELBO) likelihood — this matters whenever a downstream use case needs a calibrated probability or an anomaly score derived from likelihood, not just realistic samples.

Example

A GAN trained on celebrity face photos learns to generate entirely new, realistic-looking faces of people who don’t exist. Early in training, the generator’s output looks like colored noise, and the discriminator trivially tells it apart from real photos (D’s accuracy near 100%). As training progresses, the generator starts producing blurry face-like blobs, then plausible facial structure, then fine details like consistent lighting and hair texture — each improvement driven by the discriminator continuing to find and exploit whatever gives away the fakes, forcing the generator to fix exactly that flaw next. A production use of this same mechanism is synthetic data augmentation: a GAN trained on a rare defect class in manufacturing images can generate additional realistic-looking defect examples to balance an otherwise heavily imbalanced training set for a downstream classifier.

Referenced by

Dig deeper

Spot an error in this note?