Generative adversarial networks (GAN) explained

Generator and discriminator: the two-player game

A GAN has two neural networks with opposing goals:

• Generator G — maps random noise z (usually sampled from a standard normal distribution) into synthetic data G(z). For images, z might be 100–512 dimensions; the generator upsamples through transposed convolutions or residual blocks until it outputs an image tensor.
• Discriminator D — a binary classifier that receives either a real sample from the training set or a fake sample from the generator. It outputs a probability that the input is real. The discriminator is typically a CNN for images or an MLP for tabular data.

Training alternates (or runs jointly) between updating D to classify better and updating G to produce outputs that D misclassifies as real. Neither network has access to labeled fake/real pairs beyond the implicit label of which pipeline produced the sample — this is unsupervised generation from unlabeled data.

The original formulation by Goodfellow et al. (2014) frames this as a minimax game. The generator minimizes log(1 − D(G(z))) while the discriminator maximizes log D(x) for real data and log(1 − D(G(z))) for fakes. In practice, the generator loss is often rewritten as maximizing log D(G(z)) because it provides stronger gradients early in training when D easily rejects fakes.

Training loop and stability tricks

A typical GAN training epoch looks like this:

1. Sample a mini-batch of real data x.
2. Sample noise z and generate fakes G(z).
3. Update D to maximize discrimination (real labeled 1, fake labeled 0).
4. Update G to fool D (maximize D(G(z)) or minimize non-saturating loss).

Balance matters. If D becomes too strong too fast, gradients to G vanish — the generator receives no useful signal. If G improves too quickly, D collapses to random guessing and training stalls. Common stabilizers include:

• Label smoothing — train D on soft labels (0.9 for real, 0.1 for fake) instead of hard 0/1 to prevent overconfidence.
• One-sided label smoothing — only smooth real labels; keep fake at 0.
• Two-timescale update rule (TTUR) — use a lower learning rate for D than G, or train D multiple steps per G step.
• Gradient penalty and spectral normalization — constrain D's Lipschitz constant so it cannot make arbitrarily steep decisions (WGAN-GP, SNGAN).
• Historical averaging and experience replay — feed D a buffer of past generator outputs so it does not only see the latest G.

Unlike supervised models where you watch validation loss decrease, GAN training requires monitoring sample quality, discriminator accuracy, and distributional metrics — loss curves alone are misleading.

Mode collapse and other failure modes

Mode collapse is the most infamous GAN failure: the generator discovers a small set of outputs that consistently fool D and stops exploring the full data distribution. You might train on thousands of face photos but get the same face repeated with minor variations.

Mitigations include:

• Minibatch discrimination — D sees statistics across the batch, punishing identical outputs.
• Unrolled GANs — G optimizes against a lookahead version of D.
• Feature matching — G matches intermediate discriminator activations on real data, not just the final real/fake score.
• WGAN and WGAN-GP — replace JS divergence with Wasserstein distance and enforce Lipschitz constraints via gradient penalty.
• Data augmentation on real samples — prevents D from memorizing a narrow real manifold; see data augmentation practices.

Other issues: checkerboard artifacts from transposed convolutions with mismatched stride/kernel (fixed in DCGAN guidelines), training oscillation where D and G take turns dominating, and memorization where G copies training images verbatim — a privacy and evaluation concern.

Architecture milestones: DCGAN, conditional GAN, StyleGAN

DCGAN (2015)

Deep Convolutional GAN established practical rules: use strided convolutions instead of pooling, batch normalization in G (not always in D), ReLU in G and LeakyReLU in D, and tanh output scaled to [−1, 1] matching normalized inputs. DCGAN made 64×64 image generation reproducible and remains a teaching baseline.

Conditional GAN (cGAN)

When you need control — generate a digit given class label, or a scene given a text embedding — both G and D receive auxiliary conditioning information c. Class-conditional GANs concatenate c to z or use projection discriminators. Pix2Pix and CycleGAN extended conditioning to image-to-image translation (sketch to photo, horse to zebra) using paired or unpaired datasets.

Progressive GAN and StyleGAN

Progressive growing trains from low resolution (4×4) to high (1024×1024), stabilizing early layers before adding detail. StyleGAN (NVIDIA, 2019) introduced a style-based generator: latent codes map through adaptive instance normalization (AdaIN) layers at multiple scales, separating coarse attributes (pose, face shape) from fine detail (freckles, hair strands). StyleGAN2/3 refined artifacts and aliasing. The disentangled W latent space enables semantic editing — interpolate between faces, adjust smile or age by moving along discovered directions.

BigGAN and self-attention

Large-batch training with class-conditional batch normalization and self-attention layers (SAGAN) improved ImageNet-scale generation. Transformer blocks later appeared in generators (TransGAN), though diffusion largely superseded them for open-ended text-to-image.

Applications beyond pretty pictures

• Synthetic data augmentation — generate rare-class images for imbalanced detection tasks; must verify downstream metric gains, not just visual plausibility.
• Super-resolution and restoration — SRGAN, ESRGAN upscale low-res inputs with perceptual losses; still used in gaming and video pipelines where latency matters.
• Anomaly detection — train GAN only on normal data; high reconstruction or discriminator error flags outliers (see anomaly detection).
• Domain adaptation — CycleGAN maps between domains without paired examples (MRI modalities, seasonal scenery).
• Privacy-preserving datasets — release synthetic patient or customer records that preserve statistical properties without exposing individuals — requires rigorous membership-inference testing.
• Face de-identification and media forensics — both generation and detection of GAN fingerprints remain active research areas.

Evaluating generative quality

There is no single loss number that means "good." Practitioners combine:

• Inception Score (IS) — measures classifiability and diversity via a pretrained Inception network. Cheap but biased toward ImageNet-like images.
• Fréchet Inception Distance (FID) — compares mean and covariance of Inception features between real and generated sets. Lower is better; sensitive to sample count (use 10k+ generated images for stable estimates).
• Precision and recall for distributions — precision: are fakes realistic? recall: does G cover all modes?
• Human evaluation — still gold standard for creative applications; use pairwise A/B with sufficient raters.
• Downstream task transfer — train a classifier on synthetic data, test on real; measures utility, not just realism.

Log D accuracy hovering around 50–60% often indicates healthy balance; 95% D accuracy means G is not learning; 50% with terrible FID means G found adversarial trivial fakes.

GAN vs diffusion vs VAE: decision table

For most new text-to-image products in 2026, diffusion or flow-matching models are the default. GANs still win when you need 30+ FPS face synthesis, lightweight on-device avatars, or a well-understood latent space for a narrow domain you control end to end.

Common mistakes

• Judging training by G or D loss alone — losses can look fine while outputs are garbage; always inspect samples and compute FID.
• Insufficient discriminator capacity — a weak D lets G cheat with noise patterns instead of learning structure.
• Batch size too small — batch norm statistics and minibatch discrimination suffer; BigGAN-scale results need large batches or good approximations.
• Mismatched preprocessing — real images normalized to [0,1] but G outputs tanh [−1,1] causes color shifts and training instability.
• Skipping architectural guidelines — arbitrary layer choices resurrect checkerboard artifacts and vanishing gradients.
• Deploying without memorization checks — nearest-neighbor search in training set catches copied identities in face GANs.
• Assuming GAN synthetic data always helps — low-quality fakes can hurt downstream models; validate on held-out real test sets.

Production checklist

• Define success metrics upfront — FID threshold, human eval protocol, or downstream task lift; not "looks cool in TensorBoard."
• Freeze preprocessing pipeline — same resize, crop, and normalization in train, eval, and serve.
• Checkpoint on sample quality — save G when FID improves, not only on epoch count; exponential moving average (EMA) of G weights often yields sharper outputs.
• Version datasets and architectures — GAN outputs shift with any data or hyperparameter change; track in model registry per MLOps practice.
• Test inference latency and memory — profile batch-1 GPU/CPU forward pass; GANs are fast but large StyleGAN variants still need VRAM planning.
• Content safety and bias audit — training data demographics propagate into generated faces and scenes; add filters and provenance metadata.
• Monitor for drift in production — if G conditions on live inputs, revalidate when input distribution shifts.
• Document limitations — disclose synthetic origin for generated media; GAN fingerprints are increasingly detectable.

Key takeaways

• GANs train a generator and discriminator in opposition — unsupervised learning of data distributions without explicit density modeling.
• Stability is the hard part — balance D/G updates, use modern architectures (DCGAN rules, spectral norm, WGAN-GP), and watch for mode collapse.
• StyleGAN remains the reference for controllable image GANs — style layers and W latent space enable semantic editing at high resolution.
• Evaluate with FID, precision/recall, and human judgment — not raw adversarial loss.
• Choose GAN when speed and latent control matter; choose diffusion when maximum quality and prompt flexibility dominate.

Related reading

• Deep learning explained — neural network training, backpropagation, and optimization basics
• Diffusion models explained — the generative approach that largely superseded GANs for open-ended image synthesis
• Computer vision fundamentals — CNN backbones used in discriminators and perceptual losses
• Transfer learning explained — pretrained Inception and ResNet features for FID and perceptual loss