How AI Image Generators Actually Work: Inside the Diffusion Models Behind DALL-E and Stable Diffusion

Type a sentence into Midjourney, DALL-E, or Stable Diffusion and a few seconds later you have an image that never existed before. It feels like magic, but the mechanism underneath is almost the opposite of how most people imagine AI art works. These systems don't paint. They un-destroy.

The Idea Nobody Believed Would Scale

The core technique, called a diffusion model, started as a fairly obscure piece of math. In 2015, Stanford researcher Jascha Sohl-Dickstein and colleagues published a paper borrowing an idea from nonequilibrium thermodynamics: take an image, add a tiny bit of random noise to it, then add a bit more, and a bit more, until after enough steps the original picture has dissolved into pure static. Now ask a neural network to learn how to run that process backward — to look at noisy static and guess what the previous, slightly-less-noisy version looked like.

Do that once and you get a marginally cleaner image. Do it a thousand times, starting from pure random noise, and you get a photograph. Or at least that was the theory. In 2015 the generated samples were blurry and unconvincing, and generative adversarial networks (GANs) — which pit two networks against each other in a forger-versus-detective contest — dominated image generation for years afterward.

The breakthrough came in 2020, when Jonathan Ho, Ajay Jain, and Pieter Abbeel at UC Berkeley published "Denoising Diffusion Probabilistic Models" (DDPM). They showed that with the right training objective — essentially having the network predict the noise itself, rather than predict the clean image directly — and a U-Net architecture to do the predicting, diffusion models could match or beat GANs on image quality. That paper, now the field's most-cited reference point, quietly ended the GAN era.

From Research Curiosity to Household Tool

OpenAI's original DALL-E, released in January 2021, didn't actually use diffusion — it generated images token-by-token the way a language model generates text, treating pixels like words in a sentence. It was impressive but slow and visually inconsistent.

Everything changed in April 2022 with DALL-E 2, which swapped that approach for a diffusion-based pipeline and produced noticeably sharper, more coherent images from the same kind of prompts. Four months later, in August 2022, Stability AI released Stable Diffusion as an open-weight model anyone could download and run — a genuine inflection point, since it meant the technique no longer required a few hundred thousand dollars of cloud compute or a research lab's permission. Google's competing Imagen and Parti systems were published the same year as research models. By the end of 2022, diffusion had become the industry's default architecture for text-to-image generation, a position it still holds.

Why "Latent" Diffusion Made It Practical

Running the noise-adding-and-removing process directly on full-resolution pixels is brutally expensive — a 512x512 image has over 260,000 pixels, each needing dozens of denoising passes. Stable Diffusion's key engineering trick was to not do that at all. Instead, a separate component called a variational autoencoder first compresses the image into a much smaller "latent" representation — a compact grid of numbers capturing the image's essential structure rather than every pixel. All the actual diffusion happens in that cramped latent space, and only at the very end does a decoder blow the result back up into a full image. This is why Stable Diffusion could run on a single consumer GPU while earlier pixel-space diffusion models needed data-center hardware.

Where the Words Come In

None of this explains how a diffusion model knows to draw "an astronaut riding a horse" rather than random noise-shaped blobs. That part comes from a separate system, typically a CLIP-style text encoder, trained beforehand on hundreds of millions of image-caption pairs scraped from the web. CLIP learns to map text and images into the same mathematical space, so the phrase "astronaut" and pictures of astronauts end up near each other in that space. During generation, the text prompt's encoded representation is fed into the denoising network at every single step, nudging each round of "what did this noise used to look like" toward astronaut-shaped and horse-shaped answers instead of generic ones. Strip away the conditioning and a diffusion model will still happily generate images — it just won't generate the ones you asked for.

The Trade-off Nobody Advertises

Diffusion's iterative nature is also its biggest practical weakness: generating one image can require anywhere from 20 to 1,000 forward passes through the denoising network, which is why early systems took tens of seconds per image. Most of the speed improvements users have noticed since 2022 — tools that now generate images in one or two seconds — come from techniques like distillation and better noise schedulers that approximate the same multi-step process in far fewer steps, not from abandoning diffusion itself. It's a reminder that most of the visible progress in consumer AI tools is less about new ideas and more about relentless engineering around one idea that happened to work: start with noise, and very patiently, guess your way back to a picture.