Welcome to today's deep dive. So, um, you've
almost certainly seen those insanely realistic
AI images popping up lately, right? Yeah, like
the hyper realistic deep phase or, you know,
those totally surreal digital paintings. Exactly.
Like, uh, I don't know, a hyper detailed photograph
of a cat wearing a spacesuit on Mars. Right.
Yeah. Which is always fun. It is. But, um, Have
you ever actually stopped to wonder what is literally
happening under the hood to create those? Like,
how does a computer actually do that? Yeah, it
feels like a magic trick when you just look at
the end result. It totally does. So today, our
mission is to sort of strip away all that intimidating
math from the source material. We're looking
at a pretty dense breakdown of diffusion models,
and we want to extract the actual mechanics of
how this works. Right, because you want to walk
away actually understanding how we turn pure,
meaning - a static into, you know, a masterpiece.
Yeah. And to set the stage here, we need to look
at the big picture. What is the core goal of
a diffusion model? Well, broadly speaking, the
goal is to learn a process that can generate
brand new data that looks like it belongs to
an original data set. So like a data set of human
faces or cat photos. OK, so it wants to make
a new face that fits in with the old faces. Exactly.
But the secret to how it does this. and this
is kind of the ironic part, relies entirely on
the concept of destruction. So we're building
by destroying. Yeah, I mean, it is fundamentally
a problem of thermodynamics applied to data.
Which sounds intense. Let's break that down.
The foundation here is a two -part mechanism,
right? You've got the forward diffusion process
and then the reverse sampling process. Right,
those are the two halves of the whole system.
For the forward process, I was thinking of this
analogy. Imagine you take a pristine photograph
and you just slowly start adding TV static to
it. Frame by frame. Yeah, frame by frame. Until
eventually it's just pure meaningless noise,
just a screen of gray fuzz. That's forward diffusion,
right? That is exactly it. You start with your
clean image, which we call X0, and over a sequence
of discrete time steps, you add a tiny scaled
amount of Gaussian noise. And it's a Markov chain,
right. So step five only depends on step four.
Right. You aren't looking at the whole history
of the image. You're just iteratively degrading
it until you hit total static. OK. But why does
that matter? Why are we actively ruining good
pictures? Well, the source text brings up this
connection to non -equilibrium thermodynamics,
which was introduced to this field back in 2015.
OK, 2015. Yeah. Imagine your entire data set
of millions of images is like a tightly clustered
cloud of particles. Like floating in a highly
dimensional space. Exactly. That cloud represents
highly ordered information. By adding that Gaussian
noise over time, you are forcing those particles
to diffuse out. Oh, I see. Like gas filling a
room. Right. You're flattening that complex,
jagged data distribution into a smooth, standard,
normal distribution. It's conceptually identical
to the Maxwell -Boltzmann distribution of particles
settling to a potential well. Okay, I get the
physics analogy. But I mean, destroying an image
is easy. Anybody can mess up a picture. How on
earth does the model undestroy it? Yeah, that's
the big question. And it's where the neural network
actually comes in. Because if it's just trained
to reverse the noise of a specific data set,
isn't it just like memorizing the training data?
Like, how is it creating a novel image and not
just regurgitating a specific cat it already
saw? That's a super common misconception. But
the brilliant twist here is what the model is
actually learning to approximate. It's not memorizing
specific paths back to specific images. OK, what
is it doing then? By forcing the neural network
to watch this destruction happen step by step,
the model is trained to reverse the mathematical
process. It learns the continuous vector field
of the probability distribution. So it's charting
a new course every time. Yeah, exactly. When
you start with pure noise to generate a new image,
you're dropping a pin in a totally random spot
in that static. As the model denoises, it charts
a brand new trajectory through the data space,
steering toward areas of high data density. Oh,
wow. So it's synthesizing features. It learned
like textures, edges, colors into a configuration
that literally didn't exist before. Right. It
obeys the statistical rules of the data set,
but it builds something entirely new. OK. So
it's learning the rules of the terrain, not just
memorizing the map. That makes sense. But how
does the computer actually execute that reverse
trajectory? Well, that brings us to the DDPM.
The denoising diffusion probabilistic model.
From around 2020, right? Yeah, that was a massive
conceptual shift. Because initially, you might
think the neural network is looking at the static
and trying to guess the final pristine picture.
Right, trying to see the cat and the noise. But
it's not. The AI isn't predicting the original
image. It's only predicting the noise. Wait,
instead of trying to predict what the final picture
should look like, the AI is just predicting the
noise itself. Yes. The exact noise that was added
at that specific step. It's almost like, OK,
imagine a sculptor looking at a block of marble.
And instead of trying to see the statue inside,
they just expertly identify the dust and sweep
it away, like one layer at a time. That is a
perfect analogy. The neural network, which is
often called the backbone, is looking at step
T and predicting epsilon, which is the specific
Gaussian noise added at that exact microsecond.
So you just subtract that predicted noise and
you inch one step closer to the clean image.
Exactly. You predict the noise, subtract a fraction
of it, and step backward. And this bridges perfectly
into another mind -bending concept from the source
material, score -based generative models. Oh,
yeah, the noise conditional score networks. For
a while, the research community treated DDPMs
and score -based models as like two completely
different tracks, right? Yeah, they did. Because
DDPMs think about removing discrete steps of
noise, while score -based models use continuous
time and something called Langevin dynamics.
Langevin dynamics. OK, how does that work? They
want to find the score function. Basically, they
look at a noisy pixel and ask, in which direction
should I move this pixel to make it look more
like a real image? OK, so imagine the data space
is like a topographical map. The valleys are
just pure noise, and the mountain peaks are valid,
coherent images. So the score function is basically
an arrow pointing directly uphill. Exactly. It's
the steepest path toward a coherent image. Langevin
Dynamics pulls the random noise up that gradient
toward the highly structured data. And the crazy
part, the real mathematical breakthrough here,
was proving that predicting the noise in DDPM
and estimating that uphill gradient in the score
-based model, they're mathematically equivalent.
Yes. They are the exact same thing from different
angles. Predicting the noise to subtract, or
calculating the vector to nudge the pixels uphill,
it's two sides of the same coin. That is wild.
OK, so mathematically, we can reverse entropy.
We can turn static into art. We can. But practically,
the original DDPM takes around 1 ,000 steps to
turn noise into an image. Doing that pixel by
pixel for a high -res image sounds like it would
melt my laptop. Oh, absolutely. It was computationally
devastating. So how do we make this practical
for everyday use? How do I generate an image
in five seconds on my phone? Well, you have to
bypass the strict rules of that original Markov
chain. And the solution to that was DDIMM, the
denoising diffusion implicit model. Okay, DDIMM.
How does that speed things up? In the original
process, it's stochastic, meaning it relies on
a random walk. You can't skip steps without breaking
the math. But DDIMM sacrifices a tiny bit of
that randomness to make the reverse process deterministic.
Oh, so if it's deterministic, the path is fixed.
Exactly. And because it's a fixed trajectory,
you don't have to evaluate every single intermediate
step. You can take massive mathematical leaps.
So you can skip from step 1 ,000 down to 900.
Right, and generate a totally coherent image
in just 20 steps. And it looks just as good as
one that took 1 ,000. Wow. OK, so we skipped
steps. But even with 20 steps, a 4K canvas has
millions of pixels. Doing heavy math on all of
them is still a huge bottleneck. Which is where
latent diffusion comes in. This is the trick
that completely changed the industry. Right.
This is what stable diffusion uses. Yeah. Operating
in pixel space is computationally wasteful. A
big blue sky doesn't need millions of parameters
to be understood as the blue sky. So instead
of doing the math on a massive image, you shrink
the canvas. You use an encoder to squish the
image into a tiny lower dimensional space. The
latent space, exactly. It captures the semantic
essence of the image without keeping track of
every single pixel. You do all the diffusion
adding noise, predicting it in this tiny latent
space. And then when you're done, you just decode
it back to full size. Exactly. You're doing the
heavy lifting on a tiny blueprint instead of
the full construction site. That is so smart.
OK, so we've made it fast. We shrunk it down.
But here is my biggest question. If I type in
a prompt for a black cat with red eyes, how do
I steer this noise remover? Ah, the steering
wheel. Yeah, because if it's just removing static
to find any valid image, how does it know I want
on a black cat and not like a picture of a toaster.
That relies on something called classifier -free
guidance, or CFG. CFG. I've seen that slider
in AI art programs. Right. So to understand CFG,
think about how the model is trained. It's basically
a noisy channel concept. The model takes a text
prompt, translates it into a mathematical vector,
and feeds it in alongside the noise. OK. So it
learns to associate the word cat with the visual
pixels of a cat. Yes. But during training, we
randomly drop the text prompt about 10 % of the
time. Wait, really? Why would you hide the text
from it? Because you need the model to learn
two different ways of denoising. It learned conditional
denoising using the text prompt and unconditional
denoising, where it just tries to find any image
without a prompt. Oh, I see. Yeah, so imagine
the AI has two compasses. The unconditional compass
just points vaguely toward a coherent image,
but the conditional compass points specifically
toward a black cat with red eyes. So by comparing
the two, you can find the exact difference between
them. Exactly. You mathematically subtract the
unconditional prediction from the conditional
one. That extracts a powerful guidance vector.
And then you can just multiply that vector to
push it harder. Right, you scale it up, you aggressively
nudge the denoising process toward your specific
prompt. And you can even use negative prompting,
right, like tell it to find the vector for blurry
and steer in the exact opposite direction. Yes,
exactly. It gives you incredible control over
the generation. OK, so we know how to reverse
noise, skip steps, shrink the canvas and steer
it with text. But what is the actual engine doing
the heavy lifting? the neural network architecture
itself. Historically, the backbone was almost
always a UNET. UNETs. They're super good at processing
images, right? They are. They down sample an
image to learn the broad structure and then upsample
it to retain fine details. They have a strong
bias for spatial relationships. They assume neighboring
pixels matter to each other. Which makes sense
for pictures. But recently we're seeing a massive
shift away from units toward transformers, specifically
D -DI -T diffusion transformers. Yeah, the paradigm
is definitely shifting. Which is crazy to me
because transformers are what run large language
models like ChatGPT. They process sequences of
words. Why are they suddenly taking over image
generation? Because transformers scale so beautifully.
They don't have that built -in assumption about
local pixels. They use an attention mechanism
that lets every part of the input look at every
other part, no matter how far away it is. So
you just chop the image up into little patches,
treat them like words in a sentence, and feed
them into the transformer. Exactly. And as you
throw more computer to transformer, it just keeps
getting better. UNETS tend to hit a performance
ceiling. That makes total sense. And we're seeing
this play out in the real world right now. Let's
look at some examples from the source text. Sure.
Look at OpenAI. Dell 82 uses a really interesting
unCLIP method converting text encodings directly
into imaging codings. But their new video generator,
Sora, that uses a diffusion transformer. Right,
the DIT. Because video is just a sequence of
patches over time. Exactly. And Stability AI,
who popularized latent diffusion with a UNAB,
they actually moved to a transformer model for
stable diffusion 3. Wow, everyone is pivoting.
But there are other approaches too, right? Like
Google's Imogen. They don't use the latent space
trick at all. No. Google uses cascading diffusion
models. Basically, they start with one model
generating a tiny 64 by 64 pixel image. Just
getting the broad strokes right. Right. Then
a totally separate diffusion model takes that
and upscales it to 256. And then another model
upscales that to 224. Like an assembly line.
Yeah. And then you have meta with transfusion,
which actually combines autoregressive text generation
with denoising diffusion in a single model. Or
their movie gen, which uses flow matching. It's
incredible how adaptable this math is. The text
points out these models aren't just for images.
They're completely modality agnostic. Oh, totally.
You can use it for audio generation. Yeah. Or
human motion synthesis. Wait, motion synthesis?
Yeah, you take a noisy chaotic trajectory of
a 3D skeleton, and the model denoises the temporal
sequence until it collapses into a perfectly
smooth, realistic walking animation. That is
mind -blowing. It really works on anything. It
does. It can even be used for natural language
text generation. Okay, so just to recap the journey
we've been on, we started with this crazy idea
of thermodynamics and entropy, destroying a sandcastle
basically. Right, the forward process. Then we
learned how neural networks predict the noise
to reverse time, like sweeping dust off a marble
statue. We sped it up by skipping steps with
DDIMM, shrank the canvas with latent diffusion,
and learned to steer it using CFG text guidance.
And swapped out the engine for transformers.
Exactly. So the next time you generate an AI
image, you're not just running a filter. You're
literally watching a simulated universe of random
particles collapsing into a state of perfect
guided order. It's a localized reversal of entropy.
But before we wrap up, there is one final totally
mind -bending concept at the end of the source
material that we have to talk about. Oh. Lay
it on me. We talked a lot about how the path
from pure noise to a final image is a squiggly
random walk, right? That eto process. Yeah, that's
why it takes so many steps. Well, researchers
are now looking at something called flow -based
diffusion models and rectified flow. Rectified
flow? What are they rectifying? The trajectory
itself. Rectified flow attempts to learn a process
that makes the path between the noise and the
final image a perfectly straight line. Wait,
a perfectly straight line? Yes. It mathematically
removes the curvature from the probability space.
But hold on. If the mathematical path from static
to the image is perfectly straight, you don't
need to take 20 gentle steps to curve your way
there. Exactly. You could calculate the exact
solution in one single step. Wait, really? One
single step? Just one mathematical calculation.
From pure random noise to a flawless high -resolution
image instantly. Instantly. Wow. And if we completely
eliminate that random walk and replace it with
a single geometric calculation, are we even doing
diffusion anymore? Or have we just discovered
a totally new way to mathematically conjure reality
out of the void? That gives me chills. I mean,
if the compute cost plummets to near zero and
the generation is instantaneous, you wouldn't
just be generating static images, you'd be generating
real -time interactive realities on the fly.
Exactly. Real -time rendering of reality. You
wouldn't need a graphics card to render a video
game level. The AI would just diffuse the world
into existence at 60 frames per second based
purely on where you look. It makes you wonder
if pre -recorded media will even exist in a decade.
Something for you to chew on next time you fire
up an AI generator. Thanks for joining us and
we'll catch you on the next deep dive.