You know, we hear about artificial intelligence
like constantly these days. Oh, absolutely. It's
everywhere. Right. Spreading our emails, diagnosing
diseases, driving our cars. And the way it's
usually presented to you and me is like this
sleek, totally impenetrable black box. Yeah,
just millions of data points go in and then magic.
Exactly. Some mysterious computational magic
happens and this perfectly formed answer just
pops out. Yeah. But if you actually peek inside
that box, what you find isn't, you know, magic
at all. No, far from it. I mean, when you look
under the hood of a large language model or an
image generator, you don't find the synthetic
brain spontaneously thinking thoughts. You find
a very specific, incredibly rigorous mathematical
engine, and it's basically running a very old
optimization problem. And today we are going
to crack that black box wide open. Welcome to
today's Deep Dive. We've got a single, just remarkably
comprehensive source for you today. It's the
Wikipedia article on backpropagation. That's
the one. And our mission today is to understand
the exact mathematical engine that allows AI
to actually learn from its mistakes. Because
whenever you hear about an AI learning to do
something, it almost always boils down to this
one specific algorithm. It really does. It's
the core of it all. OK, let's unpack this. Before
we can really graph how backpropagation works,
we kind of need to understand the problem it
was actually invented to solve, right? Yeah,
you need the why before the how. Because in machine
learning, specifically in this framework we call
supervised learning, an AI doesn't start out
with any inherent knowledge. It's just a blank
slate. Completely ignorant. Its internal connections,
these numerical values that we call weights,
they are essentially just set at random initially.
So if you ask a brand new untrained network to
identify a picture of a cat, it is literally
just guessing blindly. I was actually thinking
about an analogy for this while I was reading
to the source material. Imagine you are standing
in a large room, right? And you're trying to
hit a dart board. OK, I like where this is going.
But you are completely blindfolded. You throw
your first dart and you have absolutely no idea
where it landed. I mean, it probably hit the
ceiling or like a. Potted plant almost certainly
a potted plant right now If nobody tells you
anything your next throw is just gonna be another
totally random guess you're never gonna learn
Exactly, but in supervised learning you have
a guide you have your training data Okay, so
that data acts as the person standing next to
you watching where your dart actually lands So
they say okay that throw was you know three feet
too high and two feet too far to the left They
are actively quantifying your error. Yes They
give you the exact measurements of how bad you
missed. So for your next throw, you adjust. You
change your angle. Maybe you change your force
just trying to correct for that specific mistake.
Yeah. And you keep throwing and they keep giving
you precise feedback. Until eventually you're
hitting the bullseye. Right. Now, in the mathematical
realm of a neural network, that distance between
your dart and the bullseye is measured by something
called the loss function. Or sometimes the cost
function, right? I saw both terms in the text.
Yeah, they're generally interchangeable. The
source actually mentions the squared error, which
is used for regression problems. And the neat
thing about squaring the distance of your miss
is that the math heavily penalizes a dart that
hits the ceiling, but only slightly penalizes
a dart that lands, you know, just outside the
bull's eye. That makes sense. It really wants
to fix the huge mistakes. So if we visualize
that relationship between the network's prediction
and the actual error, it forms a shape like a
giant bowl. Or, mathematically speaking, a parabola.
A big mathematical bowl, exactly. And the goal
of the AI, like its only fundamental goal, is
to find the absolute lowest point of that bowl.
The minimum. Because at that lowest point, down
at the very bottom, the error is at its smallest.
Your dart is squarely in the bullseye. Okay,
so how does it get there? Well, if we connect
this to the bigger picture, finding that minimum
point is done through a process called gradient
descent. Gradient descent. Right. Since the AI
is blindfolded, remember, it can't just look
at the bowl and jump straight to the bottom.
All it can do is feel the slope of the bowl right
beneath its own feet. Oh, I see. So it calculates
the steepest downward direction from its current
position, takes a tiny mathematical step in that
direction, and then feels the slope again. Yeah.
Back propagation is just the specific method
used to calculate that exact slope. the steepest
descent direction of the loss function relative
to the current weights. You got it. So it takes
a step. recalculates the error, adjusts the weights,
takes another step. I mean, that makes logical
sense for our single dart thrower. But reading
through this source, I really started thinking
about a sheer scale of modern AI. It's massive.
Yeah. Yeah. A neural network isn't just one dart
thrower. It has multiple layers. You've got an
input layer taking in the data, an output layer
giving the answer, and potentially hundreds of
these hidden layers in between. Sometimes thousands
of layers. And they contain millions or even
billions of individual weights. connecting all
of them together. And this is where the logistics
just seem impossible to me. If you have to calculate
the exact adjustment for every single tiny weight
across billions of connections every single time
the network makes a guess. I mean, how does calculating
the change for every possible path not just instantly
overwhelm the computer? It sounds like an infinite
math problem. Exactly. It just sounds completely
impossible. And that is the exact logistical
nightmare that stumped researchers for years.
Because if you try to calculate the error by
starting at the beginning of the network and
moving forward, Calculating how every tiny tweak
to a weight might eventually affect the final
output, the math becomes astronomically complex.
Like tracking a billion different butterfly effects.
Exactly. You end up calculating the same intermediate
steps over and over and over again. It requires
computing power that didn't exist back then and,
frankly, still barely exists today. Let's visualize
that bottleneck because I think I have a way
to picture this. It's like a massive factory
assembly line. OK, I'm with you. So a car rolls
off the end of the line and the steering wheel
is installed upside down. The forward approach
we just talked about would be like standing at
the very beginning of the assembly line looking
at the raw steel and the very first worker. Right.
And trying to mathematically simulate every possible
action every single worker could take just to
figure out how it might eventually result in
an upside down steering wheel at the end. It's
totally absurd. It's a logistical nightmare.
It is. And the core brilliance of the backpropagation
algorithm is actually right there in the name.
You don't go forward. You propagate the error
backward. Backward. You compute the gradient
one layer at a time, but you iterate in reverse
from the final output layer back to the first
input. Piss them back to the factory. You start
with the flawed car. You look at the final station.
You ask, did the last person put the steering
wheel on upside down? And if they say no, it
arrived to them that way. Right. You just take
one step backward. Did the person before them
do it? No. So you trace that specific actual
error backwards, step by step, following the
exact chain of causality. Until you find the
specific worker who made the mistake, and you
tell them to adjust their process. That is so
incredibly elegant. It really highlights the
raw efficiency of the algorithm. You aren't evaluating
every possible thing that could go wrong in the
factory. You are only evaluating the actual error
that occurred. And mathematically, the source
explains this as an efficient application of
the chain rule from calculus. The chain rule?
I definitely remember that term from high school
calculus, but usually in a very, like, abstract
way. Well, in calculus, the chain rule is just
a formula for calculating the derivative of composite
functions. Composite functions. Think of them
like Russian nesting dolls of math. Functions
hidden entirely inside other functions. Oh, okay.
I like that. In our neural network, the output
of one layer simply becomes the input for the
next layer. They are nested. So if you want to
know how a tiny change to the smallest doll in
the center affects the largest doll on the outside,
the chain rule lets you just multiply the rates
of change at each layer to find the total effect.
But the key to making this actually run on a
physical computer without, you know, melting
the processor comes down to linear algebra. Exactly.
This is the realm of matrices and vectors. If
you try to calculate the error forwards, like
we were saying earlier, you have to multiply
a matrix by a matrix. And a matrix is essentially
just a massive spreadsheet of numbers, right?
Yep. And multiplying two giant spreadsheets together
is exactly like your factory simulation. You
are trying to track every possible path of change
through the entire network simultaneously. It's
highly, highly inefficient. But with backpropagation,
by starting at the end with a known error, we
just bypass all those redundant calculations.
Right. Instead of a matrix multiplied by a matrix,
each step backward is simply multiplying a vector
by a matrix. And a vector is simpler. Much simpler.
A vector is just a single column of numbers.
In this case, it represents the specific error,
which the math denotes as a lowercase delta.
Multiplying a single column by a spreadsheet
is infinitely faster for a computer process.
Because it just avoids the duplicate calculations.
Exactly. It skips computing all those intermediate
values you don't actually need. And that one
single mathematical trick is what makes training
deep neural networks physically possible. Which
is wild. This elegant trick is the absolute bedrock
of the entire modern AI boom. I mean, without
it... And none of the AI we use today even exists.
So you would naturally assume this was invented
recently, right? For sure. Like maybe 10 or 15
years ago in some sleek Silicon Valley tech incubator.
That would make sense. But the history of this
algorithm is actually this tangled, bizarre web
that stretches back centuries. Here's where it
gets really interesting. Because the foundational
math behind this, that chain rule we just broke
down, it was first written down by Gottfried
Wilhelm Leibniz. Yes. the German mathematician
and philosopher. In the year 1676. I mean, we
are literally using 17th century math to train
modern AI chatbots. It is a profound reminder
that in mathematics, foundational truths never
expire. You know, the application of those ideas
to what we now call machine learning, that took
a very winding path. To say the least. The precursors
to modern backpropagation actually appeared in
optimal control theory back in the 1950s and
60s. Control theory, that's the math used for
engineering dynamic systems, right? Like flight
paths and industrial processes. Exactly. Researchers
like Henry J. Kelly, Arthur E. Bryson, and Lev
Pontryagin, they were trying to figure out how
to optimize multi -stage processes. Imagine trying
to optimize a multi -stage rocket flight. You
have thrust, wind resistance, fuel burn, all
interacting continuously. If you want to adjust
the starting launch angle to perfectly hit a
target in orbit, they used something called the
adjoint state method. Adjoint state method? Yeah.
It basically calculates how the final miss relates
to the initial parameters by working backward
from the target. Wait, that sounds exactly like
backpropagation. It is conceptually identical
to it. It's just applied to the continuous physics
of rockets instead of artificial neural networks.
So they had the right tools, but they were just
building a completely different house. Exactly.
But I have to say, The absolute most surprising
detail in the entire source involves a researcher
named Paul Werbos, because in 1971, Werbos developed
the backpropagation algorithm as we generally
understand it today for his PhD thesis. Yes,
he did. But he didn't do it to teach computers
how to recognize images or play chess. He developed
it to mathematicize Sigmund Freud's concept of
the flow of psychic energy. I know. It remains
one of the most unexpected crossovers in the
history of science. Werbos was essentially attempting
to map psychological theories of human behavior.
The ego, the id. Right. And the way traumatic
errors supposedly propagate backward through
the human subconscious, he wanted to map all
of that into a rigorous mathematical model. He
literally translated Freudian psychoanalysis
into calculus, which is just mind blowing. And
the frustrating part is, Werbos couldn't even
get his work published until 1981 because the
concept was viewed as to unorthodox. The utility
for computer science just wasn't recognized yet.
And that delay is a recurring theme here. The
concept just kept being discovered and then ignored.
Sepo, Linema published a concept called the reverse
mode of automatic differentiation in 1970. Which
is mathematically identical to modern back -proc.
Exactly. But it wasn't until the mid -1980s that
the dam finally broke. And that culminated in
a very famous 1986 paper in the journal Nature.
published by David Rumelhart, Jeffrey Hinton,
who, by the way, recently won the 2024 Nobel
Prize in Physics for his AI contribution. Oh,
yeah. Huge figure in the field. And Ronald Williams.
Yeah. They demonstrated experimentally that backpropagation
could train neural networks to learn internal
representations of data. They popularized the
algorithm and forced the broader scientific community
to finally pay attention. They really put it
on the map. I do have to pause here, though.
If Lin and Amah had the exact math in 1970. and
were both had it in 1971. Why did it take until
1986 for anyone in the actual neural network
field to adopt this magic shortcut? What's fascinating
here is that the delay stemmed from a fundamental
biological misunderstanding. Yeah. Back in those
days, artificial neural network design was heavily,
heavily influenced by physiologists who were
studying actual human brains. And the prevailing
biological belief at the time was that neurons
only fired in discrete binary signals, a zero
or a one, off or on. Oh, like a light switch.
Precisely. But returning to our concept of gradient
descent from earlier, backpropagation requires
calculating a slope. Right. Feeling your way
down the bowl. Exactly. And to do that, it requires
continuous, differentiable functions. You cannot
calculate the mathematical slope of a discrete
light switch. Because it's just on or off. Right.
On a graph, a step function is either a perfectly
flat horizontal line, or it's a sheer vertical
cliff. And the slope of a vertical cliff is mathematically
undefined, while the slope of a flat line is
just zero. So there's no slope to feel. Right.
Because early researchers believed artificial
neurons had to perfectly mimic this discrete
biological firing, they assumed backpropagation
was useless for neural networks. There was no
gradient slope for the blindfolded AI to feel.
Oh wow. They were trying so hard to copy human
biology that they just completely blinded themselves
to the mathematical solution. Exactly. To make
the math work, they had to abandon strict biological
realism. They needed activation functions that
were smooth, continuous curves where a derivative
could actually be evaluated. And once the field
finally accepted that and the 1986 paper proved
it worked in practice, back propagation just
completely took over. It really did. The source
lists this rapid -fire string of absolute triumphs
that followed. In 1987, a system called NetTalk
learned to convert English text into pronunciation,
literally reading out loud. Huge breakthrough.
Then in 1989, a project named Alvion used back
propagation to learn how to drive a vehicle autonomously.
That same year, Jan Lacoon published Lynette,
which could recognize handwritten zip codes on
envelopes. And by 1992, a system called TD Gammon
achieved human -level play in backgammon. It
was a golden age of rapid advancement. But, you
know, the source is very clear that backpropagation
isn't a flawless silver bullet. There are significant
limitations. And the most famous of which brings
us back to that bowl shape we talked about, the
error landscape. Right, the perfect parabola
where the bottom is the bullseye. Well, the problem
is, real world data is rarely a perfect, smooth
bowl. The error landscape of a highly complex
neural network looks more like a rugged, chaotic
mountain range. It's filled with peaks, valleys,
and flat plateaus. Oh, I see. So the danger of
gradient descent is that the blindfolded AI might
feel its way down into a small valley, hit the
bottom, and stop. Because all the slopes around
it go up, it assumes it has solved the problem.
Even though it's not the absolute bottom. It
gets stuck in what they call a local minimum,
right, like a shallow valley halfway up the mountain,
instead of finding the global minimum, which
is the true bottom of the deepest valley. Exactly.
And it can also get completely stalled on flat
plateaus where the slope is practically zero,
which provides no directional guidance at all.
Just wandering around in the dark. Yeah. Though
it is worth noting that Yann LeCun actually argues
that in highly complex, high -dimensional practical
problems, getting stuck in a local minimum isn't
really a catastrophic drawback. The good enough
valley is usually perfectly fine for the AI to
function effectively in the real world. But wait,
knowing the slope of that mountain implies the
mountain is perfectly smooth, right? If we use
activation functions that create sharp, jagged
cliffs in the mathematics, doesn't the entire
back propagation formula just fall apart? Theoretically,
yes, absolutely. Backpropagation strictly requires
that the derivatives of the activation functions
inside the neurons be known and continuous. Which
leads to a truly fascinating contradiction in
the text. Because we just established that researchers
in the 70s were stalled because backprop needs
smooth, differentiable math. It needs curves.
Right. But the text points out that in the modern
era, one of the most popular activation functions
used in massive networks, like the famous AlexNet,
is something called ReLU, or E -L -U. And ReLU
is completely non -differentiable at zero. Yeah,
it literally has a sharp jagged point on the
graph. And why does a sharp point break calculus?
Because a derivative is essentially finding the
tangent line to a curve. On a smooth curve, there
is only one way to balance a tangent line. But
on a sharp pointy corner, you can balance a line
in infinite ways, meaning there is no single
defined slope. The blindfolded AI suddenly has
no idea which way is down. Exactly. ReLU completely
breaks the foundational rule of backpropagation.
And yet, everyone uses it, and it works incredibly
well. It is one of the great ironies of modern
deep learning. Theoretically, ReLU should cause
the algorithm to crash the exact moment the network's
value hits exactly zero. But in practice, with
millions of data points and floating -point math
running on physical computers, hitting absolute
perfect mathematical zero is just so incredibly
rare that the system just hums along. So they
just ignore the math rule? Yeah. It's a case
where practical engineering simply ignores the
theoretical math constraint because the real
-world results are just too good to pass up.
Wow. So researchers had this incredibly powerful
algorithm, and they finally figured out the practical
engineering to make it work. But the dominance
of backprop wasn't just a straight line from
1986 to today, was it? No, not at all. because
the source mentions the algorithm actually fell
completely out of favor in the 2000s during a
period known as the AI winter. Yeah, they hit
a really hard wall with computational power.
The networks that researchers wanted to build
are just becoming too deep and too complex. Standard
computer processors, CPUs, they just couldn't
crunch those matrix multiplications fast enough.
Even with the backpropagation shortcut. Even
with the shortcut. So researchers have the perfect
algorithm, but we're essentially trying to run
it on Like a pocket calculator? Essentially,
yes. How did they finally get the horsepower
they needed to thaw out the AI winter? The breakthrough
actually came from a completely unrelated industry.
Video games. Really? Video games. Yep. In the
2010s, researchers realized that GPUs, graphics
processing units, were structurally very different
from standard CPUs. A traditional CPU is like
a brilliant math professor. It can solve incredibly
complex problems, but it solves them one at a
time, sequentially. Which is terrible for backpropagation,
because you need to update millions of weights
across a whole network. Exactly. Now, a GPU,
on the other hand, is designed to render millions
of individual pixels for a video game simultaneously.
Oh, I see where this is going. Structurally,
a GPU is like an army of elementary school students.
They can't do compliance calculus, but they can
perform thousands of simple arithmetic problems
at the exact same time. This is called parallel
processing. And since we established earlier
that backpropagation is mostly just multiplying
vectors by matrices, which is really just thousands
of simple multiplications. The GPU is the perfect
tool. It's the absolute perfect tool. If you
were playing a high end video game in 2012, the
graphics card humming inside your computer was
exactly the piece of hardware that broke AI from
its hibernation. That is wild. Cheap, powerful
GPUs suddenly made it physically possible to
train networks with dozens or hundreds of layers
using vast amounts of data. And that hardware
evolution ushered in the deep learning revolution
we are living in right now, powering everything
from large language models to machine vision.
And that hardware evolution is still continuing,
right? Oh, very much so. The source notes that
in 2023, a team at Stanford University actually
implemented a back propagation algorithm on a
photonic processor. Yes. Photonic, meaning using
light instead of electricity. Using lasers and
optic to compute the gradients. That sounds like
sci -fi. It does. Photons have zero electrical
resistance, and they move at the speed of light,
which promises to be exponentially faster and
more energy efficient than traditional silicon
chips. The core mathematical algorithm remains
exactly the same. It's the physical medium we
use to run it that's fundamentally changing.
So what does this all mean? We started out looking
at this mysterious black box of AI, and what
we found inside was really just an elegant optimization
problem. We found a blindfolded dart thrower,
feeling their way down a rugged mountain of error
using gradient descent. We found a mathematical
shortcut. Exactly. The chain rule, first scribbled
down by Leibniz in 1676, combined with an attempt
to map Freudian psychic energy, and it's running
on hardware originally designed to render video
game graphics. It's just a stunning testament
to how human knowledge builds on itself, even
when it takes a few wrong turns along the way.
Absolutely. But before we wrap up, the source
material mentions a connection to cognitive science
that kind of completely flips this entire narrative
on its head. It really does. You know, we spent
a lot of time discussing how early researchers
delayed the adoption of backpropagation because
they firmly believed artificial neurons had to
perfectly mimic biological brain cells, which
fire in those discrete steps. Right, the light
switch problem. But recently, the dynamic has
reversed. Cognitive scientists are now suggesting
that error backpropagation might actually explain
human brain event -related potentials. What does
that mean? Specifically, it's referring to brain
wave patterns known as the N400 and P600. Wait,
wait. Scientists are looking at how our artificial
networks learn and wondering if our human brains
are actually using the exact same mathematical
algorithm. Yes. And it raises a really profound
question for anyone listening to this. If artificial
networks learn so powerfully by propagating error
signals backward to adjust their internal weights,
and our physical brainwaves show similar measurement
patterns during language learning and prediction
tasks, it forces us to reconsider our own cognition.
It is intense. Are we humans, at a fundamental
level, just biological algorithms? Are you constantly
calculating your own gradient of error every
single time you make a mistake, propagating that
failure backward through your biological neurons?
to ensure you do better next time, the brain
might not be a black box either. It might just
be adjusting its weights, trying to get the dart
a little closer to the bullseye, something to
think about the next time you try, fail, and
learn something new.