Welcome back to the deep dive. We are really
going for it today. I mean, usually we take a
topic, we sort of peel back the layers and we
try to find that, you know, that core idea at
the center. Right. But today, today we are tackling
something that feels like it's just. It's everywhere
all at once. It's a bit of a fog, isn't it? It's
a total fog. I was walking through the airport
the other day and I saw, I'm not kidding, three
different ads for, get this, AI powered. toothbrushes.
Oh no, the toothbrush that knows your molars
better than you do. Exactly. And it just hit
me. We are reaching that point of saturation
where term just loses all its meaning. It becomes
wallpaper. Yeah. AI, machine learning, neural
networks. They're just these buzzwords we plaster
on everything to make it sound futuristic. It's
like the quantum of the 2020s. It absolutely
is. And it creates this barrier to understanding.
We hear the words, we nod along, but do we actually
know what the machinery is doing under the hood?
Most of the time, no, not at all. And for most
people, and look, I'll include myself in this
before I really start digging into these notes.
When you hear neural network, your brain immediately
goes to Silicon Valley. You think of chat GPT,
you think of generative AI, you think of these
massive data centers humming away somewhere in
the desert. It feels like a strictly computer
science topic. That is the common perception,
yes. And it's not wrong, but it's... It's fundamentally
incomplete. Incomplete. How? If you only look
at the silicon, you are missing half the story.
Actually, you're missing the original story.
It's like describing a human being as just a
calcium skeleton. You're missing the, you know,
the meat. You're missing the origin story. Right.
Because before there was any code. There was
biology. Precisely. Neural networks are not just
a computer science topic. They're fundamentally
a biological one. This whole endeavor, this entire
feel, it's the story of how we tried to reverse
engineer the architecture of thought itself.
Reverse engineering the architecture of thought.
That's a wild way to frame it. But it's true.
We looked inside our own heads. We saw how the
machinery worked, or at least how we thought
it worked. And then we tried to rebuild it. first
with theory, then with math, and now with electricity
and software. It sounds like science fiction,
but it's actually the history of the last hundred
years. It is. So, okay, our mission today is
pretty ambitious. We're going to explore this
definition of a neural network, and the source
material gives us a really clean, simple place
to start. A neural network is just a group of
interconnected units. It sounds deceptively simple,
doesn't it? a group of interconnected units,
that could be anything. It really could. It could
be a group of friends, a telephone system, a
plumbing network in a house. But the magic, and
this is the core of it, isn't in the units themselves.
It's in the connections. It's how those simple
individual units, whether they are wet, squishy
cells in your brain or, you know, lines of code
in a Python script, how they combine to perform
these incredibly complex tasks. It's the idea
that more is different. That's it. A single neuron
is dumb. A trillion neurons. That's a mind. A
single line of code is simple. Billions of them
working together can write a symphony. So this
is the core mechanism we need to understand today.
And the source material sets up this perfect
dichotomy for us. On one hand, you have neuroscience.
Right. The biological world. In that world, a
biological neural network is a physical structure.
It is your brain. It is your nervous system.
It is the hardware inside your skull. It's wet.
It's chemical. It's alive. And then on the other
hand, you have the world of machine learning.
And in that world, a neural network isn't physical
at all. It's a mathematical model. Abstraction.
A pure abstraction. Specifically, it's a model
used to approximate nonlinear functions. Okay,
we are definitely going to have to unpack. approximating
nonlinear functions later, because that sounds
terrifyingly math heavy. We will make it painless,
I promise. But that's the roadmap. We have these
two worlds, the biological hardware inside us
and the artificial software that is changing
everything today. And so the plan for this deep
dive is to follow that path. We're going to start
with the biology, the original blueprint. Then
we'll move into the surprising history of the
theory, which, as I said, is way older than I
thought it was. And we'll end up inside. the
modern artificial mind. Sounds like a plan. Okay,
let's do it. Let's start with that hardware.
The biological blueprint. I want everyone listening
to try and get a mental image of this. So if
you're not driving, maybe close your eyes for
a second. When we say biological neural network,
what are we actually looking at? You are looking
at a population of nerve cells. These are biological
neurons. And you should imagine a dense, dense
forest. But instead of trees with branches that
just kind of touch, you have these incredibly
complex spidery cells. Right. And the key, the
absolute key, is that they aren't just floating
around in isolation like islands. They are chemically
connected to one another. We're communicating.
Constantly. They're connected by what are called
synapses. The synapse. That's the connection
point, right? The little gap between them. Correct.
It's the gap between the neurons where the communication
actually happens. It's not a physical wire. It's
a chemical conversation. Now, here is the stat
from the research that really just stopped me
in my tracks. Okay. We tend to think of connections
as being one -to -one. You know, I call you on
the phone, that's one connection, or I shake
your hand. Right. A connects to B connects to
C. A nice, clean, linear chain. Exactly. But
a biological neuron, a single neuron in your
brain, can be connected to. Wait for it. Hundreds
of thousands of synapses. I'm sorry, say that
again. Hundreds of thousands for one cell. That's
not possible. How can one cell maintain that
many connections? Through an incredibly intricate
branching structure. It's not a simple chain.
It is a dense, unbelievably complex web. If you
have a population of these neurons and each one
has that many potential connections, the number
of possible pathways for a single signal to travel
is... Well, it's astronomical. It's bigger than
the number of atoms in the universe, I've heard.
It's in that ballpark. And that explains why
the brain is so much more powerful than, say,
a pocket calculator. It's not the speed of the
individual components. A transistor is way faster
than a neuron. It's the density of the connections.
It's the ultimate parallel processor. It's like
the Internet. But if every single computer was
directly connected to 100 ,000 other computers,
the bandwidth would be... Unimaginable. So, okay,
we have this insane web of connections. How does
information actually move through it? We aren't
plugging in USB cables here. It's not just pure
electricity like a copper wire, is it? No, it's
electrochemical. That's a key distinction. The
neurons send and receive these little signals
called action potentials. Action potentials.
Think of it as a little pulse, a spike of electricity
and chemistry zipping down the line. It's a binary
event. It either happens or it doesn't. Like
a heartbeat. A little blip on a monitor. Similar
concept, but happening millions of times a second
all over your brain. It zips down the axon of
the neuron. It hits the synapse. It triggers
a little spray of chemicals, neurotransmitters.
And those chemicals drift across the gap and
trigger the next neuron. So it's an electrical
signal that turns into a chemical signal and
then back into an electrical one. Exactly. But
here is where it gets really, really interesting.
And this is crucial for understanding how thought
or processing actually happens. The neuron doesn't
just passively pass the message along. It's not
a bucket brigade. It makes the decision. A single
cell makes a decision. In a way, yes. It acts
as a filter. It has a vote. The source material
highlights that a neuron can have one of two
specific roles. It can be excitatory or it can
be inhibitory. Okay, excitatory sounds pretty
straightforward. It's exciting the signal. It's
hyping it up. You got it. An excitatory role
means it amplifies and propagates the signal.
It says, yes, this is important. Pass it on.
It essentially turns up the volume on that specific
message. It pushes the next neuron closer to
firing its own action potential. Okay, so what
about the inhibitory role? That sounds like...
Like a buzzkill. It's the necessary buzzkill.
And it's arguably more important. An inhibitory
neuron does the exact opposite. It suppresses
the signal. It says, stop. This message does
not go any further. It tells the next neuron
not to fire. Yes. It makes it harder for the
next neuron to fire. So for every signal that's
saying go, go, go, you have others that are saying,
no, wait, stop. That binary choice. amplify or
suppress, that feels like the absolute root of
everything. It is. Think about it for a second.
If every neuron in your brain just shouted yes
to every signal it received, what would happen?
Just chaos. White noise. You'd be having a seizure.
That is literally what a seizure is. Uncontrolled,
cascading, electrical firing. The ability to
inhibit, to say no, is what allows for precision.
It allows for focus, for complex processing.
So inhibition is like a sculptor. It's chiseling
away all the noise so that a coherent thought
can actually emerge. That is a beautiful way
to put it. That delicate, dynamic balance between
excitation and inhibition is critical to every
single thing you do. It's what allows you to
ignore the feeling of your shirt on your skin
right now so you can focus on listening to this
conversation. Right, or to pick out a single
voice in a crowded, noisy room. Without inhibition,
we are just static, just noise. So we have these
individual units these neurons making these simple
binary choices But we aren't just talking about
one or two neurons here the source material talks
about A hierarchy of scale. Right. It's not a
flat system. We start at the micro level. Small
populations of interconnected neurons are called
neural circuits. Think of these as little specialized
teams, maybe 100 neurons or so, that are designed
to handle one very specific task. For example,
there are circuits in your visual cortex that
do nothing but detect vertical lines. That's
their whole job. Dude, just vertical lines. That's
it. And another circuit right next to it is just
for horizontal lines. Yeah. Another for diagonal
lines. And then you scale up from there. You
scale up, you combine those circuits, and you
get large -scale brain networks. These are massive,
interconnected systems that span different regions
of the brain. The visual network, the auditory
network, the language network. And when you combine
all of those, you get the full system. The brain
and the entire nervous system. It's layers upon
layers upon layers of complexity, starting from
that simple yes -no vote. And what I found really
fascinating in the reading, and you touched on
this, was that the output of this whole biological
machine isn't just thinking. I think we get trapped
in our heads a bit. We think neural networks
are for doing philosophy or calculus or writing
poetry. Right. We think of it as this purely
abstract thing. We think of the brain as a computer
that just outputs data to a screen in our minds.
Yeah. But biologically, the output is physical.
It has to be. The source explicitly mentions
that these electrochemical signals travel down
through the nervous system. They go across what
are called neuromuscular junctions and they hit
muscle cells. And that causes contraction. It
causes motion. The biological neural network
is fundamentally a machine for turning electrochemical
signals into physical movement. Everything you
do, typing an email, running for a bus, even
the tiny movements of your vocal cords while
you're speaking right now. That is the output
layer of your biological neural network in action.
It's a machine that turns thought into action.
Literally. From a purely biological evolutionary
standpoint, the entire purpose of the brain is
to move the body effectively through the world.
If you don't move, you don't find food, you don't
escape predators, you don't survive. So all that
complex processing, all those hundred thousand
connections, it all funnels down to one final
command. Contract this muscle, relax that muscle.
That just grounds it so much. We're not just
thinking machines. We're movement machines. Primarily.
So that's the biological hardware. This incredibly
complex, dense web of neurons using excitatory
and inhibitory signals to drive motion and thought.
Now, I want to pivot to the history of the idea,
because this was the part of the deep dive that
genuinely shocked me. The timeline. The timeline.
I mean, when I think of neural network theory,
my brain jumps to the 1980s, maybe the 1950s,
if we're being generous. To computer age. Exactly.
But the source material takes us back to the
19th century. It is surprising, isn't it? We
think of this as a modern tech invention, but
the theoretical base was laid down by Alexander
Bain in 1873 and then William James in 1890.
1873. Let's just... Let's contextualize that
for a minute. In 1873, people were still getting
around on horses. We didn't have computers. We
barely had reliable light bulbs in homes. The
telephone was barely a concept. What on earth
were they doing proposing neural network theories?
Well, they were psychologists and philosophers,
and they were wrestling with the biggest question
of all. What is the mind? What is consciousness?
At the time, the prevailing view was still very
spiritual or dualistic. The mind was a soul.
A separate thing from the body, the ghost in
the machine. Precisely. And Bain and James challenged
that head on. They had this core insight that
was absolutely revolutionary for the time. Which
was? They posited that human thought wasn't magic.
It wasn't some ethereal spirit floating around
in your skull. They argued that human thought
emerges from the physical interactions among
large numbers of neurons inside the brain. That
is a huge leap. To look at the gray matter in
a skull and say, the interaction of these physical
cells is what creates my subjective experience
of being alive. It's the birth of a materialist
understanding of the mind. They were basically
saying, if you connect enough of these biological
switches in the right way, you get a mind. They
couldn't prove it, of course. They didn't have
MRIs or electrodes. But the intuition was spot
on. So they laid the philosophical foundation.
But it seems like it just sat there for a while,
right? As a theory. It did. It remained largely
in the realm of philosophy until we get to the
mid -20th century. That's when things really,
really started to accelerate. We enter the era
of what was called connectionism. Connectionism.
That sounds like a networking event for LinkedIn.
Laughs. Not quite. In the 1930s and 40s, connectionism
was this new approach. It was about using artificial
networks to model biological ones. Scientists
started asking, OK, if the brain really works
by connecting all these units, can we build a
simplified model that does the same thing? It's
the shift from just observing how does it work
to asking, can we build one? It's the engineering
mindset taking over. And this leads us directly
to 1943. Two names that are incredibly important,
Warren McCulloch and Walter Pitts. The inventors
of the perceptron. Well, they created the model
that the perceptron was based on. They created
the first mathematical model of a neuron. The
perceptron. It really sounds like a Transformers
character. Beware the perceptron. It does, doesn't
it? But the perceptron is essentially the great
-great -grandfather of chat GPT. It was the first
simple artificial neural network. So what did
they do? What was the breakthrough? They did
something brilliant. They took that biological
neuron we just talked about, the one that gets
signals, adds them up, and decides to fire, and
they turned it into algebra. They said, okay,
a neuron takes inputs, it sums them up, and if
that sum hits a certain threshold, it fires.
We can write an equation for that. They abstracted
the biology into math. Exactly. And that was
a monumental step. But right around that same
time in 1949, we get another giant, Donald Hebb.
And this is a name that comes up constantly in
neuroscience. He's famous for Hebbian learning.
Yes. This seems to be the golden rule of neuroscience.
The one thing everyone knows. It is because Hebb
gave us the mechanism for how the network learns.
You see, McCulloch and Pitts built the car, but
they didn't know how to drive it. The weights
were fixed. Hebb explained how the driving works.
And what was his insight? The source material
puts it really clearly. It's the idea that neural
networks change and learn over time by strengthening
a synapse every single time a signal travels
along it. Strengthening a synapse every time
a signal travels along it. So if I practice a
tennis swing, the signal to my arm goes down
a certain neural path. And just because it went
down that path, the path itself gets stronger.
Exactly. It's like a trail in the woods. This
is the best analogy for it. Imagine you're standing
in a dense forest. There is no path. But you
need to get from point A to point B. The very
first time you walk it, you are hacking through
brush with a machete. It's slow. It's hard. It
takes a lot of effort. But you leave a trail
behind you. A faint one. A very faint one. But
if you walk that same route tomorrow, it's a
little bit easier. You've already trampled some
of the grass. Right. If you walk it every day
for a month, the dirt starts to pack down. If
you walk it for a year, it becomes a clear, wide
path. You can eventually run down it without
even thinking about where you're putting your
feet. And that is what a synapse does. Literally.
On a physical, chemical level, the connection
becomes more efficient. The physical structure
of the synapse changes to allow the signal to
pass faster and more strongly. This is the biological
basis for practice makes perfect. It is the physical
manifestation of learning in your brain. It's
wild to think that learning isn't just an abstract
concept. It's a physical thickening and strengthening
of connections in your head. You are literally
terraforming your own brain with your thoughts.
It gives a whole new... weight to the idea of
habit forming, doesn't it? Every time you have
a thought, you are physically sculpting your
brain to make it easier to have that exact thought
again. That is both incredibly inspiring and
slightly terrifying at the same time. It really
is. And the timeline keeps moving. So in 1956,
we had another biological breakthrough that's
mentioned in the text. So Adichan discovers the
functioning of second -order retinal cells, these
Horizontal cells. Right. This was a really crucial
discovery for the biological side of things because
it showed us that the processing wasn't just
happening deep inside the brain. It was happening
right at the sensor. In the eye itself. Yes.
The retina is literally a part of the brain that's
pushed out to the front of your face. And those
cells were doing their own compression and processing
like edge detection before the signal was even
sent down the optic nerve. It proved that these
networks are distributed and hierarchical right
from the beginning. So we have the theory from
James. We have the perceptron math from McCulloch
and Pitts. We have Hebbian learning explaining
how connections strengthen. We have Svetochin
examining the retina. And then in the late 50s,
things start to get physical in the artificial
world. They get very physical. In 1957, a researcher
named Frank Rosenblatt... implements the perceptron
in hardware. Hardware. So this wasn't a simulation.
This wasn't software running on a chip. No, chips
as we know them didn't exist yet. This was a
physical machine. It was called the Markahaya
perceptron. And it was the size of a room. It
had hundreds and hundreds of cables like a giant
telephone switchboard. It had a camera that looked
at shapes. And inside, to represent the weights
of the connections, it had potentiometers, basically
physical volume knobs that were attached to little
electric motors. You're kidding. Motorized volume
knobs. I'm not. To learn. The machine would physically
turn the knobs to adjust the electrical resistance
in the circuits. It was a physical, mechanical
embodiment of a neural network. That is incredible.
That's like a steampunk AI. It basically was.
But this is also where we see the great divergence,
the big split. The split between the two worlds
we talked about in the intro, the biological
and the artificial. Right. Up until this point,
the main goal was largely to model biology, to
try and understand the brain. But as these artificial
networks started being built, And as they got
better, they began to drift away from their biological
counterparts. Why? Researchers realized that
if your only goal is to solve a math problem,
like recognizing a letter of the alphabet or
calculating a missile trajectory, you don't actually
need to be perfectly biologically accurate. You
don't need all the squishy, messy chemistry.
You don't. You can simplify. You can cheat. You
just need the system to work. So the engineers
and computer scientists started to focus purely
on machine learning applications. They stopped
trying to build an exact copy of a brain and
started trying to build a machine that could
learn, period. And that's the path that led us
to where we are today. It's the fork in the road
that created the modern AI industry. So let's
cross that bridge now. We are leaving the wet,
squishy world of biology and 19th century psychology,
and we are entering the artificial mind. The
domain of pure software. And the source material
notes this transition explicitly. Early networks
were these physical machines, like Rosenblatt's
room -sized monster. Today, they are almost always
implemented in software. And we have this definition
again. An artificial mathematical model used
to approximate nonlinear functions. All right.
We have to tackle that phrase now. It's the elephant
in the room. Please. Nonlinear functions. What
does that mean in plain English? And why is it
such a big deal that these networks can approximate
them? Okay. To understand nonlinear, you first
have to understand what linear is. Think of a
linear function as a perfectly straight line.
It's simple, predictable, cause and effect. If
I double the input, I double the output. Every
time. Okay. Like buying apples at a store? The
perfect example. If one apple costs $1, two apples
cost $2, 10 apples cost $10, you can draw it
on a graph, and it's a perfectly straight line
going up. It's predictable. It's easy. Got it.
But the real world... the real world is almost
never linear real world relationships are messy
they're complex think about the relationship
between say the amount of coffee you drink and
your productivity oh that is definitely not a
straight line not at all if you have one cup
Your productivity goes up. You feel alert. You
have two cups. Maybe it goes up a little bit
more. But if you have 10 cups of coffee, you're
vibrating under your desk and your productivity
hits absolute zero. Exactly. The line goes up,
then it curves over and then it crashes down.
That is a nonlinear relationship. The output
does not move in a straight, predictable line
with the input. Ah, OK. So it handles the curve.
It handles the complexity. And here's why this
matters for AI. Almost every interesting problem
in the world is nonlinear. Identifying a face
in a photo is a highly non -linear problem. There
isn't a straight line equation where pixel X
plus pixel Y equals Steve. Because Steve might
be turning his head or could be standing in a
shadow or wearing glasses today. Exactly. The
relationship between the arrangement of pixels
and the identity of Steve is incredibly complex
and messy. It twists and turns. Standard linear
math fails miserably at this kind of problem.
Neural networks are designed specifically to
handle that messiness. They are approximation
machines for complex, messy realities where A
plus B does not always equal C. So they are the
off -road vehicles of mathematics. They can handle
the rough, bumpy terrain where other methods
would get stuck. That is a fantastic analogy.
Yes. They can map any curve, no matter how wiggly
it is. Okay, that is actually really, really
helpful. So how do they do it? Well, what does
the architecture of one of these software networks
look like? Because we can't see the synapses
anymore. It's just code. We visualize it in layers.
The source material breaks it down into three
main types of layers that you'll always find.
First you have the input layer. That's where
the information enters the system. Right. If
it's a facial recognition system, the input layer
is just the raw pixels of the photo. It takes
the image, say a grid of a thousand by a thousand
pixels, and it turns it into a list of a million
numbers. So it digitizes the world. Precisely.
Then you have the hidden layers. These are the
intermediate layers where all the processing
happens. This is the black box where the real
magic occurs. And finally, the output layer.
The final result. A single neuron might light
up that corresponds to this is a photo of Steve
or this is a span. Or maybe a set of numbers
that means turn the steering wheel 15 degrees
to the left. Input, hidden output. Seems like
a simple enough structure. But what is actually
happening inside those hidden layers? How does
an artificial neuron think? Because it doesn't
have a brain. It's just a number. This is where
we get into the math, but stick with me. It's
pretty intuitive. The source says the signal
input to a neuron in one of these layers is a
number. Specifically, it is a linear combination
of the outputs from all the connected neurons
in the previous layer. Linear combination. That
just sounds like it's mixing a bunch of things
together. That's exactly what it is. Imagine
you are a single neuron in one of the hidden
layers. You're like a chef. You have all these
ingredients coming in from the previous layer.
The neurons in the previous layer are all shouting
things at you. One is saying, I see a strong
vertical line here. Another is saying, I see
a shadow over here. And a third says, I'm detecting
the color blue. The ingredients are the signals
from the last layer. Yes. But, and this is the
absolute key to everything, the neuron doesn't
listen to them all equally. It pays more attention
to some than others. It relies on weights. Weights.
Okay. This is bringing us right back to heavy
learning and the trail in the woods. It's the
software version of that. The behavior of the
entire network depends on the strengths or weights
of the connections between the neurons. In the
software, a weight is just a number that represents
that strength. So just like the biological synapse
gets physically stronger. In an artificial network,
the weight number just gets bigger. If the weight
on a connection is a high positive number, the
connection is strong. The signal gets amplified.
If the weight is a low number or a negative number,
the signal gets ignored or even suppressed. So
if I'm an artificial neuron trying to identify
a cat, and I'm connected to a neuron in the previous
layer that has detected pointy ears, I might
have a really high weight on that connection.
I'm listening very closely to the pointy ears
guy. Exactly. You'd say, pointy ears, that's
super important for being a cat. Multiply that
signal by 10. And if I'm connected to a neuron
that detected has wheels... You might have a
strong negative weight on that connection. You'd
say, wheels, cats don't have wheels. Suppress
that signal. multiply it by next five, and make
it count against the final total. Usually. Usually.
Unless you're training it on cartoon cats, which
is a whole other problem. So the neuron takes
all these inputs, multiplies each one by its
corresponding weight, and then it just sums them
all up. That's the linear combination. But then
there's one more critical step, the activation
function. The gatekeeper. The gatekeeper, exactly.
The neuron's final output signal is calculated
from that big summed up number based on this
activation function. It's a simple little piece
of math that decides, okay, is this total number
high enough to be important? Is it strong enough
to fire and send a signal onto the next layer?
So this is the artificial version of that excitatory
versus inhibitory decision we talked about in
the brain. That's precisely what it is. It introduces
the non -linearity we need. Without the activation
function, the whole network would just be one
giant, very complicated multiplication problem.
It would still be linear. It couldn't learn those
complex curves. The activation function allows
it to say, unless the evidence is this strong,
just ignore it. It creates boundaries and allows
for complex decisions. So to recap, the neuron
sums up all the weighted inputs, it runs that
total through the activation function, and then
it decides whether to pass a message on and how
strong that message should be. It's remarkably
similar to the biological process. It's a mathematical
mirror of it. It's an elegant simplification.
But here is the million -dollar question, or
the trillion -dollar question now. In biology,
we learn by practice. The signal travels down
the path. The synapse gets physically thicker.
How does a software program learn? How does it
know what the weights for all those connections
should be in the first place? Because there are
billions of them. Trillions in the really big
modern models like GPT -4. And you're right.
You can't set them by hand. You can't sit there
as a programmer and dial in a trillion knobs
to the right settings. So how do we do it? How
does it figure it out? That is the process we
call training. And the source material gives
us two somewhat intimidating terms for this process,
empirical risk minimization and backpropagation.
Let's focus on the concept first, though. How
do you train the beast? Okay. So ideally, you
want the network to look at a picture of a cat
and say, cat. But when you first start, all those
trillions of weights are just set to random numbers.
The network is a complete idiot. It knows nothing.
So you show it the first picture of a cat. And
it runs the numbers through all the layers. And
at the end, it guesses. laughs, swing and a miss.
A huge miss. Now, the math knows it's wrong because
you, the trainer, have a pre -existing data set.
You have the answer key. You've labeled that
picture cat. So the system can calculate the
error. It knows how wrong it was. Exactly. It
calculates the loss or the cost. Ideally, the
loss should be zero. Here, the loss is huge.
Using a clever algorithm called back propagation,
it goes backwards through the network. Backwards
from the answer to the input. Yes. This is the
genius of it. It starts at the output layer and
moves back towards the input layer. It looks
at all those weights that contributed to the
final decision and it says, OK, who screwed up?
Which connection, which weight contributed most
to this ridiculous toaster decision? It's like
a manager walking back through the assembly line
after a defective product comes out at the end.
That is the perfect analogy. The manager walks
back down the line and says, you, station four,
you tightened that bolt way too much. That made
it look metallic and shiny. Let's nudge your
weight down a little. And you, station two, you
didn't add nearly enough weight to the fuzziness
neuron. Let's nudge your weight up. It mathematically
tweaks all the weights just a tiny bit in the
direction that would have made the error smaller.
It signs blame. It assigns blame mathematically
and adjusts the weights to reduce the error.
And then it does it again and again and again
and again. Millions, sometimes billions of times.
Sees another cat. It guesses dog. Big error.
Adjust all the weights. It sees another cat.
Guesses lion. Smally error, but still wrong.
Adjust the weights again. It sees another cat.
And finally, it guesses cat. Success. The error
is zero. Keep those weights for now. So training
is essentially just this. automated repetitive
process of showing the network an example seeing
how wrong it is and using calculus to nudge all
the connection strengths until the output matches
the data you want that is all it is it feels
like intelligence it looks like learning but
under the hood it is a brute force optimization
problem it is statistics and calculus finding
the path of least resistance through a mathematical
landscape with trillions of dimensions that is
Slightly underwhelming, but also incredibly impressive
at the same time. It takes the magic out of it,
but it adds this whole new layer of awe at what
pure, relentless math can actually achieve. So
that's the mechanism. We have the architecture,
we have the weights, we have the training process
with backpropagation. Now I have to ask about
the big buzzword, deep learning. You hear it
everywhere. Deep neural networks. Is that just
marketing, or does it actually mean something
specific? It is a specific technical term, and
the definition is surprisingly concrete and,
well, simple. The source says a deep neural network
just refers to neural networks that have more
than three layers. That's it. The definition
of deep is just more than three. That's the technical
threshold. Typically, it includes the input layer,
the output layer, and at least two hidden layers.
Anything with two or more hidden layers gets
to call itself deep. So deep just refers to the
depth of the layers. It's a structural definition.
Yes. Early networks back in the 80s and 90s were
shallow. Maybe they had one hidden layer. But
the modern networks, the ones running GPT or
AlphaGo or self -driving cars, they are incredibly
deep. We're talking dozens, sometimes hundreds
of layers. Why does that depth matter so much?
Why is that the secret sauce? Why not just have
one really, really wide hidden layer with billions
of neurons? Hierarchy. That is the secret. Deep
networks learn in a hierarchy of features, from
simple to complex. Think about how you, a human,
recognize a car. Okay. You don't just see car
all at once. Your brain processes it in stages.
The first layer of neurons in your visual cortex,
as we mentioned, might just learn to recognize
simple things. Edges, gradients, patches of color.
The most basic building blocks. Vertical lines,
horizontal lines. Exactly. The next layer up...
takes the outputs from that first layer and learns
to combine them. It says, okay, if I see four
lines arranged in a rectangle, that's a shape.
If I see a continuous curve, that's a circle.
It learns simple shapes. So wheels and windows.
Right. And the layer above that combines the
shapes into components. It says a circle next
to another circle on top of a rectangle that
looks like a wheel assembly. Or this specific
collection of shapes looks like a door handle.
And the final layers combine those components
into the abstract concept of car. So each layer
is building on the abstraction of the one before
it. From pixels to lines to shapes to objects
to the final concept. Exactly. It moves from
the very concrete to the very abstract. That
is what deep learning allows. It allows the computer
to build a complex understanding of the world
from simple parts, step by step, just like we
think our own brains do. And that is why we use
them for so many things now. The applications
listed in the source are pretty broad. It's a
huge list. Yeah, it's extremely broad because
this hierarchical feature learning turns out
to be applicable to almost... every complex problem
you can think of the list here includes predictive
modeling adaptive control facial recognition
handwriting recognition general game playing
and of course the big one now generative ai and
if you look at that whole diverse list they all
share one common thread they are all problems
where the rules are incredibly hard to write
down explicitly right You can't sit down and
write a computer program with a million if -then
statements for what a cat looks like because
there are just too many variations. Exactly.
You can't program it line by line. You can't
say if it has pointy ears, then it's a cat. Well,
a fox has pointy ears. If it has whiskers, a
rat has whiskers. You can't capture the essence
of catness with a set of logical rules. But you
can train a deep neural network to just sort
of feel it out. You can train a network to approximate
the nonlinear function of Katniss. That is the
power of this technology. It allows us to solve
problems that we humans know how to do intuitively,
like recognizing a face or understanding a sentence
or driving a car, but that we don't know how
to explain logically or mathematically. We just
let the network figure out the math for us by
looking at millions of examples. We're basically
mimicking our own intuition that we can't even
articulate ourselves. Perfectly put. We're outsourcing
our intuition to... So we have traveled quite
a distance today. We started this deep dive with
a single biological neuron vibrating with potentially
100 ,000 synapses, sending these tiny electrochemical
signals through a dense forest in our heads.
We walked through the gaslight era of William
James and Alexander Bain when the very idea that
thought was physical was a radical concept. We
saw the perceptron, that room -sized tangle of
wires and motorized knobs. And we watched that
moment where biology and engineering went their
separate ways. And we ended up inside these deep,
multilayered software architectures of generative
AI, where pure calculus is acting as the teacher,
adjusting trillions of weights to get the right
answer. It really is quite a journey when you
lay it all out like that. And for me, the biggest
takeaway, the thing I'm going to be thinking
about later this week, is that convergence at
the end. The convergence of the two worlds we
started with. Yeah. We started by trying to model
the brain. We created this simplified mathematical
approximation. And now... We're using that math
to solve the exact same problems, like facial
recognition or understanding language, that the
biological brain evolved over millions of years
to handle in the first place. It's come full
circle. We outsourced a biological function to
a silicon chip by first copying the biological
architecture. It raises a bit of a provocative
question for the end, doesn't it? It certainly
does. The source material leaves us with this
final comparison of what learning is. In the
brain, it's Hebbian learning. Strengthening the
synapses via signal travel. The famous line,
neurons that fire together wire together. And
in the machine, it's about modifying weights
via back propagation. It's about minimizing a
mathematical error function. The mechanisms are
totally different. One is wet chemistry. The
other is cold calculus. But the result is strikingly
similar. A system that improves with experience.
So the final thought is this. Are we just mathematical
models ourselves, approximating our own biology?
Or is the math just a really, really good shadow
of the real thing? That is the question, isn't
it? If we can replicate the result of thought,
of creativity, of learning, using just simple
math and optimization on a massive scale, does
that mean that our own thought is just simple
math and optimization on a massive scale? That
is the dangerous and fascinating thought. If
a machine can write a poem by just minimizing
the error between its output and all the poems
it's ever read, does that diminish our own poetry?
Or does it just mean our brains are doing the
same thing just with a different kind of hardware?
I don't know if I'm ready to answer that. I think.
I like to think I'm more than a linear combination
of weighted inputs. I like to think there's still
a ghost in my machine. Laughs. We all like to
think that. But the trillions of weights in your
brain might disagree. They might just be firing
according to their training data, just like the
AI. On that slightly existential note, I think
we are going to wrap up this deep dive. Thank
you for listening. Yes. Hopefully we have strengthened
some of the right synapses in your brain today.
And hopefully we didn't inhibit too many of the
others. Exactly. We'll see you in the next one.