In 1966, some of the absolute smartest computer
scientists on the planet thought they could teach
a machine to see the world over a single summer
break. Which is just wild to think about now.
Right. I mean, they literally set it up as an
undergraduate project at the MIT AI lab. The
whole plan was to just, you know, Attach a camera
to a computer, hand it over to the students,
and basically say, hey, write a program to make
the machine describe what it sees. Yeah, they
genuinely assumed it would be an easy problem
to solve, just a quick summer assignment. Exactly.
And now... decades later, we realized that teaching
a machine to quote -unquote see is actually one
of the most complex mathematical bubbles in human
history. It really perfectly captures the sheer
hubris of early computer science, doesn't it?
I mean, back then, we just thought vision was
a hardware problem. Like, you know, if we just
build a better electronic eye, the computer will
automatically understand the physical world.
Right. But the reality is so much more complicated.
So welcome to the deep dive. Today, we are taking
a massive stack of research spanning decades
of computer science, physics, and biology, and
our mission is to unpack exactly how we actually
solved this puzzle. Because it's a huge puzzle.
It is. We want to know how we took machines,
which, let's be honest, at their core are just
rocks made of silicon, billions of microscopic
switches turning light into math, and taught
them to not just capture a grid of pixels, but
to deeply understand what those pixels actually
mean. And that distinction is crucial. Okay,
let's unpack this. Because it's one thing to
teach a computer to capture a static photograph.
It is an entirely different universe to teach
it, that the pixels in that photograph represent,
say, a dog catching a frisbee in three -dimensional
space. Right. And what's so fascinating here
is that this transition isn't just about cameras
at all. It's really about the disentangling of
symbolic information from raw data. Disentangling
symbolic information. Exactly. And to do that,
scientists have had to construct these incredibly
complex models using geometry, physics, and learning
theory. It's this massive interdisciplinary effort.
And I actually ask you, the listener, to think
about that as we go through this today. What
stands out to you when you consider that the
devices around you are just constantly, relentlessly
trying to interpret your physical world? It's
honestly kind of mind -blowing when you start
looking at the real mechanics of it. And to really
appreciate the invisible systems running on your
phone or in your car right now, you have to look
at why that 1966 summer project failed so spectacular.
Oh, it completely crashed and burned. Yeah. And
I was thinking about this. It's like... assuming
that because breathing is totally effortless
for human beings, building a fully functional
artificial lung from scratch is just a fun weekend
DIY project for a college kid. That is a great
analogy. Because seeing is so instinctive for
us, we just naturally assumed it would be computationally
simple for a machine. Which I think reveals a
really profound truth about artificial intelligence
in general. The things that are traditionally
hardest for computers, like calculating the exact
trajectory of a rocket to the moon are very difficult
for us humans. Right. I certainly can't do that
math in my head. Exactly. But the things that
are easiest for us, like instantly recognizing
a familiar face in a crowded room or just telling
the difference between a flat shadow and a solid
object. The stuff a toddler does without trying.
Right. Those things require millions upon millions
of data points for a machine to process. So the
summer of 66 comes and goes. The undergrads,
shocker, do not solve human vision. And the field
actually has to slowly grind forward decade by
decade. Very slowly. Yeah, by the 1970s, they
weren't even trying to recognize faces anymore.
They were literally just trying to write algorithms
that could detect basic edges and straight lines,
just looking for the boundaries of objects. Because
without a boundary, an object doesn't actually
exist to a computer. It's just a meaningless
sea of color. Wow. And then as we move into the
1990s, the focus really shifted to rigorous mathematical
optimization. To get a computer to understand
3D space, researchers had to start borrowing
concepts from a field called photogrammetry.
OK, this is a great concept to pause on, because
photogrammetry sounds incredibly dense. But it's
basically just the science of making 3D measurements
from 2D photographs, right? Precisely. I mean,
if you take a picture of a building from the
front and then another picture from the side,
your human brain naturally understands the 3D
shape of that building. We just stitch it together
in our heads. Right. But a computer doesn't.
So they use this concept called bundle adjustment
theory. Bundle adjustment. Yeah, it's this incredibly
heavy mathematical process used to calculate
the exact angle and position the camera was in
for every single shot. Once the computer calculates
exactly where the camera was in space, it can
triangulate the light and build a sparse 3D reconstruction
of the whole scene. That sounds like a staggering
amount of math just to know what a building looks
like. It is. But even with all that 3D math.
The real modern breakthrough, like, the thing
that brought the field to where it is today was
the shift to deep learning and CNNs, right? Convolutional
neural network. Absolutely. CNNs changed everything.
And I really want to make sure we explain how
a CNN actually looks at an image, because it
fundamentally changed the game. Before this,
programmers were like, manually writing code,
saying, OK, look for a circle, now look for two
eyes. Which is just impossible to scale. You
simply can't write a manual rule for every single
object in the universe. Right. So a convolutional
neural network does something entirely different.
Imagine you're sliding a tiny mathematical magnifying
glass. They call it a filter over an image just
pixel by pixel. Gaining the whole thing. Yeah.
And the first layer of the network just uses
that filter to look for something super basic,
like a horizontal line. Then the next layer takes
those lines and looks for a curve. Building up.
Exactly. The next layer combines lines and curves
to find a shape, like say a wheel or an ear.
It literally builds visual comprehension from
the ground up entirely on its own. And it works.
I mean, when you look at massive benchmarks like
the ImageNet challenge, which tests AI on millions
of images across a thousand different object
classes, deep learning models now perform basically
close to the level of humans. Close to humans,
yes. But they still make some truly bizarre mistakes.
Oh, totally. Like... These advanced algorithms
can easily categorize a highly specific, fine
-dried breed of dog or a rare species of bird
something most normal humans would completely
fail at. Right, it's superhuman at trivia. Exactly.
but they still struggle immensely with tiny details
that you or I would process without a second
thought. For instance, a small ant resting on
a flower stem. Oh, interesting. Yeah, to a CNN,
the ant might just look like noise or a glitch
in the stem's texture. It misses the context
entirely. Or think about images distorted by
camera filters. Like, if someone puts a weird
sepia tone or a heavy glitch effect over a photo
on social media, a human looks past the filter
instantly. We just know, oh, it's a person with
a weird filter. Right. But that same filter...
can completely trip up the neural network because
the foundational pixel colors have drastically
changed. And the reason for this limitation goes
right back to the fact that pure mathematics
just wasn't enough to solve the problem of true
visual understanding. Math only gets you so far.
Exactly. Researchers eventually realized they
had to bridge the gap between silicon hardware
and biological wetware to really teach a machine
to see they had to start borrowing heavily from
nature. which means crossing over into entirely
different scientific disciplines. I mean, just
to design the physical image sensors that capture
the light in your smartphone camera, we have
to rely on solid state physics and quantum mechanics.
It's incredible when you think about the layers
involved. It really is. Physics explains how
the light bounces off a coffee cup on your desk.
And then quantum mechanics is required to fully
understand how those individual photons hit the
digital sensor and convert into an electrical
charge. But the biological inspiration is where
the real leap in computational processing happens.
Neurobiology has massively influenced computer
vision over the decades. Because we had to study
ourselves first. Right. Over the last century,
scientists have extensively studied mammalian
eyes, neurons, and how the physical brain processes
visual stimuli. And that research led to an early
direct in the 1970s called the Neocognitron.
Right, developed by Kunihiko Fukushima. This
was an artificial neural network specifically
designed to mimic the primary visual cortex of
the actual brain. Yes. And to understand how
these biological inspired networks learn, and
frankly how they fail, there is a brilliant sort
of simplified example from the research regarding
how a network learns to detect objects. Let's
look at the starfish and the sea urchin. Oh,
this is a classic way to explain it. It makes
so much sense. So imagine you are training a
neural network to tell the difference between
a starfish and a sea urchin. You feed it hundreds
of images of both. Over time, the network creates
nodes, these little decision centers, representing
visual features. Okay. For the starfish, it strongly
correlates a ringed texture and a star -shaped
outline. For the sea urchin, it matches a striped
texture and an oval shape. Seems totally logical.
Starfish equals rings and stars. Sea urchin equals
stripes and ovals. It's just assigning mathematical
weights to those physical features. It is. But
here is the catch. What if, during that massive
training process, there happens to be a single
sea urchin image that has a slightly ringed texture
instead of stripes? Just a weird anomaly in the
training data. Exactly. The network creates a
weakly weighted association. It essentially notes,
OK, a ringed texture might occasionally mean
sea urchin. OK, I see where this is going. Right.
Now, you run the train network on a brand new
image. It correctly spots a starfish in the sand
because of the strong ring and star signals.
But because of that one weird training image,
the ringed texture of the starfish also sends
a tiny weak signal suggesting there might be
a sea urchin nearby. The machine is getting a
little confused. Yeah. And now imagine a completely
random oval clamshell is sitting next to the
starfish in the image. The network wasn't trained
on clamshells at all. But the clamshell is an
oval. Yes. And the oval shape of the shell triggers
the sea urchin's strong oval node. So it takes
the weak ring signal from the starfish and combines
it with the strong oval signal from the clamshell.
Exactly. And suddenly the network outputs a completely
false positive. confidently declares there is
a sea urchin in the photo when it's really just
looking at pieces of a starfish in a shell. Here's
where it gets really interesting to me because
that mechanism is so fundamentally human. How
so? It completely reminds me of teaching a toddler
to speak. If you only ever show a toddler brown
dogs and you point and tell them dog, the very
first time they see a brown cat walking down
the street, they might confidently point and
say dog. That's perfect. Yes. They're overweighting
the wrong features, the color brown and the fact
that it has four legs, and they are completely
ignoring the shape of the ears or the tail. It's
the exact same false positive as the neural network.
And if we connect this to the bigger picture,
what we're really seeing here is an incredible
interdisciplinary feedback loop. It really is.
The study of biological vision helps us build
better computer vision algorithms. And in turn,
designing these complex artificial neural networks
actually gives neuroscientists a working testable
model to understand our own neurobiology. We
are essentially reverse engineering ourselves.
So we have the biological inspiration from the
brain and we have a quantum physics in the camera
sensors. But how does a modern system practically
process an image? in the real world, like step
by step. The actual mechanics of it. Yeah, let's
walk through the actual pipeline. But first,
I think we really need to clarify the difference
between image processing and computer vision,
because I hear them conflated all the time. That
is a crucial distinction to make. In image processing,
the input is an image, and the output is just
an enhanced image. Like Instagram filters. Exactly.
Think of hitting the auto -enhance button on
your phone, adjusting contrast, rotating a photo,
or removing background noise. The computer doesn't
actually know what is in the picture. It's just
blindly manipulating light values. Computer vision,
on the other hand, takes an image or video as
input, and the output is a decision. It's a 3D
model, an analysis, or a specific system behavior
based on what it actually understood. OK, so
the output of computer vision is an action. So
how do we get from raw light to a concrete decision?
There is a very strict five -step pipeline these
systems go through. To make this concrete for
you listening, let's use a real -world example
as we go through the steps. Let's imagine a camera
on a sterile factory line inspecting microscopic
wafer computer chips for tiny defects. Perfect
example. So step one is image acquisition. Now
this isn't just a standard webcam. Depending
on the application, it could be lidar, radar,
or ultrasonic sensors. But in our factory example...
In the factory, it's likely a high -resolution
microscopic camera capturing the raw photon data
reflecting off the silicon wafer as it moves
super fast down the conveyor belt. Okay, so it
snaps the raw data. Then comes step two, pre
-processing. Before the heavy lifting begins,
the system has to clean up that data, right?
Yes. It might remove the glare from the harsh
factory lighting or enhance the contrast so the
tiny microscopic circuits are actually detectable
by the math. Which perfectly sets up step three,
feature extraction. This is where those mathematical
filters we talked about earlier come in. The
system starts pulling out the lines, the sharp
edges of the circuits, or localized interest
points like corners and metallic blobs. Exactly.
So does this mean the computer's basically playing
a high -speed, multi -dimensional game of connect
the dots before it even knows what it's looking
at? That is an absolutely excellent way to visualize
it. It's looking for structural primitives. It
doesn't know it's a microchip yet. It just knows
there are 20 parallel lines and a shiny metallic
blob over here. Just shapes and lines. Right.
And this raises a really important question about
the fundamental nature of this pipeline, which
is data reduction. Because once you have those
features, you move to step four, detection and
segmentation. Right, the system has to decide
which specific regions of the massive image are
actually relevant. It segments the image, essentially
isolating the specific tiny solder joint on the
chip from the completely useless background of
the conveyor belt. It's separating the signal
from the noise. Exactly. And finally, step five,
decision -making. The true genius of this pipeline
isn't just its ability to take in massive amounts
of data. It's the system's ability to ruthlessly
throw data away. Yes. It distills millions of
pixels and all those complex geometric features
down to a single, simple, application -specific
choice. Is that isolated solder joint cracked?
pass or fail. It really is an incredible funnel
of information. Billions of mathematical calculations
just to say yes or no. And once you perfect that
specific pipeline, acquiring, extracting, segmenting,
and deciding, you can deploy it literally anywhere.
Anywhere at all. And the scale of this is staggering
right now. For 2024, the leading areas of computer
vision were industry, with a massive market size
of $5 .22 billion. Medicine was at $2 .6 billion,
and the military sector was just shy of a billion
dollars. The real -world applications across
those sectors really show how adaptable that
five -step pipeline is. Take agriculture, for
example. Well, the farming applications are wild.
They really are. Engineers have actually developed
open source vision transformer models to help
farmers monitor crops in real time. OK, I want
to clarify vision transformers, because they
are a huge leap beyond the CNNs we talked about
earlier. A massive leap. Yeah. Older models look
at an image chunk by chunk, sliding that little
filter around like we said. But a vision transformer
looks at the entire image all at once. It's a
holistic view. Right. It uses an attention mechanism
to understand how a brown spot on the bottom
of a leaf relates to a yellowing edge on the
complete other side of the plant. And because
of that holistic view, this model can automatically
detect strawberry diseases from visual data with
98 .4 % accuracy. It's revolutionary for global
food security. And then in medicine, it's doing
very similar pattern recognition, but it's looking
inside us. Instead of a strawberry, it's a human.
Exactly. Systems can extract information from
ultrasonic or x -ray images to spot tumors, detect
arteriosclerosis, or even measure blood flow.
It's doing the grueling pixel -by -pixel analysis
that human eyes just get too fatigued to do reliably
over an eight -hour shift. And then you have
autonomous vehicles and space exploration. We're
talking about everything from the obstacle warning
systems in consumer cars using cameras and the
LiDAR, all the way to NASA's Curiosity rover
and the Chinese space agency's U -22 rover navigating
the moon and Mars. It's everywhere. So what does
this all mean? When you step back, you really
have to marvel at the universal language of light
and geometry. The exact same foundational pipeline
being used to inspect tiny strawberries in a
field in California is steering a robotic rover
through the radioactive dust of Mars. It is incredibly
versatile, and in those really high -stakes environments
like the military and autonomous driving, there
is a specific concept that becomes absolutely
critical, sensor fusion. Sensor fusion, meaning
more than one camera. Exactly. It is rarely just
one camera operating in isolation. A self -driving
car might use visible light cameras to read a
red stop sign, thermal imaging to spot a pedestrian
walking in the dark, and radar to calculate the
exact speed of an oncoming truck. That's a lot
of incoming data. It is. The system automatically
fuses all that disparate data to reduce the complexity
of a busy intersection. It has to make life or
death strategic decisions in real time based
on a synthesized view of the entire world. OK,
so we've seen how this technology spans from
agriculture to space. But there is one application
from the research that absolutely blew my mind.
We are actually using computer, quote unquote,
site. to replicate the human sense of touch.
Yes. Tactile feedback for robotic hands. This
is honestly one of the most cutting -edge and
frankly brilliant engineering workarounds I've
ever seen in the field. I have to explain this
mechanism to you because it is so clever. To
give a robotic finger a sense of touch, you would
normally assume they just, you know, stick a
pressure sensor on the outside of the finger.
Right? That's what we tried to do for years.
But those external sensors break, or they just
aren't sensitive enough for delicate work. So
instead, engineers suspend a tiny camera inside
a flexible hollow silicon dome. And this dome
acts like a squishy fingertip. Right. And embedded
evenly all throughout the inside of that silicon
dome are little painted point markers, just little
dots. And when that robotic finger presses against
a physical surface, say an egg or a soft piece
of fabric, the silicon naturally deforms and
squishes. Right. And the internal camera is just
sitting in the dark inside the finger, just watching
those point markers shift around in real time.
The computer then analyzes that visual shift,
the microscopic movement of the dots, to determine
the exact physical imperfections of the surface
it's touching. It's brilliant! It's literally
like looking at the shifting shadows on a blanket
to figure out exactly what kind of object is
hiding underneath it. That analogy is spot on.
Grasting objects effectively and instantly adjusting
grip strength based on the texture and slip of
an item has historically been one of the absolute
hardest mechanical challenges in robotics. It's
so hard for machines to pick things up gently.
Exactly. Traditional strain gauges on the outside
of a robotic hand are just too fragile. So by
translating the physical sensation of touch into
a visual processing problem inside a protected
silicon dome, engineers have completely bypassed
those hardware limitations. They used vision
to solve touch? They using vision algorithms
to solve a mechanical physics problem. It's beautiful.
It's just a stunning evolution. I mean, we started
this deep dive looking at a group of researchers
in 1966 who naively thought they could and solve
artificial sight as a quick summer project. And
look where we are now. Yeah, today we are looking
at this invisible. hyper complex matrix of algorithms
that can diagnose life -threatening diseases,
drive cars through rush hour traffic, and literally
allow robots to feel the textures of the world
using cameras hidden inside silicon fingertips.
It is a profound testament to decades of rigorous
interdisciplinary science. You have physics,
biology, mathematics, and engineering all converging
to teach a machine how to interpret our reality.
And it's really something to remember as you
go about your day. Every time you grants it your
phone and it unlocks using facial recognition,
every time you drive a modern car with lane assist,
or even every time you eat a piece of automatically
sorted perfectly right produce from the grocery
store, you're relying on this technology. You're
relying on that five -step pipeline. Exactly.
You are interacting with machines that are constantly
acquiring, extracting, and deciding based on
the visual world. But you know, as these systems
become more and more integrated into our daily
lives, there's a final detail from the research
regarding typical tasks that I really think we
should consider before we go. Oh, right. It brings
up the highly complex and somewhat controversial
topic of emotion recognition. Right. Using facial
recognition to not just identify who you are.
but how you're feeling. Exactly. Computer vision
systems are currently being trained to classify
human emotions based on the microscopic geometric
movements of our facial expressions. However,
the data explicitly points out that psychologists
strongly caution against this. Psychologists
are sounding the alarm. Yes. Decades of psychological
research suggest that internal subjective emotions
cannot be reliably detected from facial expressions
alone. A smile doesn't always mean happiness.
A furrowed brow doesn't always mean anger. The
human face is incredibly nuanced. I mean, we
mask our feelings all the time or express them
in ways that completely contradict our internal
state. Precisely. Which leaves us with a lingering,
somewhat provocative question to ponder. As computer
vision becomes perfectly ruthlessly precise at
mapping the geometry of our physical world down
to the exact millimeter of a smile, what happens
when we deploy that exact same geometric math
to judge our inner feelings? That's a heavy thought.
Are we genuinely creating a machine that truly
understands us, or are we just building a highly
efficient automated system for misinterpreting
human nature? Wow. It really brings us right
back to the beginning, doesn't it? We've spent
decades trying to teach a machine to translate
light into a 3D world. But when it comes to the
muddy waters of human emotion, the camera might
see everything, but the machine still might be
completely in the dark. Thanks for joining us
on this deep dive.