OK, let's unpack this, because I think it's something
we all do without even realizing it. Oh, absolutely.
Think about what happens when you wake up in
the morning. You open your eyes. and instantly,
effortlessly, you just see. Right, there's no
conscious effort at all. Exactly. You don't have
to consciously calculate the geometric distance
to your alarm clock or analyze the ambient lighting
conditions to figure out that the blurry shape
on the floor is your dog and not a pile of laundry.
You just instinctively know. Right. We take the
sheer computational power of our eyesight completely
for granted, but getting a machine you know,
a computer, to do that, to get a machine to open
a digital eye and actually comprehend the three
-dimensional scene in front of it, it turns out
that is one of the most complex, mind -bending
puzzles in the entire history of engineering.
It really is. Because computer vision isn't just
about taking a digital photograph. A digital
photo is just raw data. I mean, it's just a flat
two -dimensional grid of millions of pixels.
And the computer doesn't naturally know what
those pixels actually represent, right? Exactly.
To a computer, it's just numbers. Computer vision
is the extraction of high -dimensional data from
the real world to produce, well, numerical or
symbolic information. So giving it actual meaning.
Precisely. It is essentially the process of turning
a chaotic flood of light, shadow, and geometry
into a series of autonomous, logical decisions.
We are teaching machines not just to record the
world, but to genuinely understand it. And today
we are decoding that exact translation. We're
doing a deep dive into a massive stack of documentation,
pulling from decades of computer science history,
hardware specs, and algorithmic breakdowns to
really explore the mechanics of computer vision.
There is a lot to cover. There really is. And
for you listening, whether you're unlocking your
phone with your face, or driving past an autonomous
vehicle on the highway, or even just buying produce
that's been optically inspected on a conveyor
belt, this deep dive is your cheat sheet. Yeah,
it's everywhere now. We are ripping the lid off
the invi— technological eyes that are constantly
inspecting, navigating, and shaping your daily
life. And you know, to really grasp how sophisticated
this technology is today, we have to look back
at how completely we misunderstood the challenge
in the first place. Oh man, the hubris of the
1960s. It's hilarious in hindsight. The pioneer
scientists of artificial intelligence back then
just assumed that vision was the easy part of
intelligence. Yeah, back in 1966 at MIT, The
AI lab famously assigned computer vision to an
undergraduate student as a summer project. Just
a quick summer gig. They literally thought they
could attach a camera to a computer. and in a
few months just write a program that would have
the machine describe what it saw. They fell into
a classic cognitive trap, really. How so? Well,
because human vision operates consciously, right?
You don't feel your brain sweating when you look
at a coffee mug. Right. It just happens. Yeah.
So they assumed the computational load was incredibly
light. They thought the hard part of AI was the
reasoning, you know, playing chess, solving complex
math theorems. So they figured perception Just
translating a flat picture into a list of 3D
objects would be a simple stepping stone. Exactly.
It's as if we thought teaching a computer to
see was like teaching it to read a thermometer,
but it turned out to be like teaching it to understand
poetry. That is the perfect way to frame it.
And because it was like poetry, that one summer
project stretched into decades of grueling incremental
research. Just trying to get the basics down.
Yeah. Through the 1970s, researchers realized
they had to start with the absolute mathematical
basics. They couldn't just say, hey, find the
chair. Right. Because the computer doesn't know
what a chair is. Exactly. They had to write algorithms
strictly focused on extracting edges, just looking
for where light pixels suddenly turned into dark
pixels. Just to try and label lines and infer
basic 3D structures from 2D blocks. Right. And
then the 1980s arrive and the field gets flooded
with these intense mathematical models to handle
lighting and depth. Yeah, things with great names
like scale space. which was a mathematical framework
for allowing computers to recognize an object,
whether it was two feet away or 20 feet away,
or contour models, which were known as snakes.
I love that name. Snakes are a great example
of this era's logic, actually. How so? An active
contour model, or a snake, is essentially a digital
rubber band. You drop it onto an image, and the
algorithm shrinks that rubber band inward. Until
it hits something? Right, until it hits an area
of high gradient. a sharp change in contrast,
which usually signifies the edge of an object.
The rubber band wraps around that shape, helping
the computer trace a distinct boundary. That's
so clever. And it wasn't just finding edges,
right? By the 1990s, the field of computer vision
started colliding with computer graphics. Yes,
big time. They adopted a concept called bundle
adjustment, which comes from photogrammetry.
And photogrammetry is basically the science of
making 3D measurements from 2D photographs. So
like if a camera moves through a room taking
pictures, bundle adjustment is the incredibly
heavy math used to triangulate all those different
viewing angles to simultaneously estimate both
the 3D shape of the room and the exact path the
camera took. It was a slow mathematical grind
of geometry and physics for decades. But then
we hit the modern era. The boom. The boom. The
entire landscape shifted violently with the explosion
of deep learning and convolutional neural networks,
or CNNs. The famous CNNs. Yeah. Instead of humans
hand coding what an edge or shadow looks like,
we just started feeding millions of images into
these networks. Though it learns on its own.
Exactly. A convolution is basically a sliding
mathematical filter. The network slides this
filter over the pixels of an image layer by layer,
automatically learning to detect edges, then
textures, then shapes, and finally, whole objects.
And it worked really well. Unbelievably well.
Within a few years, these CNNs were surpassing
human benchmarks on massive datasets like ImageNet,
which contains millions of images sorted into
thousands of object classes. Okay, I want to
push back a little on this idea of modern AI
surpassing humans or having vision completely
solved. Fair enough. CNNs are incredibly powerful,
but there's this fascinating contradiction in
the technology that came up in the source material.
Yeah, the dog breeds versus the filters. Yes.
On one hand, a CNN can easily classify a dozen
fine -grain dog breeds or specific species of
birds with a level of accuracy that you or I
would totally fail at. Oh, I wouldn't even know
where to start with bird species. Right. But
on the other hand, feed that same supercomputer
an image with a modern digital camera filter,
like a basic Instagram color tint, and it gets
completely confused. It totally breaks down.
Or it will fail to detect tiny obvious details
like a small ant crawling on the stem of a flower
or a person holding a thin quill in their hand.
Humans never get tripped up by a color filter.
What's fascinating here is why that paradox exists.
You and I don't just see with our eyes. We see
with our entire life experience. Oh, that's a
good point. We have massive amounts of contextual,
organic knowledge about how the world operates.
If you see a photo tinted a slightly purple,
your brain instantly knows it's a lighting effect
or a filter, and you just discount it. You don't
suddenly think the dog is a different species.
Exactly. And if you see a hand positioned with
the fingers pinched together hovering over a
piece of paper, your brain actively expects to
see a pen or a quill, so you spot it immediately.
Even if it's super thin or blurry. Because computers
don't have that organic context. They aren't
expecting anything. Precisely. They are disentangling
symbolic information using models built entirely
on geometry and statistics. So it's all just
math? All math. When a digital filter subtly
distorts the numerical pixel values across an
entire image, the statistical model breaks down.
The computer doesn't conceptually know what a
filter is. It just sees bad numbers. Exactly.
It only knows that the mathematical patterns
it relies on to identify a dog have suddenly
vanished into statistical noise. Which really
points to the massive gap between humans and
machines. The reason computers get confused by
a simple filter is that they don't have biological
eyes or biological brains. To fix that, engineers
had to figure out how to build an artificial
eye and brain from scratch. And that requires
a literal Frankenstein recipe of scientific fields.
Here's where it gets really interesting, though,
because you literally need quantum mechanics
just to snap a digital picture properly. I know.
It sounds like an exaggeration, but it is absolute
reality. It blew my mind. Most computer vision
systems rely on solid -state image sensors, typically
CMOS sensors, that detect electromagnetic radiation.
And to design those sensors, you need a deep
understanding of solid -state physics. Wait,
hold on. Quantum mechanics? Just to take a digital
picture? Yeah. I thought cameras were just capturing
bouncing light through a glass lens. Where? Where
does the quantum realm actually come into play
here? It happens the moment the light hits the
digital sensor. When photons of light travel
through the camera lens, they strike a grid of
silicon pixels. Because of the photoelectric
effect, which is a quantum mechanical phenomenon,
those photons knock electrons loose within the
silicon. So the light is physically freeing electrons.
Exactly. The sensor collects these freed electrons
in microscopic wells, and the number of electrons
in each well tells the computer exactly how bright
the light was at that specific pixel. That is
wild. You literally cannot engineer a modern,
high -definition camera sensor without relying
on the quantum interactions between light and
matter. Okay, so quantum physics gives us the
raw hardware, you know, the artificial eye. But
what about the brain? How do we get the software
to interpret those electron wells? That's where
we turn to biology. Right. Over the last century,
scientists extensively studied neurobiology,
like human eyes, neurons, and the visual cortex,
just to create a blueprint for machines. And
there is a massive historical anchor from the
1970s, Kunihiko Fukushima's neocognitron. A legendary
piece of work. Yeah, it was an early neural network
directly inspired by biological vision, specifically
how simple and complex cells in an animal's visual
cortex respond to lines of light at specific
angles. But if we are literally trying to copy
the human brain, are we just building a digital
clone of a human, or is a computer vision system
doing something fundamentally different? It is
doing something fundamentally different. Neurobiology
provides the architectural inspiration, but the
execution is purely mathematical. So it's not
a true brain? Not at all. Biology relies on physiological
processes, like chemical synapses between living
neurons. Computer vision relies on nodes with
mathematical weight associations. Can you give
an example? Sure. Let's look at how a network
learns to recognize textures. Imagine training
a network to recognize various sea creatures.
Like a starfish? Yeah. The starfish might strongly
activate visual feature nodes for a ringed texture
and a star outline. Meanwhile, sea urchins might
strongly activate nodes for a striped texture.
and an oval shape. But nature isn't always that
clean. What happens if you feed the system an
image of a rare sea urchin that actually has
a ringed texture instead of stripes? Well, the
network adjusts the math. It creates a weekly
weighted association between the urchin category
and the ringed texture node. Oh, I see. So the
next time it sees an image, it isn't just checking
off a rigid list of features. It is calculating
massive matrices of associated mathematical weight
patterns across millions of nodes simultaneously.
Biology gave us the blueprint of a layered network.
But the computer executes this through linear
algebra and multi -dimensional signal processing.
Exactly. So we have physical sensors designed
with quantum mechanics and algorithms inspired
by neurobiology, but executed through heavy math.
How does a computer actually process a live scene
in real time? It follows a very strict pipeline.
Right. Let's walk through the actual assembly
line of artificial perception. The typical pipeline
of a computer vision system is incredibly structured.
It almost always starts with image acquisition.
Getting the raw data. Right. And this isn't just
snapping a 2D photo. This could be gathering
3D volume scans from a medical device or pulling
sonic data from radar and ultrasonic cameras.
And you can't just feed that raw data straight
into the brain, right? It has to go through preprocessing.
It has to be cleaned. Yeah. The system has to
clean the data, reducing noise, enhancing the
contrast, or resampling the image so the coordinate
system is perfectly aligned. And from there,
you move into feature extraction. And this is
where it gets really granular. It's less like
looking at a whole picture and more like looking
through a tiny microscopic straw. That's a great
analogy. The computer slides that straw across
an image, and it is completely blind to what
the object actually is. It only notices when
the pixels inside that straw suddenly shift from
light to dark or when a mathematical gradient
spikes. And that sudden drop off, the computer
logs that as an edge or a corner. Exactly. And
once it has mapped all those edges and corners,
the pipeline moves to detection and segmentation.
Putting the pieces together. The system decides
which of those extracted regions are actually
relevant. It segments the image, mathematically
isolating the foreground object from the background
noise. Finally, it moves to high -level processing
and decision making. Right, like, is this segmented
shape a car drifting into my lane? Or does this
manufactured circuit board pass quality inspection?
It's like preparing a gourmet meal. Acquisition
is buying the groceries, pre -processing is washing
the vegetables, feature extraction is chopping
them into exact shapes, and decision -making
is the final taste test. I love that. But what
happens when the groceries are spoiled? Say the
camera lens is smudged, or there's heavy motion
blur from a car moving fast, or bad lighting
introduces a ton of static to the image. It happens
all the time in the real world. A human can squint
and still figure out what they're looking at.
Does the computer just throw the degraded image
away and give up? If we connect this to the bigger
picture, it makes sense how they handle it. This
is where a subfield called image restoration
becomes absolutely critical. So they fix it.
Yeah. Computers don't just discard degraded data.
They actively reconstruct it. They use highly
sophisticated local structural models to separate
the noise from the actual signal. How does that
work? Take a median filter, for example. If you
have a corrupted pixel, say it's glaringly bright
white because of sensor noise, the algorithm
looks at a three by three grid of the surrounding
pixels. Just a tiny neighborhood around the bad
one. Right. It takes all nine of those numerical
values, lines them up, and finds the exact median
value. It throws out the extreme white noise
and replaces the corrupted pixel with that median
number, instantly smoothing out the image without
blurring the real edges. It's literally guessing
what should be there based on the neighborhood.
Yes. They even use techniques called in -painting
to fill in entirely missing parts of an image.
Just making up missing data. Kind of. If there
is transmission interference or severe motion
blur, the system analyzes the existing local
image structures, the lines and edges that are
still visible, and uses multi -dimensional math
to control the filtering process and rebuild
the image to what it was intended to be. To manage
this entire assembly line, the system relies
on image understanding systems. or IUS. This
framework breaks perception into three levels
of abstraction. The low level is just the raw
primitives, those edges and textures we talked
about. The intermediate level builds those primitives
into continuous boundaries, surfaces, and 3D
volumes. Getting closer to a real object. Yeah,
and the high level is where the math finally
clicks into a semantic concept, an object, a
scene, or a specific event. And we cannot ignore
the physical hardware required to run this intensely
heavy pipeline. Oh, absolutely. Traditional CPUs
simply cannot handle the sheer volume of parallel
math required to calculate millions of pixel
values 60 times a second. We are now seeing dedicated
vision processing units, or VPUs, integrated
directly into devices. So we've built this insanely
complex, noise -filtering, quantum -powered,
biologically -inspired deep learning pipeline.
That's a mouthful. Right. So what is it actually
doing out there in the world right now? Because
the sheer scale of its application is staggering.
The market sizes alone tell the story of its
dominance. In 2024, the leading sector for computer
vision was industrial machine vision with a market
size of 5 .22 billion U .S. dollars. The medical
sector reached 2 .6 billion and military applications
were nearly a billion dollars. The specific use
cases are where it gets truly wild. Take agriculture.
Oh, yeah. There is an open source vision transformer
model out there right now that farmers use to
monitor their fields. It automatically analyzes
the visual data of crops and detects diseases
on strawberry plants with a ninety eight point
four percent accuracy rate. That is incredible.
It is literally securing food supply chains or
look at industrial manufacturing. They deploy
high speed vision processing setups that can
track. thousands of frames per second. Thousands.
Yes. They are inspecting speeding glass bottles
on a production line, instantly flagging microscopic
cracks. They use it to check silicon wafers,
the foundational pieces of our computer chips,
for defects that are entirely invisible to the
human eye. That's wild. They even use optical
sorting to automatically blast undesirable foodstuff
off of bulk conveyor belts using targeted jets
of air. Then you have autonomous vehicles. And
I'm not just talking about the driver assist
features in your car. I'm talking about space
exploration. Yes. NASA's Curiosity rover and
the Chinese space agency's U -22 rover use computer
vision algorithms for SLAM. That's simultaneous
localization and mapping. Right. They ingest
camera data to simultaneously build a 3D map
of the Martian or lunar surface while pinpointing
their exact location within that map. It's vital
for navigation. Or unmanned aerial vehicles,
UAVs, flying high over dense forests, analyzing
visual spectrums to detect the earliest thermal
signs of wildfires before human towers can spot
them. But perhaps the most mind -bending application
emerging right now is how computer vision is
being used for tactile feedback. Yes. This part
completely broke my brain. The idea that vision
is being used to create the sensation of touch.
It's brilliant. Engineers are now embedding rubber
artificial skin layers with tiny strain gauges.
You place the skin over a robotic finger, trace
a surface, and the computer measures the upward
push on tiny rubber pins to map. micro -indulations.
But the far crazier version of this is the silicon
domes. Right. Engineers take a tiny, high -definition
camera and suspend it inside a flexible, translucent
silicon dome like a robotic fingertip. They embed
a grid of equally spaced point markers on the
inside of that silicon. When that robotic fingertip
presses against an object, the silicon physically
deforms. So the camera's watching from the inside.
Exactly. The internal camera watches how those
point markers shift, stretch, and stored. The
computer then uses that visual data to calculate
the exact geometric pressure being applied to
the mold. It's using a camera to watch the inside
of its own skin deform, which lets the robotic
hand essentially feel the shape and firmness
of whatever it's grasping. It's incredible. So
what does this all mean? I have to push back
on our core premise here. We call this whole
field computer vision. But if a camera suspended
inside a silicon dome is being used to help a
robot feel a surface, Hasn't vision just crossed
over into a completely different sensory category?
This raises an important question though about
the fundamental trajectory of the technology.
Machine vision is rapidly converging with robotics
and visual computing. So it's merging. It is
evolving away from a replication of human eyesight
and turning into a generalized environmental
awareness. The algorithms don't actually care
if the input data represents visible light, physical
pressure or radar waves. It just processes high
dimensional data into autonomous decisions. Just
numbers to them. Exactly. You see this vividly
in modern military applications. The latest missile
guidance systems don't just lock on to a pre
-programmed set of coordinates. They don't. No,
they are launched into a general theater of operation
and then they use locally acquired image data
to actively deliberate and select their targets
upon arrival. Vision has become synonymous with
autonomous thinking. That is both awe -inspiring
and frankly terrifying. It really highlights
the incredible trajectory we've unpacked today.
We've come a long way from a summer project.
We really have. We started with a naive undergraduate
summer project in 1966, thinking a computer could
just look at a camera feed and print out the
list of objects. And today we have a multi -billion
dollar global infrastructure. We're leveraging
the quantum mechanics of the photoelectric effect
and the biological blueprints of the visual cortex
to spot six strawberries, to navigate rovers
across the Martian landscape, and to give robotic
hands the physical ability to feel. And if there's
one lingering thought I'd leave you with, it's
this. Computer vision is no longer limited to
the visible light spectrum. What do you mean?
Well, these processing pipelines are ingesting
multi -dimensional data from medical CT scanners,
hyperspectral imagers, and ground penetrating
radar. Oh, wow. They are quite literally seeing
in wavelengths and dimensions that the biological
human eye cannot physically comprehend. We have
spent 60 years trying to teach machines to mimic
our limited understanding of the world. Right.
But are we rapidly approaching a point where
machines won't just mimic us, but will eventually
uncover, map, and act upon a version of reality
that is completely invisible to us? Wow. A reality
right in front of us that we just don't have
the hardware to perceive. It brings us right
back to that alarm clock in the morning. Exactly.
We open our eyes, and we naturally assume the
world we see is the only world that exists. We
take the puzzle of perception completely for
granted. But the machines they're seeing much
more They are just opening their eyes and what
they are starting to see is vastly bigger than
we ever imagined Thanks for joining us on this
journey. Keep questioning what you see and we'll
catch you on the next deep dive