Imagine walking into like a massive sold out
stadium. Oh, yeah, like 80 ,000 people. Exactly.
100 ,000 people all shouting at the exact same
time. It's just this this physical wall of white
noise. Right. You can't hear anything. You can't
make out a single conversation, let alone, you
know, a single word. It is completely overwhelming.
But what if you had this magical dial on your
headphones? OK, I like where this is going. You
turn it one click. and suddenly that chaotic
background noise just drops. You just hear the
collective, like, underlying mood of the crowd.
And then you turn it one more click, and you
clearly hear the three main chants echoing around
the arena. You've somehow extracted the actual
meaning hidden inside all that screaming. And
the crazy thing is that dial actually exists
mathematically. Right. I mean, you're basically
filtering out the statistical noise to find the
true underlying structure. It's it's what data
scientists call finding the signal. And that
is exactly what we are doing today, because,
look, you the listener, we all live in an era
of just absolute information overload. Oh, totally.
We're drowning in it. You are bombarded with
massive data sets every single day, whether you
realize it or not. And the tool that acts like
that magical headphone dial It's called Principal
Component Analysis, or PCA. Yeah, PCA. It's this
120 -year -old mathematical engine that secretly
dictates how your intelligence is measured, how
your country is ranked globally, even how algorithms
know what you're going to buy. Yeah, it's everywhere.
But it has a massive blind spot. So today, we're
taking a deep dive into a stack of research,
including a really comprehensive overview of
PCA. to understand how modern science makes sense
of the chaos. And critically, where that math
goes terribly wrong. Right. Because it does go
wrong. It really does. But, you know, it is the
hidden architecture behind so much of our world.
When you have a data set with hundreds or, I
don't know, thousands of variables, human brains
just shut down. Yeah. I mean, I can barely handle
a spreadsheet with 20 columns. Exactly. We can't
visualize a thousand dimensions. So PCA is the
mathematical technique that takes that massive
high -dimensional data and basically translates
it into a lower dimensional space. While keeping
the most important info intact, right? Right.
You don't want to lose the actual signal. See,
jumping straight into high -dimensional mathematical
spaces is usually where my brain just blue -screened.
Fair enough, yeah. Before we get into the heavy
mechanics of how a computer actually calculates
this stuff, we really need to visualize what
dimensionality reduction looks like in physical
space. Well, the source material offers a really
great geometric way to think about this. It describes
PCA as fitting a P -dimensional ellipsoid to
the data. An ellipsoid, so like a 3D oval, like
a rugby ball or a zeppelin or something? Yeah,
think of a swarm of bees. If you track their
positions, they might form this giant fuzzy cloud
in the shape of that rugby ball. Right, they're
swarming around the queen or something. Exactly.
And each axis of that ellipsoid represents what
we call a principal component. Now, what's fascinating
here is that if one axis of that rugby ball is
really short or narrow, it means the variance,
like the actual spread of the data along that
direction is super small. Because the bees just
aren't moving much in that specific direction.
Right. So you can just drop that axis entirely
to simplify the picture. You just delete it.
You basically just ignore it. You reduce the
dimensions, but you retain the axes where the
swarm is spread out the widest. Because that's
where the most important information, or movement,
actually lives. Exactly. Okay, let's unpack this
with another analogy. Because when I was reading
the source, I was thinking of it like casting
a shadow of a complex three -dimensional object
onto a flat 2D wall. Oh, that's a really good
way to picture it. Like if you have a physical
teapot. Right. and you shine a flashlight on
it straight from the front, the shadow on the
wall might just look like a solid boring circle.
Yeah, you wouldn't even know it's a teapot. Exactly.
You've successfully reduced the dimensions from
3D to TD, but you've chosen a terrible angle.
You lose all the information about what the object
actually is. The essence of the object is completely
lost in that specific projection. But... If you
just rotate that teapot until the spout and the
handle are perfectly silhouetted against the
wall, you instantly recognize it. Right. You
are rotating the object to find the angle that
gives you the maximum spread, the most recognizable
silhouette. In PCA terms, you're finding the
plane with the maximum variance. Yeah, and the
algorithm does exactly that, but sequentially.
Step by step. Right. So the first principal component,
that first axis, is a derived variable that explains
the absolute most variance in the data. It's
the longest axis of our rugby ball. Exactly.
It's the best possible silhouette. Then the algorithm
looks for the second principal component. This
one explains the next most variance, but with
a strict geometric rule. It has to be completely
orthogonal. Orthogonal meaning? At a perfect
right angle to the first one. Ah, okay. So it
has to be completely uncorrelated to the first
component. Right. It's capturing brand new information
that the first one missed without any overlapping.
And it proceeds like that, step by step, finding
new orthogonal directions until all the variance
in your dataset is totally explained. So it takes
a messy dataset where everything is highly correlated
and tangled up. And transforms it into a neat,
independent set of principal components. Okay,
so visualizing rotating a teapot is one thing.
But how does a computer actually rotate a spreadsheet
with, like, a thousand columns? Yeah, the math
part. The source mentions it operates using covariance
matrices and eigenvectors. Yes. Let's translate
that into plain English. How does the math actually
find this maximum variance? Well, let's build
one together. Imagine a data set with just two
variables, height and shoe size. Okay, pretty
standard. If you plot 100 people on a graph with
height on the vertical axis and shoe size on
the horizontal axis, the dots don't just form
a random circle. No, they'd form like a diagonal
line. Right. A diagonal cigar -shaped swarm pointing
up and to the right. Because taller people naturally
tend to have larger feet. The two variables are
moving together. Exactly. And that relationship
is what a covariance matrix captures. It's basically
a mathematical grid that calculates exactly how
much every single variable in your data set changes
in tandem with every other variable. Got it.
And once you have that grid of relationships,
the math calculates the eigenvectors. OK. The
scary word. What is an eigenvector actually doing
to our cigar shaped swarm of dots? Honestly,
an eigenvector is essentially just the computer
drawing a brand new line straight through the
longest part of that cigar shape. Oh, that's
it. That's it. Instead of sticking with the original
height and shoe size axis, PCA creates a new
diagonal axis that captures both. So this new
line. is our first principal component. Yes.
It represents a hidden latent concept, like maybe
we just call it overall size. OK, that makes
sense. And the eigenvalue is just the number
that tells you exactly how much the total variance
is captured by that new line. OK, that feels
like pure elegant logic. Yeah. The math just
finds the natural variance. It does. But the
source material points out a massive trap here
that kind of completely shatters that illusion
of mathematical objectivity. Ah yes, the scaling
trap. The scaling trap. It turns out PCA is incredibly
sensitive to how human beings format the numbers
before the algorithm even boots up. If we connect
this to the bigger picture, the scaling trap
reveals how fragile this kind of data analysis
can actually be. The source explicitly notes
that if you multiply just one variable in your
data set by 100... Just one column. Right, just
one column. The first principal component will
almost entirely align with that single variable.
Because the algorithm is blindly hunting for
variance. Yeah. Like if you artificially inflate
a number, the algorithm acts like a dog seen
as quarrel. Yeah, that's exactly it. It just
chases the biggest number and ignores the actual
relationships. The text gives... a brilliant,
simple illustration of this involving temperature.
Imagine you're analyzing a data set that includes,
say, temperature and humidity. If you record
that temperature in Fahrenheit, the numbers are
going to be larger and have a wider numeric spread
than if you record the exact same physical temperatures
in Celsius. Right, because the gap between freezing
and boiling in Fahrenheit is 180 degrees, but
in Celsius, it's only 100. Exactly. It's the
exact same physical heat just written with bigger
digits. Right. So if you run PCA on the Fahrenheit
data set, you will get a completely different
principal component than if you run it on the
Celsius data. Wow. The algorithm will mathematically
conclude that temperature is a much more dominant
important factor in the Fahrenheit data set purely
because the digits themselves are larger. That
is wild. If your variables have different units
like mixing temperature in degrees, mass in kilograms,
distance in millimeters, the CA becomes completely
arbitrary. The variable with the largest absolute
numbers just dominates the shadow on the wall.
Completely. See, we tend to treat algorithms
as these unbiased arbiters of truth. You know,
if a computer spits out a graph, we just assume
it's objective science. Right, we trust the machine.
But if the human running the numbers doesn't
perfectly standardize the data beforehand, The
truth the algorithm finds is totally warped.
Like, a casual choice between Celsius and Fahrenheit
completely changes what the computer tells you
is the fundamental structure of your data. Which
is why that human prep work is mandatory. You
have to mean center the data. Mean centering.
Meaning what, exactly? It means subtracting the
average so everything is centered on zero. And
you also have to standardize the unit so every
variable starts with equal variance. So you level
the playing field before the computer looks at
it. Exactly. Without that human intervention,
the math is entirely blind. It's honestly terrifying
that a simple unit choice can warp the data that
badly. Because given how fragile and sensitive
this algorithm is to human input, you'd think
we'd only use it in highly controlled lab experiments,
right? I would think so. But we actually use
it to define incredibly messy, subjective human
concepts. The source outlines applications that
directly impact almost every aspect of society.
Yeah, because PCA handles chaotic, multivariable
environments so well, it really excels anywhere
humans are trying to pin down abstract ideas.
Take psychology, for example. OK, let's talk
about psychology. In the early 1900s, researchers
hypothesized that human intelligence wasn't just
a single metric. It was a complex mix of spatial
reasoning, verbal skills, mathematical deduction,
memory. But you can't put a physical ruler up
to someone's brain and measure deduction. Right.
You can only give them tests. Right. So in 1904,
Charles Spearman developed factor analysis, which
is mathematically intertwined with PCA, to find
the underlying components of intelligence. 1904.
Yeah, and by 1924, a psychologist named Thurstone
was using it to look for 56 distinct factors.
56. Right. And if a person scored highly on a
vocabulary test, a reading comprehension test,
and a verbal analogy test, the math would recognize
that those variables all moved in tandem. Just
like heightened shoe size. Exactly. PCA draws
a new axis through that variance and calls that
hidden latent variable verbal intelligence. And
this mathematical transformation is literally
the bedrock of the standard IQ tests we still
use today. We are using a mathematical concept
from 1901 to define what it means to be smart.
Pretty much, yeah. And we do the exact same thing
in global economics, which is crazy to me. The
United Nations uses the human development index,
the HDI, to rank countries. Right. But how do
you mathematically rank development? A spreadsheet
for a country has columns for life expectancy,
education levels, per capita income, infrastructure.
It's a huge mix of data. You cannot add a year
of life expectancy to a dollar of GDP. It's comparing
apples to tractors. Well, by applying PCA to
all those diverse indicators, you extract the
principal components. The text notes that the
city development index actually started with
about 200 indicators from 254 global cities 200
columns of noise exactly 200 columns of noise
but PCI boiled that massive complexity down it
found that the coefficients roughly correlated
to the average costs of providing services oh
wow yeah creating a single powerful metric for
effective physical and social investment it found
the core signal of development hiding inside
millions of disparate data points here's where
it gets really interesting though It gets even
more granular than that. It's used heavily in
market research. Oh, absolutely. The source brought
up the 2013 Oxford Internet Survey. They asked
2 ,000 people a massive battery of questions
about their attitudes and beliefs regarding the
internet. I can only imagine the data there.
Right. Questions like, do you use the internet
to avoid thinking about your problems? Or do
you use it to find romantic partners? Do you
use it to pay bills faster? A giant spreadsheet
of thousands of survey answers is essentially
that stadium of people shouting, you just can't
read it. So the researchers ran PCA on that exact
survey data. Yep. The math looked at how all
those answers covariate. It found that people
who answered yes to avoiding work also tended
to answer yes to playing online games. Makes
sense. It clustered these behaviors and extracted
four principal dimensions that completely summarize
our relationship with the Internet. Four dimensions
out of all those questions. Just four. The text
lists them as escape, social networking, efficiency,
and problem creating. You feed the machine thousands
of arbitrary survey bubbles, and the math draws
four new axes. It reveals that human beings fundamentally
use the internet to escape reality, talk to friends,
work faster, or cause trouble. It's amazing.
It literally found the psychological silhouettes
of our digital lives. It did. And if we move
from psychology to pure biology, PCA is also
heavily utilized in neuroscience. Neuroscience.
Wait, so we're going from ranking national infrastructure
to mapping physical brains. Yes. When neuroscientists
record brain activity, a single electrode might
pick up the electrical action potentials, the
microscopic spikes from several different neurons
firing all at once. So it's literally the stadium
analogy, but inside your head. Yes, exactly.
It's a jumble of electrical noise. So neuroscientists
use a technique called spike -triggered covariance,
which relies heavily on PCA to sort those spikes
out. Okay, how did that work? They reduce the
dimensionality of the waveforms to figure out
exactly which specific properties of a visual
or auditory stimulus are causing a specific individual
neuron to fire. I am just marveling at the versatility
here. Carl Pearson invents a mathematical formula
in 1901 to solve abstract mechanics problems.
Right. And today... That exact same formula maps
firing neurons, ranks the development of nations,
measures your IQ, and figures out why we're addicted
to social media. It is an omnipresent lens. It
is a remarkable lens, but... And this is crucial,
lenses can distort reality just as easily as
they focus it. The source material is very clear
about the limitations of PCA. When this mathematical
tool is misused, or its fundamental assumptions
are ignored, it causes severe real -world friction.
OK, let's talk about when the magic dial on the
headphones breaks. Because if this math governs
so much of our world, we really need to know
where it fails. Well, the most fundamental flaw
is that PCA relies entirely on a linear model.
Linear meaning straight lines. Right. It assumes
the data behaves in straight lines, like our
cigar -shaped scatterplot from earlier. But the
text warns that if a data set has a hidden nonlinear
pattern, say, the data points form a complex
curve, or a U -shape, or a spiral. What's PCA
do, then? It won't just fail to find the pattern.
It will draw a straight line right through the
middle of the spiral and steer your analysis
completely backward. You'll draw entirely the
wrong conclusions. Yikes. And there are also
specific scientific fields where the basic requirements
of PCA literally break the laws of physics. Yes.
Astronomy is a fantastic example from the text.
We talked earlier about how you absolutely have
to mean center your data, right? Where you subtract
the average so everything centers around zero.
Right. That is mandatory prep work. Or the eigenvector.
will just point toward the center of the data
instead of the actual variance. But in astrophysics,
you are measuring light. You are analyzing signals
from stars and galaxies. And astrophysical signals
are strictly non -negative. Because you cannot
have negative light. Exactly. But when astronomers
run PCA and do that mandatory mean removal process,
the mass subtracts the average from everything.
So if a star emits five units of light, and the
average is 10, the math says that star now emits
negative five units of light. It creates what
the text calls unphysical negative fluxes. The
algorithm basically invents a scenario that cannot
exist in reality. just to satisfy its own equations.
This raises an important question, though. I
mean, the source notes that astronomers often
have to abandon PCA entirely and use a different
method called non -negative matrix factorization.
NMF, right. Right, NMF. Precisely because it
respects the physical boundary that you can't
have less than zero light. But think about that.
If a rigid algorithm forces astrophysicists to
literally invent negative light, we have to ask
what it is doing to data in the saucer sciences,
where the boundaries of reality aren't as clear.
That is a very good point. The source highlights
a major controversy in population genetics that
illustrates this exact danger. Oh, the Erinilike
paper. Yes. This part of the source was fascinating.
So in August 2022, A molecular biologist named
Aaron Elheik published a paper analyzing 12 different
PCA applications in population genetics. For
context for you listening, PCA is constantly
used to map genetic variation and trace ancient
human migrations. Right, you take thousands of
genetic markers, run PCA, and it clusters people
together to show who migrated where. But Elheik
analyzed these highly cited studies, and according
to the source, he called the results erroneous,
contradictory, and absurd. Wow. He didn't hold
back. No, he didn't. He argued that the results
achieved in these genetic studies were characterized
by cherry picking and circular reasoning. Which
goes back to the prep work, right? Exactly. Because
PCA is so incredibly sensitive to how the data
is scaled and which variables are included, it's
very easy to manipulate the method. If a researcher
wants to prove that two ancient populations are
distinct, they can pre -select the specific genetic
markers that happen to vary between those groups,
run PCA, and the algorithm will obediently draw
a massive line dividing them. So is PCA just
being used as a math -washing tool here? Math
-washing. That's a great term. I mean, researchers
can use this incredibly complex, dense math to
make their cherry -picked subjective data look
like rigorously objective science. To a layperson
or even a peer reviewer, if you say the principal
component analysis proves these groups are genetically
distinct, it just sounds unassailable. It sounds
like an absolute fact. You assume the computer
did the work, not the biased human who prepped
the data. The math itself isn't lying, but the
human choices made before the math dictates the
outcome. And the source also mentions researchers
at Kansas State University who found another
flaw. Oh, the autocorrelation thing. Yeah, they
found that PCA can be seriously biased by sampling
errors if something called autocorrelation isn't
handled correctly. How does autocorrelation bias
the results? So autocorrelation is when data
points influence each other sequentially. Think
of taking a temperature reading every single
minute. The temperature at 1 .00 p .m. and 1
.01 p .m. are going to be basically identical.
Right, it doesn't change that fast. They aren't
independent variables. They are correlated with
themselves over time. If you feed that into PCA
without correcting for it, the algorithm sees
all those identical minute -by -minute readings
as a massive block of variance and just skews
the principal components toward it. It thinks
it found a massive signal, but it's really just
the same data repeating. Exactly. It's a reminder
to always question the assumptions baked into
the data models we encounter in the news. We've
gone from casting shadows of teapots to measuring
IQs to negative starlight to... It's been quite
a journey. So what does this all mean? For you
listening, think of PCA as a mental model. It
is a lens for focusing on what truly matters,
finding the axis of maximum variance in a sea
of statistical noise. Yeah. In your own life,
when you are overwhelmed with information at
work or in the news, try to ask yourself, what
is the principal component here? What is the
one underlying variable that is actually driving
all this noise? It's honestly a brilliant shortcut
to reducing the overwhelm. It's an incredibly
powerful framework for navigating the modern
world. But, you know, there is one final really
provocative concept from the source material
regarding PCA and information theory. Oh yes,
referencing Linsker's work. Exactly, referencing
a researcher named Linsker. The underlying math
proves something profound. Dimensionality reduction
always results in a loss of information. Always.
It only minimizes that loss under very specific,
perfectly bell -shaped data distributions, what
statisticians call Gaussian noise models. If
your data isn't perfectly symmetrical Gaussian
noise whenever you reduce dimensions, something
real is permanently destroyed. You cannot cast
a 2D shadow without permanently losing the 3D
reality of the object. The shadow is simpler
to understand, but it is fundamentally less real.
That's beautifully put. So here is a final thought
to mull over. If we spend our entire lives constantly
trying to use mental PCA, you know, trying to
distill the overwhelming chaotic data of our
daily experiences, our relationships, our politics
into simple digestible principal components.
What vital, quiet nuances are we permanently
filtering out? By turning down the dial on the
stadium noise to only hear the loudest, most
dominant chants, what beautiful complex whispered
conversations are we permanently losing as mere
noise? Sometimes the noise isn't just noise.
Sometimes it's the whole point. Until next time,
keep diving deep.