Imagine holding an apple in your hand, like right
now. You can instantly see its height, its width,
its depth. Great. You can rotate it. Exactly.
You just turn it around, judge its weight, and
understand its physical shape in the real world.
And your brain, it evolved to do this flawlessly.
Yeah, it's highly specialized for three -dimensional
space. But what if I asked you to visualize an
apple that exists in, like, a thousand dimensions
at the exact same time. You, I mean, you physically
cannot do it. Right. The human brain just hits
a biological brick wall there. We're completely
trapped in our default operating system of, you
know, up, down, left, right, forward, and backward.
And that is a massive problem when you step into
the world of modern data. Because today, we're
generating datasets where a single object doesn't
just have three properties. It has hundreds or
even thousands of variables attached to it all
at once. It really is the ultimate cognitive
bottleneck of our era because we've built these
machines that can easily collect and process
thousands of dimensions of information, but we
can't see it. Exactly. If human beings cannot
actually see those relationships, we can't intuitively
understand them. So we're just left staring at
these giant spreadsheets of numbers. completely
blind to the hidden structures inside, we desperately
need a bridge between that mathematical complexity
of the data and, well, the visual limitations
of our own minds. Which brings us to the mission
of our deep dive today. We are looking at a really
dense, highly technical set of source materials
detailing a statistical method with, frankly,
a very intimidating name. Oh, yeah. It's called
t -distributed stochastic neighbor embedding.
Or, thankfully, T -S -N -E for short. Much better.
Yeah, way better. So our goal for you, the listener,
is to decode this complex math. We're going to
figure out how it acts as this massive dimensional
translator, taking those impossible layers of
data and just crushing them down into two or
three dimensional maps that you and I can actually
look at and comprehend. OK, let's unpack this.
To truly understand how T -S -N -E pulls off
this translation, we should probably look at
its foundation. The algorithm builds on something
called stochastic neighbor embedding, which was
originally developed by Jeffrey Hinton and Sam
Rolwis. And then later, Lawrence Vandermaten,
working alongside Hinton, proposed the T distributed
variant that, you know, really became the industry
standard. In the formal language of the field,
it is a nonlinear dimensionality reduction technique.
Wait, I have to stop you there because nonlinear
dimensionality reduction is exactly the kind
of phrase that makes people like immediately
turn off their brains. Fair enough. It's a mouthful.
Yeah. Let me try an analogy here to ground this
for the listener. So imagine you are tasked with
organizing a massive party. OK. Let's say there
are a million guests crammed into a giant convention
center. And your job is to group them together
on a flat, two -dimensional map of the room.
A seating chart from hell. Right. Exactly. But
you aren't just grouping them by their age or
their hometown. Right. You have to group them
by, say, 500 different personality traits simultaneously.
And that is your high dimensional space. Each
individual guest at the party is a data point.
And those 500 distinct traits, those are the
500 dimensions. Right. But here's where my brain
completely breaks trying to draw that map. Like
if I put all the heavy metal fans in one corner,
that's great. Sure. But what if Half of those
heavy metal fans are also obsessed with baking
sourdough bread, and the other half are amateur
botanists. Yeah, they get split up. Exactly.
And what if the botanists share sleep habits
with the classical music fans who are all the
way on the other side of the room? You can't
just squish 500 overlapping interests onto a
flat piece of paper without tearing those relationships
apart. You've just hit on the exact mathematical
problem TSNE is designed to solve. Oh, wow. It
acts as the ultimate party planner for this impossible
scenario. What's fascinating here is that the
algorithm completely abandons the idea of fixed
rigid geometry. What do you mean by rigid geometry?
Like, instead of trying to use a ruler to measure
the distance between these people's traits, it
uses probability. It asks a much more fluid question.
It says, based on these 500 traits, what are
the odds that these two specific people would
find each other in this massive crowd and start
talking? Oh, OK. So it trades a ruler for a set
of odds. Exactly. How does that actually play
out in the math, though? The source material
breaks the process down into two main stages,
starting with that high dimensional space. Yeah.
So first, TSNE looks at the original data of
those 500 trait party guests. And it computes
probabilities that are proportional to how similar
the objects are. OK. To do this, it centers a
Gaussian distribution, which you can just picture
as a classic bell curve. Right, a standard bell
curve. Over every single data point. The algorithm
essentially says, if I were to randomly pick
a neighbor for the specific point based on the
shape of this bell curve, what is the conditional
probability I would pick this other specific
point? So objects that share a lot of traits
get a very high probability of being neighbors.
objects with nothing in common get a microscopic
probability. It sounds like it's building this
massive invisible web of percentages in the high
dimensional space. Like everyone has a specific
percentage chance of standing next to everyone
else. That is stage one completed. Now, the algorithm
has to move to stage two. It takes a blank, flat,
two -dimensional map and scatters new, low -dimensional
points onto it. Just randomly. Just scatters
them out. The ultimate goal is to move those
flat points around until the web of probabilities
on the flat map matches the web of probabilities
in the 500 -dimensional space as closely as physically
possible. Okay. Now, to connect those two stages,
the text introduces a mechanism called minimizing
the Kohlbach -Leibler divergence. or KL divergence
using gradient descent. Yes. Let me see if I
can visualize this one. Is KL divergence basically
acting like a strict referee? A referee. Yeah,
like the referee is constantly watching the flat
map and comparing it to the high dimensional
space. And if the map puts two people next to
each other who actually have nothing in common,
the referee throws a flag and assigns a penalty
score. I like that. And gradient descent is just
the mathematical process of tweaking the map
over and over to minimize the referee's penalties.
The referee analogy for KL divergence is spot
-on. I mean, technically it is an asymmetric
measure of how one probability distribution diverges
from a second expected distribution, but... Right.
Referee is easier. But let's upgrade your idea
of gradient descent. It's often described as
taking tiny steps down a mathematical hill, but
in the context of TSNE... it's much more visceral
than that. Imagine the points on your flat map
are all connected by a complex system of physical
springs. Springs, like actual coiled tension
springs. Exactly. When the KL divergence referee
throws a flag because two points are placed poorly,
it's like adding massive tension to the springs
connecting them. So gradient descent is the process
of those springs. violently snapping and yanking
the points across the map to relieve the tension.
That's a great image. Yeah. The algorithm continually
calculates the gradient, which is the direction
of the pole, and updates the map, letting the
points push and pull each other until the entire
system settles into a state of minimal tension.
That sounds incredibly chaotic. and mathematically
exhausting for a computer to calculate. Like
every single point is pulling on every other
point constantly. It is intensely demanding.
The source actually notes that for a data set
with n elements, standard TSNE runs in what computer
scientists call O of n squared time and requires
O of n squared space. Meaning if you double the
amount of data, The computational cost doesn't
just double, it quadruples. Precisely. If you
have a thousand data points, the computer is
managing a million calculations. Right. If you
scale that up to a million data points, you're
suddenly asking the machine to handle a trillion
calculations. The memory and processing power
required become just a massive bottleneck as
the data set grows. So assuming you have, like,
a supercomputer that can handle the springs snapping
back and forth a trillion times, what does this
all mean for the actual map? Like how does it
avoid just lumping everyone together in the middle?
What do you mean? I mean, if everything shares
at least some tiny similarity with everything
else, wouldn't the tension just crush all the
points into one dense overlapping black hole
in the center of the screen? That black hole
scenario is exactly what would happen without
a few very clever mathematical interventions.
To prevent the map from collapsing in on itself,
TSNE relies on a user -defined parameter called
perplexity. Perplexity? Yeah. Which is, ironically,
how I feel when I try to read statistical mathematics.
Yeah, it's a great name. In this context, perplexity
is a number you choose before running the algorithm,
typically set somewhere between 5 and 50. Okay.
The technical definition involves matching the
Shannon entropy of the conditional probabilities,
but functionally you can interpret perplexity
as a smooth measure of the effective number of
neighbors a point has. Okay, let me translate
that real quick. You're basically giving the
algorithm an attention span. You're telling it,
hey, Only focus on the closest 30 people to this
point and safely ignore the thousands of other
people who are further away. That is exactly
what it does. And the algorithm adapts to the
density of the data to make it happen. It adjusts
the bandwidth of those Gaussian bell curves we
talked about earlier. So they change size. Yeah.
In a really crowded part of the high -dimensional
space, it shrinks the bell curve to focus only
on the immediate neighbors. In a sparse, empty
area, it widens the bell curve to cast a larger
net. That's smart. It ensures that every single
point gets to care about the exact same effective
number of neighbors, regardless of how packed
the data is. So it adjusts its own zoom lens
depending on the crowd. That's really cool. But
the source material mentions another huge obstacle
the algorithm has to overcome, something incredibly
ominous. sounding, the curse of dimensionality.
Ah, yes. The curse of dimensionality is a foundational
nightmare in data science. When you use regular
Euclidean distance. Like just measuring with
a ruler. Right, the simple straight line distance
we use in the physical world to measure the space
between two objects. It completely loses its
meaning in high dimensions. How does distance
lose its meaning? I mean, a mile's a mile, isn't
it? Well, in three dimensions, yes. But in high
dimensions, volume expands exponentially. Think
of a multi -dimensional orange. OK. As you add
hundreds of dimensions, the math dictates that
almost all of the volume of that orange gets
pushed to the very outer surface. If it were
a 500 -dimensional orange, it would be 99 % peel
and almost zero juice. Wait, really? So the data
points all migrate to the edges? Yes. And because
everything is pushed to the vast outer crust
of the space, almost all points seem completely
equidistant from each other. If distances lose
their ability to discriminate, the probabilities
of things being neighbors become too similar.
Asymptotically, they just converge to a constant
number. So if everyone is equally far away from
you, you don't actually have any neighbors. The
whole concept of a neighborhood just breaks down.
Exactly. To fix this on the final flat map, Vanermaten
and Hinton did something ingenious. they completely
abandoned the Gaussian bell curve for the low
dimensional map. OK, what did they use instead?
They swapped it out for a heavy -tailed student
t distribution, specifically a Cauchy distribution
with one degree of freedom. This is actually
where the t in TSNE comes from. Oh. Okay, let's
visualize a heavy tail. If a normal bell curve
looks like a hill, where the slopes swoop all
the way down to touch the floor, this Cauchy
distribution looks like a hill where the slopes
hover above the ground. Right. Like they stretch
out infinitely without ever touching zero. What
does that actually achieve though? It artificially
inflates the space. Because the tails hover above
the ground, there is a much higher mathematical
probability of finding data points way out on
the fringes. It essentially gives the algorithm
the permission it needs to push dissimilar objects
much, much further apart on the flat map than
they strictly are in the high -dimensional space.
Oh, I see. It gives the springs permission to
stretch. It artificially shoves groups away from
each other to keep the visual clean, preventing
that overlapping black hole we were worried about.
Yes, it creates those beautiful, distinct visual
islands of data that TSNE is famous for. Here's
where it gets really interesting, though, because,
I mean, we've been deep in the theoretical math
for a while now. Gaussian kernels, Cauchy distributions,
O of n squared complexity. We've been in the
weeds. Yeah, but this isn't just an academic
exercise happening on a chalkboard. The real
-world applications of this algorithm are astonishing.
Like, when you take this ability to find hidden
islands of data and apply it to actual human
problems, it is quite literally saving lives.
Oh, definitely. The range of fields relying on
TSNE to see hidden patterns is just staggering.
Take medicine, for example. The source highlights
researchers using TSNE for analyzing EEG signals
to detect epileptic seizures. Which makes perfect
sense when you think about it. And EEG is recording
the chaotic, high -dimensional electrical noise
of the entire human brain millions of overlapping
signals. To a human eye looking at a raw chart,
it's just a wall of static. But T -SNE can map
that electrical chaos and visually group the
moments in time that look similar. It pulls the
distinct, hidden signature of a seizure out of
the static and plots it as an isolated island
on a map. It's incredible. The text also emphasizes
its use in exploring breast cancer data, specifically
computer -aided diagnosis, or CADEX. Wait, how
does the algorithm map a tumor? Well, tumor doesn't
just have one or two features. When doctors analyze
tissue, they're looking at cellular density,
the thickness of cell walls, genetic markers,
dozens or hundreds of microscopic variables.
Ah, okay. That is a high dimensional space. Exactly.
TSNE flattens all of those variables. It allows
a researcher to plot a new patient's tumor profile
on a screen and see instantly if it visually
lands on the island of known benign tumors or
if it clusters with the malignant ones. It takes
the microscopic and makes it entirely visual.
And it completely jumps industries, too. In technology
and computer security, researchers use it to
analyze the behavior of off -the -shelf anti
-virus engines, like grouping how different software
reacts to millions of lines of malicious code.
It's heavily utilized in arts and culture too.
Natural language processing uses T -S -N -E to
generate word embeddings from 19th century literature.
I love this example so much. It maps the vocabulary
habits of authors. It plots how often they use
specific prepositions or adjectives. Suddenly
you can take an anonymous piece of text, run
it through T -S -N -E, and watch it visually
cluster right next to Charles Dickens. Because
the algorithm recognized his subconscious high
-dimensional word choice fingerprint. Yes. to
learn features from music audio using deep belief
networks. Earth scientists use it for geological
domain interpretation, identifying complex material
types in geochemical data scattered across the
globe. From the electrical storms in our brains
to 19th century literature to the literal dirt
under our feet. The primary takeaway here is
that TSNE is not just a math trick. It is a fundamental
lens. When professionals across these entirely
different disciplines are drowning in variables,
TSNE acts as a translator. It converts abstract
numbers into spatial intuition, giving researchers
a kind of superhuman sight. But, and this is
a massive, but, the source material has an entire
section dedicated to the outputs of TSNE that
serves as a giant flashing warning label. Yes.
This raises an important question, maybe the
most important question of our entire discussion.
Given everything we know about how the algorithm
artificially inflates space and uses heavy tails,
How much should we trust our own eyes when we
look at a TSNE map? Wait, so you're telling me
the visual clusters we see on these maps, those
beautiful distinct islands of data that are helping
doctors and geologists, might be completely fake?
They absolutely can be, and this is where critical
thinking becomes mandatory. According to the
source, TSNE plots often seem to display clusters,
but those visual groupings are heavily manipulated
by the chosen parameterization. You mean the
settings? Right. Remember that perplexity number
you have to choose, the attention span. If you
set that number wrong for your specific data
set, TSAE can and will produce visual clusters
even in data that has absolutely no clear grouping
in reality. So it will just invent quirks. because
it's trying so hard to push things apart with
that heavy -tailed t -distribution. Like, it
wants to make a map so badly, it will draw borders
where none exist. Yes, it will show you false
findings. But the distortion gets even more counterintuitive
than that. Let's say you do have real mathematically
verified clusters. On a TSNE map, you might see
one cluster that is huge and spread out and another
that is tiny and dense. You might see two clusters
right next to each other and a third one way
off in the corner of the screen. Naturally looking
at that, my brain immediately at least says,
OK, the big cluster has more variance inside
of it, and the two clusters close together are
closely related, while the one far away is an
outlier. And your brain would be completely fundamentally
wrong. The source explicitly states that the
size of clusters produced by TSNE is mathematically
not informative, and neither is the distance
between different clusters. Are you kidding?
Nope. The algorithm distorts the macrogeography
of the map to preserve the local neighborhoods.
The heavy tail inflates space unpredictably.
You cannot use a ruler on a TSN plot. You cannot
say cluster A is closer to cluster B than cluster
C. That is wild. The big picture is essentially
a highly distorted illusion. So it successfully
brings the data down into two dimensions, but
it fundamentally warps the global distances just
to make the local neighborhoods look good. Exactly.
So if the distances are meaningless and the clusters
might be fake, what is the solution? How do researchers
actually use this without getting fooled by their
own data? The solution is a process the text
calls interactive exploration. You cannot just
run the TSNA algorithm once, print out the picture,
and publish your paper. You have to actively
play with the parameters. You have to run it
with different perplexities, validate the findings
against other analytical methods, and truly understand
the math beneath the image. Which totally explains
why the software section of the article highlights
tools that are specifically designed for this
kind of iteration. Because of that brutal O of
N squared computation time we mentioned earlier,
developers have built approximation methods.
Right, to speed things up. Yeah. There is a C++
implementation called Barnes -Hutt, available
directly from Vandermaten, which groups distant
points together as a single massive point to
save calculation time. Yes, and the R package,
Artsny, or ELKI, which also utilizes that Barnes
-Hutt approximation to make the algorithm viable
for massive data sets. And Python users, they
rely on scikit -learn, which is hugely popular
and handles both exact solutions and the Barnes
-Hutt approximation. There's TensorBoard, the
Julia package, T -Series. The point is, all of
these tools are being built not to generate a
single static image, but for progressive visual
analytics. Exactly. They are built so you can
turn the dials in real time and watch how the
map changes. Because if the map radically transforms
and your beautiful clusters just disappear the
moment you change the propensity from 20 to 30,
those clusters probably weren't real in the first
place. Exactly. The truth of the data has to
be robust across different parameters. It has
been mathematically proven that with the right
parameter choices, TSNE can recover well -separated
clusters, but you have to do the rigorous work
to find those correct settings. It demands an
active, highly skeptical user. So to summarize
our deep dive today, TSE &E is this brilliantly
complex, beautifully flawed translator. It takes
the incomprehensible scale of high dimensional
space, whether that's hundreds of variables in
a breast cancer tumor or 500 personality traits
at a massive convention. And it uses probabilities
and heavy tail distributions to squish that reality
down into a 2D map we can actually comprehend.
It really is a tool that has fundamentally revolutionized
data visualization, but it demands that we do
not take its visual outputs at face value. It
serves as a perfect example of why we can never
let algorithms do our thinking for us. It provides
a unique perspective, but it does not provide
an absolute truth. Right. Which brings me back
to where we started. We talked about how our
brains evolved for three -dimensional space,
how we are biologically wired to look at the
physical world and immediately find structure.
We look up at a random scattering of stars and
we see a hunter with a belt. We see faces in
the clouds. We are deeply wired pattern recognition
machines. We will trust our eyes over abstract
numbers almost every single time. And that leaves
a lingering question for you to mull over. We've
just learned that a TSNE map will show you cluster
sizes and distances that look incredibly meaningful.
But mathematically? They are just an illusion
of the translation process. So in an era where
we increasingly rely on complex algorithms to
compress impossible hyperdimensional reality
into something our primate brains can digest,
are we actually using these visualizations to
uncover the hard truth of the data? Or are we
just using them to paint a picture that our human
brains are biologically desperate to see? A very
necessary reminder to always question the lens
through which we view the world. Thank you for
joining us on this custom deep dive. Keep exploring,
keep tweaking those parameters, and most importantly,
keep questioning the data around you.