Have you ever wondered how a computer actually
reads? Oh, that's a profound question, actually.
Right. Because, you know, it doesn't have a brain.
Yeah. It doesn't understand poetry or emotion.
So how does it actually know what a word means?
Well, for decades it didn't. A word given to
a machine was literally just a meaningless string
of characters. Like a barcode. Exactly like a
barcode. The computer had no idea that dog and
puppy were related at all. They were just two
completely different files. But obviously that
changed. To understand how predictive text works
on your phone today, or how translation algorithms
got so eerily accurate, we're taking a deep dive
into a foundational 2013 Google breakthrough.
It was an absolute paradigm shift. It really
was. And we're drawing our insights today from
a really detailed Wikipedia breakdown of a model
called Word2Vec. A classic. Yep. Our mission
here is to skip the dense computer science jargon,
keep the mind -blowing aha moments, and explore
how researchers essentially taught machines the
meaning of language. By turning words into math.
Exactly. But to get to that point, we first have
to understand how this algorithm transforms a
piece of text into a specific location on a very
complex map. Right, the spatial concept. Yeah.
At its core, Word2Vec is what's called a shallow
two -layer neural network. It digests a massive
corpus of text. We're talking millions or billions
of words. Just a huge amount of data. And it
maps every unique word to a high dimensional
vector space. Okay, so let's unpack this. the
documentation mentions, these spaces typically
have anywhere from 100 to 1 ,000 dimensions.
Which is literally impossible for a human brain
to visualize. Right. Trying to picture 1 ,000
dimensional grid will just melt your brain. So
instead, think of it like a giant unimaginably
complex galaxy. I like that analogy. In this
galaxy, every single word gets its own specific
coordinate. It's a star. And because of how the
algorithm processes the text, you end up with
like The action verb solar system way over in
one corner of the galaxy. Oh, and the capital
cities are clustered together in a completely
different corner. Yes. But what's fascinating
here is the specific mechanism the model uses
to figure out where those stars belong. Right.
It relies on a mathematical principle called
cosine similarity. I've heard that term, but
how does it actually work geographically in our
galaxy? So imagine drawing a straight line from
the center of our semantic galaxy out to the
star for the word walk. OK, got it. Then you
draw another line from the center to the star
for the word ran. Cosine similarity simply measures
the angle between those two lines. Oh, wow. Yeah.
If the angle is very small, the lines are pointing
in almost the exact same direction, meaning the
words are highly related. So it's judging a word
by the company it keeps in the text. Precisely.
You might say, I went for a walk to the store
or I ran to the store. The surrounding context
is virtually identical. Right, so the math literally
pushes their stars closer together. Their angle
shrinks. That makes so much sense. And the same
mechanism applies to, like, geographical relationships
too, right? Absolutely. The lines drawn to Berlin
and Germany will have a very narrow angle between
them. It even works for transitional words. But,
and however, are mathematically mapped as neighbors
because they serve the exact same contextual
function in a sentence. Because they're always
surrounded by the same types of opposing clauses.
Exactly. Okay. But knowing that every word gets
a coordinate based on its neighbors still leaves
a massive question. How does it calculate the
coordinates from scratch? Yes. The breakdown
of Word2Vec shows it doesn't just use one method.
It relies on two distinct model architectures
to learn these representations. The two engines
under the hood. They're known as continuous bag
of words, or CBOW, and the continuous skip gram
model. CBOW and skip gram. Right. Both of them
iterate over the vast text corpus, sliding a
window over a few words at a time, but they approach
the math from completely opposite directions.
So when you look at how CBOW operates, it's essentially
playing a massive game of Mad Libs. It's a fill
-in -the -blank test. That's a great way to put
it. It takes the surrounding context words, combines
their mathematical values, and uses that combined
context to predict the missing middle word. Right.
So if the sentence is the cat blank on the mat,
CBOW looks at the cat on and the mat and mathematically
guesses that the missing center word is probably
sat. And we actually call it a bag of words because
it assumes the order of those context words doesn't
even matter. Wait, really? Yeah. Mat, the oncat,
would yield the exact same prediction. That is
wild. And then you have the other architecture,
skipgram, which flips the entire process inside
out. Right. So if CBOW is playing Mad Libs, skipgram
is like being given a single word and being asked
to hallucinate the entire sentence around it.
Exactly. You feed the machine the word sat. And
it has to predict that the surrounding neighborhood
of words is likely the, cat, on, and map. You
give it the center point, and it hallucinates
the neighborhood. Yep. And Skip Graham introduces
a really interesting technical nuance here. It
doesn't treat all the surrounding words equally.
It actually weighs nearby context words more
heavily than distant ones. So the word immediately
next to sat has much more influence on the math
than a word five spots away. Which makes intuitive
sense. The word cat is far more related to sat
than the word the at the very beginning of the
sentence. Precisely. So is one of these engines.
objectively better than the other? Well, the
authors noted a specific trade -off. CBOW is
generally much faster to train because it's kind
of smoothing over the context, but SkipGram,
because it forces the model to predict multiple
different context words from a single starting
point, does a much better job representing infrequent
or rare words. So it squeezes more signal out
of less data. You got it. Let's linger on the
math. powering these predictions for a second
though, because the architecture is described
in the source as linear, linear softmax. Which
is quite the mouthful. Yeah, and I want to make
sure you, listening to this, can really visualize
what the machine is actually doing when it guesses
a word. Okay, let's break it down. Let's stick
with the cat sat on the mat. In standard bag
of words models, the context is just a simple
tally, right? The appears twice, cat appears
once. Right. But in continuous bag of words,
the system takes that simple tally and multiplies
it by a massive matrix. And multiplying by a
matrix sounds incredibly abstract. So think of
this matrix like a giant audio mixing board in
a recording studio. Oh, I love the mixing board
analogy. Every single word in the English language
has its own slider on this board. When the machine
is trying to predict the missing word in the
cat blank on the mat, it pushes up the sliders
for the cat on and matte. And by pushing up those
specific sliders, the mixing board outputs a
massive list of numbers, basically one raw score
for every single word in its vocabulary. Thousands
of numbers. Exactly. And finally, the system
applies a mathematical function called softmax.
Softmax simply takes all those raw scores and
squashes them into a probability distribution
that adds up to 100%. So the machine looks at
the output and says, I am 90 % sure the missing
word is sat, 5 % sure it's slept, and 0 .001
% sure it's refrigerator. Exactly. But here is
the crucial part. How does it learn? If it guesses
refrigerator and gets it wrong, it reaches back
into that giant mixing board and slightly adjusts
the internal connections. The weights. The weights,
yes. It adjusts them so that sat gets a higher
score next time. It does this billions of times
until the sliders are perfectly tuned. And whether
the system uses CBOW to predict the center word
or skip gram to predict the surrounding words,
that linear, linear softmax structure, that mixing
board is essentially the same, isn't it? It is.
The only difference is the specific goal it is
trying to maximize during training. Okay, so
once these engines finish crunching the data,
once they finish adjusting millions of sliders
on that mixing board, the resulting model accidentally
unlocks something that mimics human reasoning.
Yes. This is the famous 2013 discovery by Tomas
Mikolov and his team. Magic trick. They realized
the model had implicitly learned semantic and
syntactic patterns, and more importantly, those
patterns could be reproduced using basic algebra.
Which brings us back to our opening equation.
You take the vector, the specific coordinate
for the word brother, subtract the coordinate
for man, and add the coordinate for woman. And
the math just works. The resulting location in
that thousand -dimensional galaxy lands almost
exactly on the star for the word sister. It's
incredible and it works for geography too. Paris
minus France plus Germany lands you right next
to Berlin. Wow. It even captured syntactic changes
like verb tenses, walking minus walk plus run
equals running. So the model had captured multiple
different degrees of similarity between words
purely by analyzing their proximity in the training
text. Pure proximity, yeah. Okay, wait. I have
to push back on this. Go for it. If the model
is just adjusting sliders on a soundboard based
on a sliding window of text, does it actually
understand the concept of a brother or a sister?
That's the big question. Or is this just like
the ultimate game of statistical guilt by association?
I mean, it doesn't know what a sibling is. It
just knows what letters usually stand next to
it. And that skepticism is entirely warranted.
The scientific community heavily debated this
exact point. I can imagine. Researchers like
Joachim Goldberg and Omer Levy analyzed why Word2Vec
was so successful, and they actually argued that
the explanations given for this algebraic magic
trick were, and I quote, Very hand -wavy. Hand
-wavy. A highly technical computer science term.
Right. But they pointed out that the model simply
aligns perfectly with a concept in linguistics
called J .R. Firth's Distributional Hypothesis.
Which is what? The hypothesis states that the
meaning of a word is defined entirely by its
use in context. Word2vec just proved you can
mathematically map that context at a massive
scale. Oh, I see. It performs inference for a
simple generative model for text. So it's brilliant
statistics mapping the g... of language, but
it is not cognitive understanding. Here's where
it gets really interesting though. Even if it
is just statistical guilt by association, if
we can mathematically map the context of English
words to discover hidden relationships, what
happens if we apply this exact same math to languages
that aren't spoken by humans at all? The ripple
effect. The underlying architecture proved so
robust that researchers immediately started looking
beyond human vocabulary. It scaled into biology.
Exactly. Because a DNA sequence is essentially
just a long string of repeating letters. It's
a text corpus. So researchers created biovectors
or biovec. They applied this exact same sliding
window logic to biological sequences. They mapped
DNA, RNA, and protein sequences. Which is mind
-blowing. It really is. By treating amino acid
sequences like words in a sentence, biovectors
could characterize biological sequences and find
underlying biochemical and biophysical patterns.
So it deduced the functional meaning of a protein
sequence just by looking at the company it kept.
Yeah. There's even a variant called DNA2vec that
proved you could use cosine similarity. You know,
that angle between two lines to measure the actual
physical similarity of genetic sequences. Unbelievable.
And the scaling didn't stop at microscopic biology
either. It scaled up linguistically. Instead
of just mapping single words, researchers developed
Doc2vec. Right, which generates distributed representations
for variable -length texts. Meaning it captures
entire sentences, paragraphs, or whole documents.
Exactly. Rather than just finding the coordinate
for the word Berlin, Dr. Veck can find a coordinate
for an entire encyclopedia article about Berlin.
It uses architectures incredibly similar to CBOW
and Skipgram, but it adds a unique document identifier
as an extra piece of context on that mixing board.
OK, that makes sense. And from there, the technology
evolved into Top2Vec, which estimates representations
of entire abstract topics. Wait, how do you even
calculate the coordinate for an abstract topic?
You can't just draw a circle around a vague concept
in a thousand -dimensional space. That's tricky.
You have to squash it down somehow to make sense
of it, right? That is exactly what happens. Top2Vec
takes those document embeddings and reduces their
dimensions using a technique called UMA. UMA.
Yeah. Think of UMA like taking a complex 3D globe
of the Earth and flattening it onto a 2D piece
of paper without distorting the relative distances
between the cities too much. So it literally
flattens the semantic galaxy into a readable
map. Yes. And once the space is flattened, it
scans that map using a clustering algorithm called
HDDS scan. If UM draws the map, HDDS scan is
the tool that looks for the densely populated
cities on that map. It finds tight clusters of
similar documents, calculates the centroid of
that cluster, and calls that the topic vector.
So whatever word embeddings are situated closest
to that centroid become the title of the topic.
You got it. It's literally finding the center
of gravity for a conversation. And that mechanism
also found a massive application in medicine,
specifically in radiology, with the creation
of intelligent word embeddings, or IWE. Which
was huge because one of the biggest challenges
with the original Word2Vac is how it handles
unknown or out -of -vocabulary words. Right.
If it has never seen a word in its training data,
it just assigns it a random, essentially useless
vector. And in medicine, a radiologist's report
is chaotic. They use bizarre abbreviations, ungrammatical
telegraphic phrases, highly specialized jargon.
A doctor at one hospital might write, patient
shows, guys size E on the interior or whatever,
while a doctor across the country uses completely
different shorthand for the exact same issue.
Exactly. So researchers combined Word2Vec with
a semantic dictionary mapping technique to create
IWE. Ah. They mapped all those different variations
into the same vector space. Because of this,
an IWE model trained on messy data from one specific
hospital could successfully translate and understand
reports from a completely different institution.
So it generalized the meaning of radiological
jargon across the entire medical field. It did.
I mean, Word2Vec reshaped genomics, it transformed
topic modeling, and it decoded medical jargon.
It was an absolute revolution. Undoubtedly. Yet
if you look at the literature today, the straight
word2vec approach is officially described as
dated. It is. So what does this all mean? How
does a breakthrough that profound become a relic
in less than a decade? Well, if we connect this
to the bigger picture, it speaks to the hyper
accelerated pace of AI research. Though the history
of word2vec is full of irony. Oh, so. When Tomas
Mikolov and his team first submitted the original
Word2Vec paper in 2013, it was actually rejected
by the reviewers for the International Conference
on Learning Representations. Wait, really? The
AI establishment completely dismissed the paper
that arguably kicked off the modern embedding
revolution? It happens more often than you'd
think. That is hilarious. But today, the industry
has moved on to transformer -based models like
Elmo and BERT. Transformers add multiple neural
network attention layers on of a word embedding
model. Instead of just looking at a fixed sliding
window of context, they dynamically weigh the
relevance of every word in a sentence against
every other word, no matter how far apart they
are. Oh, wow. But the foundation turning those
words into math still relies on the trail Word2vec
blazed. There's another fascinating wrinkle to
why Word2vec's supremacy faded, though. It wasn't
just about Transformers being better. It turns
out Word2vec's initial magic might have been
a bit of an illusion. Ah yes, the 2015 study
by Levy and his colleagues. Right. They demonstrated
that much of Word2vec's allegedly superior performance
wasn't actually a result of the model architecture
itself. Which was a huge realization. CBOW and
Skip Graham weren't necessarily the secret sauce.
The superior performance was heavily driven by
the choice of specific hyperparameters. And hyperparameters
are basically the little dials and knobs the
researchers tune before they start trading the
model. Exactly. For instance, subsampling high
-frequency words. Words like the or and appear
so often they provide almost no informational
value. Right. Tuning the dial to remove or subsample
them speeds up training and clarifies the data.
Another dial is the context window size. The
authors found a sweet spot of a 10 -word window
for SkipGram and a five -word window for CBOW.
Looking at five words to the left and five words
to the right. Yep. Then there is the training
algorithm itself. You can use something called
hierarchical softmax. OK, what is that? Instead
of calculating the probability for all 50 ,000
words in the dictionary every single time it
guesses a word, it uses a Huffman tree. A Huffman
tree. Think of it like playing 20 questions.
Is the word a noun? Yes. Is it an animal? Yes.
It drastically cuts down the math. Oh, that makes
the training significantly faster. It really
does. Alternatively, you can use a technique
called negative Instead of updating the mixing
board sliders for the one correct word and the
49 ,999 wrong words, it updates the correct word
and just a handful of negative samples. Like
maybe five random wrong words? Exactly. It's
a massive shortcut. And Levi's team showed that
if you took these specific hyperparameter settings,
these perfectly tuned dials, and applied them
to older, more traditional natural language processing
approaches, those older methods suddenly performed
just as well as Word2Vec. So it wasn't that the
neural network was infinitely smarter. The Google
team had just figured out the absolute perfect
combination of settings for the data. It was
a master class in parameterization. The documentation
details how dramatically these parameters affect
model quality. Overall accuracy inherently increases
with the number of words used in training and
the number of dimensions in the vector space.
But there's a massive computational cost to turning
those dials up, isn't there? Huge. team reported
that doubling the amount of training data results
in an increase in computational complexity, mathematically
equivalent to doubling the number of vector dimensions.
Yikes. Yeah, the curve gets steep incredibly
fast. This perfectly explains why companies spend
billions of dollars on data centers to train
these things. Every time you turn the dial up
to make the model a little smarter, the math
gets exponentially harder. It does. In fact,
one study from 2017 found Word2Vec only really
outperforms older techniques. like latent semantic
analysis, if it's trained on a medium to large
corpus, meaning more than 10 million words. If
you have a small data set, the older models actually
work better. Because there is no single magic
algorithm in AI. It is always an interplay between
the architecture, the sheer volume of data, and
the precise tuning of the hyperparameters. For
a skipogram model on a medium corpus, the absolute
sweet spot seemed to be 50 dimensions, a window
size of 15, and 10 negative samples. It's really
like finding the exact recipe for a cake. You
can't just throw ingredients in a bowl. The oven
temperature and the baking time matter just as
much as the flour itself. That's a perfect analogy.
this all full circle, to you listening to this
deep dive right now, why does a 2013 algorithm
that is now officially considered dated actually
matter to your daily life? It's a valid question.
It matters because the legacy of Word2vec is
literally at your fingertips. The next time you
use a predictive text feature on your phone or
you watch your browser seamlessly translate a
foreign webpage or you read an article about
AI analyzing thousands of medical reports to
find a cure, You are looking at the direct descendants
of word2vec. It fundamentally shifted the paradigm
from rule -based linguistics to distributional
semantics. It proved mathematically that meaning
is, in fact, just the company a word keeps. And
that leads me to a final thought I want to leave
you with. We've spent this whole time talking
about words and DNA and documents, but let's
apply this logic to something a little closer
to home. Oh boy. If an algorithm can perfectly
deduce the deep semantic meaning of a word simply
by mathematically mapping its closest neighbors,
could that exact same distributional principle
be applied to us? That is a wild thought. Right.
If a computer mapped all your digital connections,
your interactions, your geographic proximities,
your friends, would it understand your meaning
better than you do? Are you just the sum of the
company you keep? Something to think about. Thank
you for joining us on this deep dive. We'll catch
you next time.