Have you ever stopped to think about just how
incredibly messy human language actually is?
No, it's a nightmare. Right. I mean, we were
constantly dealing with slang and sarcasm, metaphors,
words that change meaning entirely just depending
on the tone of your voice. Exactly. And you and
I navigate this like poetry of everyday life
without even breaking a sweat. But how do you
turn that fluidity of human vocabulary into cold,
hard mathematics? That is the big question. It
is. And today we are opening up a stack of research
for a deep dive into a 2014 breakthrough called
GloVe. We really want to see the exact moment
artificial intelligence figured this out. It
really is one of the most fascinating problems
in computer science, because for decades, computers
were effectively just playing a giant, very fast
game of matching letters. Just pattern recognition.
Right. They didn't understand what the words
meant at all. They just knew that the letters
D -O -G matched other instances of D -O -G. Right,
right. So if you asked an early computer a question
about a canine, it wouldn't even know to look
for the word dog because it had no concept that
those two things were related. Wow, yeah. So
to get to the modern artificial intelligence
we interact with today, there had to be a bridge,
you know, a way to translate the abstract concept
of semantic meaning into pure numbers. OK, let's
unpack this. Imagine you are trying to teach
an alien. our language. I love this analogy.
Right, but instead of handing them a dictionary
with actual definitions, you only give them a
massive spreadsheet showing how often certain
words sit next to each other in a sentence. Just
raw tallies. Exactly, just the raw tallies. And
that is essentially what GloVe is. That is a
brilliant way to conceptualize it. So GloVe,
which stands for Global Vectors, was developed
as an open source project at Stanford University,
and it launched in 2014. OK. At the time, it
was designed to compete with another famous model
called Word2Vec. And it fundamentally changed
natural language processing, or NLP. It was like
the turning point. It was. It was the crucial
stepping stone that proved we could actually
quantify meaning. But before we get into the
heavy math of vectors and algorithms, we need
to take a step back. Because the source material
points out something I found just incredible.
The 1950s connection. Yes. The 2014 code for
this algorithm. is actually rooted in a linguistic
philosophy from the 1950s. It is, and this provides
the entire foundation for the math we are going
to talk about. There is a famous quote from the
English linguist J .R. Firth from 1957. Oh, well,
lay it on us. He said, you shall know a word
by the company it keeps. You shall know a word
by the company it keeps. I mean, it sounds almost
like an old proverb. It does, but it perfectly
encapsulates the theory behind GloVe. You see,
GloVe is what we call an unsupervised learning
algorithm that creates vector representations
of words. Right. The ultimate goal is to map
words into a meaningful space. Okay, let's pause
right there because if you are sitting there
listening to this on your commute, hearing the
phrase meaningful space might sound a bit abstract.
Fair enough, yeah. Think of a vector as an address
in a massive, floating, multi -dimensional city.
Okay, city of words. Exactly. The algorithm's
job is to assign every single word and address
in this city so that the mathematical distance
between two words is directly related to their
semantic similarity. So words that mean similar
things end up physically closer together. Yes.
Like car and truck, they end up physically closer
together in this mathematical space. And GloVe
achieves this by combining two different model
families, global matrix factorization and local
context window methods. OK, hold on. global word,
word co -occurrence statistics. I mean, I know
we want to understand the mechanics, but we need
to keep this grounded for the listener who just
wants that aha moment. Right, right. How do we
actually count the company of word keeps in plain
English? Well, what's fascinating here is that
the underlying mechanism is incredibly intuitive
once you visualize it. The algorithm doesn't
read a book the way you or I do, you know, comprehending
the narrative or the characters. It just scans
the text looking for neighbors. So let's visualize
it. Think of the algorithm as a spotlight moving
across a sentence. The room is totally dark.
and the spotlight only illuminates a few words
at a time as it moves from left to right. I like
that. And that spotlight is what the researchers
call the context window. Precisely. And the size
of that spotlight matters. Let's look at the
exact example provided in the Wikipedia text
to see how this counting works. Okay, let's do
it. Imagine the sentence is, coined from global
vectors, is a model for distributed word representation.
Let's say our context length, our spotlight,
is set to three words. So if the scotlight is
centered on the word model, does it just log
every single word in the sentence as its company?
Not quite. There is a strict distance limit.
If we center on the word model, which is the
eighth word in that sentence, we only look three
words to the left and three words to the right.
Okay. So the 11th word in the sentence is word
because word is within three spaces of model.
The algorithm records that model is in the context
of word. They are company. But the 12th word
in the sentence is representation. So that is
four spaces away from model. Exactly. So representation
falls outside the spotlight. literally does not
exist to the algorithm in this specific instance.
It just logs a zero. Just a zero for that pairing.
And there is another really important rule here.
A word is never considered to be in its own context.
Wait, really? Yeah, so the word model is not
in the context of the word model. The only way
a word counts as its own company is if it literally
repeats within the same spotlight window. Oh,
like if I said... I don't think that that is
a problem. The first, that is right next to the
second that. So they fall into each other's context.
You've got it. So the algorithm does this for
the entire corpus, the entire collection of text
it's studying, which could be billions of words.
It counts up every single time word A appears
in the context of word B, and this creates a
massive spud sheet, a matrix of co -occurrence
statistics. So counting is just the first step,
right? Right. Next comes the probabilistic modeling.
And here is where we find a crucial nuance. Co
-occurrence is not a two -way street. I am so
glad you brought this up because this asymmetry
is wild to me. The probability of seeing word
A when you are looking at word B is not the same
as the reverse. It's really not. The source gives
a brilliant example of this with the words much
and a do. A do is this archaic word. I mean,
in modern English you almost never hear it unless
it's in the phrase, much ado about nothing. Exactly.
So if the algorithm spots the word ado, the probability
that the word much is standing right next to
it is practically 100%. But if we flip it around...
the logic changes entirely. The word much is
incredibly common. We say much better, too much,
how much? So if the algorithm spots the word
much, the probability that a do is the word next
to it is near zero. The relationship is entirely
asymmetric. Okay, so our alien spreadsheet now
has these massive asymmetric tallies of how often
every single word hangs out with every other
word. But tallies are just raw numbers. Here's
where it gets really interesting. How does a
computer use raw tallies to understand the actual
meaning of a word? That's the leap. Yeah, like
how does it know the difference between say ice
and steam? Well, to answer that, the researchers
provided a brilliant data table from a corpus
of six billion words. Six billion. Six billion.
They looked at the probabilities of four specific
words appearing in the context of ice versus
steam. The four context words were solid, gas,
water, and fashion. Okay, let's walk through
that because this is the real magic trick of
the whole algorithm. It really is. So first,
they looked at the word water. both ice and steam
are forms of water, right? So they both appear
frequently in the context of the word water.
If the computer only looked at that raw frequency,
ice and steam would be mathematically indistinguishable.
OK, and what about the word fashion? Well, neither
ice nor steam has much to do with fashion. So
the probabilities for both are incredibly low.
Again, indistinguishable. So looking at words
they both relate to, or words neither relates
to, that doesn't help at all. Exactly. The breakthrough
comes from looking at the ratio of their probabilities.
A ratio. Yes. The algorithm takes the probability
of a word appearing near ice and divides it by
the probability of that same word appearing near
steam. And this is where the word solid and gas
come in. Right. When they calculated the ratio
for the word solid, the result was a relatively
large number, 8 .9. Okay. This tells the algorithm
that solid is much more closely related to ice
than it is to steam. Oh, wow. But when they calculated
the ratio for the word gas, the result was a
tiny decimal, 0 .085. So that tells the algorithm
that gas is heavily skewed towards steam and
away from ice. Precisely. But for words like
water or fashion, where the occurrence rates
are similar, the ratio just hovers right around
one. I just marvel at the elegance of this. I
mean, the computer has absolutely no idea what
physics is. None. It doesn't know what temperature
is. It has never felt ice or been burned by steam.
It simply uses ratio math to realize that one
of these words behaves like a solid and the other
behaves like a gas, meaning it's just ratios.
It is deeply elegant. But here is the catch.
There's always a catch. Always. Having a spreadsheet
with billions of words and trillions of ratios
is practically useless because it's just too
massive. It takes up way too much memory and
computational power. We need to squish that giant
spreadsheet down into a tight, highly dense map
of coordinates. Right. Going back to your analogy
of the multi -dimensional city, we need to assign
them actual addresses. Yes. And to do that, the
model uses a technique called multinomial logistic
regression. Wait, you're losing me? We just said
meaning is ratios. Why are we bringing in regression
equations? Because the regression is the compression
tool. Essentially, the idea is to learn a specific
vector, an address, for each word, such that
when you plug those vectors into a formula, the
math spits out a number that approximates the
logarithm of those co -occurrence probabilities
we just talked about. Okay, and just to clarify
for everyone listening, when we say logarithm
here, think of it as a way to scale down massive
unwieldy numbers into manageable steps so the
computer doesn't get overwhelmed by the sheer
scale of the data. Exactly. But if you just run
a standard regression equation, your math is
going to get hijacked by noise. Noise. Like what
kind of noise? Imagine someone makes a typo in
a blog post, and suddenly two words that have
absolutely no real relationship are sitting next
to each other in the spotlight window. Oh, I
see. If the algorithm treats that single occurrence
with too much weight, it completely throws off
the semantic meaning. So how does GloVe fix the
noise? By using a weighted function. They tweak
the math. so that the weight of the relationship
ramps up slowly as the absolute number of co
-occurrences increases. OK. They cap the maximum
count at 100 occurrences, meaning once two words
appear together 100 times, the algorithm considers
the relationship fully established. It just stops
counting them. It doesn't give extra weight to
the hundred and first time or the millionth time.
Why cap it at exactly 100? Well, think about
a word like the. If the word the appears a billion
times next to dog, we don't want at having a
billion times the mathematical weight of a rarer,
much more descriptive word like golden. Oh, that
makes total sense. So they used a fractional
power curve, specifically setting a parameter
called alpha to 3 fourths. So it acts like a
governor on an engine. Yeah. It lets the weight
grow quickly at first, but then slattens it out
so common words don't drown out the meaningful
ones. Exactly. All while making sure rare typos
don't break the system either. Precisely. And
this results in something truly magical. Because
we have compressed all these ratios into spatial
coordinates, you can actually do geometry with
definitions. Yes. This is the classic aha moment
of word embeddings. Because once you have these
vectors, you can literally do math with words.
You really can. If you take the vector address
for king, subtract the vector for man, and add
the vector for woman, the resulting coordinate
you arrive at in this multi -dimensional space
is the vector for queen. That's crazy. The algorithm
mapped the latent concept of gender and royalty
strictly through spatial distance. That blows
my mind every single time I hear it. And there's
a funny little quirk about the final math here,
too. Two vectors. Yeah. The model actually generates
two separate vectors for every single word based
on how the matrix is split. But when it came
time to actually use the model, the researchers
found empirically that just adding those two
vectors together created the best final representation.
Right. It wasn't necessarily a grand theoretical
design, it just worked better in practice to
add them up. If we connect this to the bigger
picture, that pragmatic approach is exactly what
made gloves so wildly successful. Once this model
was trained, it wasn't just a theoretical academic
exercise. It was put to use in the real world
almost immediately. Give me some examples. Where
was this alien spreadsheet actually being used?
The applications were surprisingly diverse. Because
it mapped semantic relationships so well, it
was used to find relations between zip codes
in cities or to link companies to their specific
products. Oh, wow. It became a core component
for finding synonyms in massive databases. Have
you ever heard of spaCy? the Natural Language
Processing Library, right? Yes. spaCy used GloVe
to build semantic word embedding features. If
you queried a word, spaCy would use GloVe's vectors
to compute the top list of words that matched
it. And it did this using distance measures like
Euclidean distance and cosine similarity. Let's
visualize that, because cosine similarity sounds
really intimidating. It does. But imagine drawing
a line from a center point to the word ice, and
another line from the center to steam. Cosine
similarity is just measuring the angle between
those two lines to see how closely they point
in the same direction. Exactly. While Euclidean
distance is just taking a ruler and measuring
the straight physical gap between the two points.
It's literal geometry applied to language. And
because of that, it was even used in healthcare.
Yes, I really wanted to dig into this because
the source mentioned it's used in psychology.
How does a math equation help detect psychological
distress? By analyzing the structural distance
of the words patients use in interviews, you
see a healthy, organized mind tends to use language
where the concepts group together in expected,
semantically cohesive ways. But researchers found
that they could use GloVe to flag underlying
distress by mapping a patient's vocabulary. If
a patient was jumping between concepts that were
mathematically very far apart in GloVe's multidimensional
space, it indicated a level of cognitive disorganization.
Oh, wow. Yeah, a distress that a doctor might
completely miss from just a surface -level conversation.
The geometry of their vocabulary revealed their
actual state of mind. That is just astounding.
a doctor using an algorithm to map a patient's
cognitive state based on the physical geometry
of their work. It's incredible. But as revolutionary
as it was, GloVe had a massive blind spot, like
a fatal flaw in its design. The source points
out that it is essentially helpless when it comes
to homographs. Ah, yes. Homographs. Words that
have the exact same spelling but entirely different
meanings. Right. Because GloVe is an unsupervised
algorithm that assigns a single static vector
based purely on the spelling of a word, it mashes
those different meanings together into one mathematical
point. It's like having a contactless in your
phone where you happen to know two completely
different people named John Smith. Oh, that's
a good way to look at it. One is your plumber
in Chicago, and the other is your cousin in London.
But your phone's software is so rigid that it
forces them into a single contact file. Right.
So now you have this Frankenstein John Smith
contact that mixes up their addresses, their
jobs, and their personalities. If you ask the
algorithm about John Smith, it's going to give
you a very confused, blended answer. That analogy
hits the nail on the head. If you feed glove
the word bank, it creates one vector that averages
out the context of a river bank and a financial
bank. It just blends them. Yes. It cannot distinguish
between the two based on the immediate sentence
structure because the vector address is permanently
fixed. And that limitation is precisely why glove,
despite its brilliance, was eventually superseded.
Right. Because technology moves incredibly fast.
The source notes that by 2022, the industry had
essentially left glove behind. The state of the
art in natural language processing moved on to
transformer -based models. Like BERT. Exactly,
like BERT. These newer models solved the homograph
problem by adding multiple neural network attention
layers on top of the word embeddings. So they
don't just assign a static vector anymore? No.
They dynamically adjust the word's mathematical
meaning based on the entire sentence around it,
every single time it's used. So what does this
all mean? We started this deep dive looking at
the messy poetic nature of human vocabulary.
And what we found in GloVe was the vital stepping
stone that proved a concept. It really did. It
proved that meaning, this abstract, uniquely
human experience, could actually be solved as
a geometry problem. By simply counting how often
words keep company and compressing the ratios
of those relationships into a map of coordinates,
we taught machines to understand the difference
between ice and steam. It's going to legacy.
And you should care about this because every
time your phone autocompletes a text message
or every time you chat with a modern AI assistant,
you are interacting with the direct descendants
of GloVe's global vectors. This raises an important
question, though. We've seen how Glove learns
meaning purely by observing the statistical proximity
of words in a massive text corpus. Right. But
that corpus, that giant alien spreadsheet, was
written by us. Uh -oh. Yeah. So if an algorithm's
entire understanding of the world is based on
the company our words keep online, what historical
biases, hidden cultural prejudices, or completely
irrational human associations are permanently
baked into those mathematical distances without
us even realizing it? Wow, if the aliens' only
dictionaries are sprite feed, they learn our
flaws right along with our facts. Precisely.
It's a fascinating mathematical permanence of
human quarks, something to really think about
the next time your AI suggests the perfect word.