You know, if I were to suddenly just like...
jump out from behind a door and yell boo at you
right now, you'd be surprised. I would absolutely
jump. Right, your heart rate would spike, you'd
probably jump back a foot or two, maybe drop
whatever you're holding. It's this very visceral
emotion, it's messy, it's physiological, and
it is just a deeply human experience. Yeah, and
it's also a completely subjective experience,
like you can't look at someone who just got jump
scared and say, ah yes, my sensors indicate you
were exactly 4 .7 units surprised by that. Exactly.
But what if you actually could? What if surprise
wasn't just this fleeting feeling, but a cold,
hard, mathematically rigorous metric? Like a
specific number you could calculate to the decimal
point to measure exactly how confused or uncertain
or just utterly blindsided a system is. It does
sound a bit like science fiction, right? Intempting
to quantify confusion. It really does. But today,
we are looking at the secret invisible metric
that dictates how every single artificial intelligence
on your phone actually thinks we are diving into
the math of surprise. Yeah, and yet it is the
foundational mathematics powering the language
models, the predictive text, and really the AI
we interact with every single day. We're talking
about a concept known as perplexity. Welcome
to today's deep dive. Our mission for you today
is to demystify exactly how data scientists,
statisticians, and modern AI mathematically measure
this concept of uncertainty. So we're going to
strip away the magic of AI and just look at the
hard numbers. OK, let's unpack this. So to really
grasp this, we have to... step away from the
massive, multi -billion parameter neural networks
for a minute. Right. Take a step back from the
chat GPTs of the world. Exactly. If we want to
understand what perplexity actually means in
information theory, we have to ground the concept
in everyday physical reality. And surprisingly,
this isn't some new AI fad born in the 2020s.
It was actually originally introduced back in
1977. Wait, 1977? That is essentially the Stone
Age in tech time. I mean, we're talking about
the era of the Apple II and, like, the scope.
It really is, which just highlights how fundamental
the math is. It was introduced in the context
of early speech recognition by a team of researchers
at IBM, Frederick Jelinek, Robert Leroy Mercer,
Lola Ball, and James Baker. OK, so a whole team
trying to figure this out. Right. They were trying
to get computers to understand human speech,
and they just hit a wall. They needed a reliable
way to measure the raw difficulty of a speech
recognition task. Specifically, they needed to
measure uncertainty for a discrete probability
distribution. A discrete probability distribution?
Okay, I want to make sure we don't lose anyone
in the jargon right out of the gate here. We're
essentially talking about things like flipping
coins and rolling dice, right? Like things with
a set countable number of outcomes. That is the
perfect starting point, yeah. The mathematics
of perplexity dictate that a fair coin toss has
a perplexity of exactly two. A fair six -sided
die roll has a perplexity of exactly 6. Okay.
So generally speaking, if you have a probability
distribution with exactly n outcomes, and every
single outcome has an equal probability, so a
1 and n chance, the perplexity is simply n. Got
it. So if I'm trying to visualize this, if I'm,
say, k -ways perplexed, I picture it like standing
at a crossroads. Oh, I like that. Yeah. Like,
if I'm standing at a fork in the road with three
perfectly identical paths to choose from and
no signs telling me where to go, I am three ways
perplexed. Exactly. If a random variable has
a uniform distribution over k outcomes, it has
the exact same level of uncertainty as me rolling
a fair k -sided die. So if I have 20 equally
viable choices, my perplexity is 20. It's just
that feeling of having options, but absolutely
no clue which one is going to happen. What's
fascinating here is how this connects to a broader
concept called information entropy. Perplexity
isn't just a basic tally of how many paths are
at your crossroads. It is mathematically defined
as the exponentiation of information entropy.
Whoa, okay, big words. Yeah, I know. But basically,
entropy is a way to translate that feeling of
uncertainty into a concrete number based on logarithms.
Okay, entropy and logarithms. Walk us through
how that actually works under the hood, because,
I mean, why do we need logarithms to understand
a simple dice roll? It really comes down to how
we measure information itself. Claude Shannon.
who was basically the father of information theory,
figured out that we can measure the surprise
of an event in specific units. OK. And depending
on the base of the logarithm you use for the
calculation, the units change. If you use base
two, the entropy of the distribution is measured
in Shannon's, or much more commonly, bits. Bits,
like computer bits. Exactly. And if you use the
natural logarithm, base e, It's measured in nats.
OK, so entropy tells me how many bits of information
I'm missing before the die lands. Precisely.
But talking about having, say, 2 .58 bits of
uncertainty is incredibly abstract for a human
brain to process. Yeah, I have no idea what 2
.58 bits feels like. Right, and that is exactly
where perplexity comes in. Perplexity takes that
abstract entropy measurement and exponitiates
it. It basically reverses the logarithm. It turns
those bits back into a number that feels intuitive
to us. Oh, I see. It converts 2 .58 bits back
into six, meaning you are as confused as someone
rolling a six -sided die. The larger the perplexity,
the harder it is for an observer to guess what's
going to happen next. That feels incredibly clean
and logical when everything is fair and uniform.
A six -sided die is six, a coin is two. But the
real world isn't a fair casino, like dice are
loaded, coins are scuffed. Can this metric actually
handle non -uniform situations where some outcomes
are way more likely than others? It absolutely
handles non -uniform distributions. In fact,
that's its primary purpose. But this is exactly
where our human intuition starts to completely
clash with the mathematics. Here's where it gets
really interesting because I would naturally
assume that perplexity is just a mirror image
of how likely I am to guess correctly. Like,
if I am playing a game where I have a massive
probability of winning, my uncertainty, my perplexity
should be almost zero, right? And that is the
exact trap a lot of people fall into. Perplexity
is a measure of the difficulty of a prediction
problem as a whole. It is absolutely not just
a straightforward representation of the probability
of success on a single guess. Yeah, I actually
stumbled over this totally counterintuitive math
paradox when I was looking into the source material.
Let's imagine a scenario where you have two choices.
One of those choices has a 0 .9 probability of
happening. That's a 90 % chance. The other outcome
is a 10 % chance. If I always bet on the 90 %
outcome, my probability of making a correct guess
is 0 .9. I would feel incredibly confident. I
mean, I'd bet my my house on it. You'd assume
your uncertainty is minimal. But the math tells
a very different story here. If you actually
calculate the perplexity for that 90 -10 split,
you don't get a number that cleanly maps back
to 90%. Wait, let's actually run the numbers
on that. How does the perplexity formula process
a 90 % probability? Well, it uses those logarithms
and negative exponents we mentioned earlier.
For the 90 % outcome, you take 0 .9 to the power
of negative 0 .9. OK. Then you multiply that.
by the 10 % outcome, which is 0 .1, to the power
of negative 0 .1. And when you compute all that,
the perplexity spits out a value of 1 .38. 1
.38. OK, so I am 1 .38 ways perplexed. But if
I try to reverse engineer that back into a probability
percentage like taking the inverse, 1 divided
by 1 .38, I get 0 .72. That is 72%. It's absolutely
not 90%. My actual chance of success is 90%,
but the perplexity metric is actually like my
odds are only 72%. Why is there this huge like
18 % disconnect between my odds of winning and
the mathematical measure of my surprise? It goes
right back to what entropy actually measures.
We have to look at something called optimal variable
length coding. That sounds incredibly dense.
It does, but just think of it like Morse code.
In Morse code, the most common letters in the
English language, like E, get the shortest possible
code. Just a single dot. Rare letters, like Z,
get these long, complex codes to keep them distinct
without wasting time on the common stuff. OK,
so you design a system that uses the least amount
of energy for the most frequent events. Yes.
Entropy. and by extension perplexity, measures
the expected or average number of bits required
to encode the outcome using that kind of optimal
code. It provides insight into the information
gain expected when you finally learn the outcome.
Information gain. Yeah, think about it. If you
bet on the 90 % outcome and you win, you don't
really learn very much. You expect it to win.
10 % of the time, you are going to lose. And
when you lose that bet, the surprise is massive.
You gain a ton of information about the unpredictability
of the system. Oh, wow. I see. So the perplexity
is evaluating the entire shape of the uncertainty.
It's factoring in the catastrophic surprise of
that 10 % chance happening. It's not just handing
me the raw odds of winning a single bet. It's
this holistic measure of the inherent chaos in
the environment I'm betting in. Exactly. And
that distinction becomes critical when scientists
use this math in the real world. Because in the
real world, whether we're predicting the stock
market or trying to get a machine to translate
French to English, the true underlying probabilities
of the universe are completely unknown. We don't
have a cheat sheet. Right. We don't. So if we
don't know the actual odds of the environment,
how do data scientists actually use this metric?
They use it to evaluate models. Let's say there's
an unknown true probability distribution out
there in the world. We'll call this true reality
P. OK, true reality is P. We don't know P, but
we really want to predict it. So we gather a
bunch of data, a training sample drawn from P,
and we build a mathematical model based on that
data. We'll call our proposed model Q. So P is
actual reality, and Q is our AI's best guess
at how reality works. Exactly that. Now, to figure
out how good our model Q really is, we obviously
can't test it on the data it already learned
from. That would be cheating. Right. It already
knows the answers. Yeah. So we have to see how
well it predicts a completely separate test sample,
like a fresh batch of data that was also drawn
from the real world. P. You know, this immediately
reminds me of being a student. Like, think of
a low -perplexity model like a student who genuinely
studied the underlying concepts for an exam.
Oh, that's a great analogy. Right. So the unknown
distribution, P, is the teacher's actual exam
paper. The model, Q, is the student's brain,
filled with the logic they developed from doing
practice problems. When that student sits down
to take the final test, the new test sample,
if their mental model of the subject is good,
they assign a really high probability to the
questions that appear. They aren't blindsided.
Right. They are simply less surprised by the
test sample. And mathematically, a better model
assigns higher probabilities to the events that
actually occur in the test data. Because it predicts,
well, It is fundamentally less surprised by what
reality throws at it, which results in a lower
perplexity value. Low perplexity models do a
better job of compressing the test sample. Compressing
it like a zip file on a computer. Very similar,
actually. Because the model already anticipated
the patterns in the data, it requires fewer bits
per test element on average to encode the information.
That makes total sense. And if we connect this
to the bigger picture, the exponent. in these
complex mathematical formulas actually represents
something called cross entropy. Cross entropy.
Cross entropy looks at the empirical distribution
of the test sample, what actually happened in
reality, and compares it to what our model Q
predicted would happen. So it's like a direct
comparison between the student's expectations
and the teacher's grading key. Yes, exactly.
And what that actually means on paper is defined
by a concept called Kullback -Libler divergence,
or KL divergence. OK, KL divergence. KL divergence
is essentially a penalty. It measures the distance
between two probability distributions. It calculates
the extra wasted bits of information you're forced
to use simply because your model Q is slightly
wrong about reality P. OK, so if the student's
brain Q is a perfect match for the teacher's
exam P, the distance is zero. There is no penalty.
Yes. And that is the only scenario where perplexity
is completely minimized. The divergence between
x expectation and reality drops all the way to
zero. OK. We have covered flipping coins, 90
-10 math paradoxes, and evaluating how well a
model predicts a sequence of test events. But
rolling a die or taking a multiple choice test
is one thing. How on earth does this math scale
up to a whole paragraph? Like, how does this
power natural language processing or the large
language models like chat GPT that are generating
thousands of words in a row? Well, in natural
language processing, or NLP. We aren't just predicting
a single isolated event anymore. We are evaluating
entire sprawling text documents. We look at a
corpus, which is a massive structured collection
of texts. And a language model is essentially
a probability distribution mapped over those
entire texts. But documents are incredibly messy.
Like you might have a test sample that's a three
word text message and another that's a 300 page
historical biography. How can you possibly compare?
the perplexity of a model on those two wildly
different things. You're right. You can't compare
them raw. You have to normalize the math. And
this brings us to token normalized perplexity.
Token normalized perplexity? Right. In NLP, a
token is usually a single word or sometimes just
a piece of a word. To make meaningful comparisons
across different lengths of text, you calculate
the overall perplexity of the document, and then
you normalize it by the total number of tokens.
So we're basically boiling the AI's massive confusion
down to a per word average. What does that actually
look like in practice? There was a really famous
historical benchmark for this using the Brown
Corpus. The Brown Corpus? Yeah, the Brown Corpus
is a collection of one million words of American
English compiled to cover all sorts of varying
topics and genres. Back in 1992, the absolute
state -of -the -art Lois published perplexity
on the brown corpus was about 247 per token.
Okay, so what does this all mean? If I'm a computer
scientist in 1992 and my AI gets a perplexity
of 247, what is actually happening inside the
machine? Well, think back to our dice analogy.
An AI with a token normalized perplexity of 247
is just as confused when looking at the test
data as if it had to choose uniformly and independently
among 247 different, equally likely possibilities
for every single word it tries to predict. Whoa!
It's rolling a 247 -sided die for every single
word in a sentence. That sounds agonizingly difficult.
I mean, if you just naively guessed out of 247
options, your accuracy rate would be 1 divided
by 247, which is about 0 .4%. You would be wrong
99 .6 % of the time. It sounds like a terrible
model, doesn't it? But there is a brilliant nuance
here that underscores exactly why we can't just
blindly trust a single mathematical metric. Oh,
I love this part from the reading the the loophole
yes the math shows that if your perplexity is
247 naive guessing gets you a 0 .4 percent accuracy
rate but if you completely throw that sophisticated
model in the trash and you just program a dumb
machine to blindly guess the word the T H E for
every single word in the English language you
will actually achieve an accuracy rate of 7%.
It's a massive discrepancy. 0 .4 % versus 7%.
And it highlights the deeply nuanced nature of
predictiveness. I mean, how is a blind, repetitive
guess of the mathematically outperforming a state
-of -the -art 1992 AI model metric? Yeah. Why
shouldn't we just build an AI that screams the
all day? Because it comes down to the types of
statistics the models are actually utilizing.
Guessing the word the constantly is based on
unigram statistics. Right. A unigram just looks
at the frequency of single words in total isolation.
V is the most common word in the English language.
So purely by brute force, it hits 7 % of the
time. But the model that achieved the 247 perplexity
score in 1992 wasn't utilizing unigrams. It was
utilizing a trigram statistic. OK. A trigram
looks at sequences of three words. It's actually
looking at the context. Yes. A trigram model
tries to predict the next word based on the context
of the two preceding words. It is trying to learn
the actual grammatical structure, the rhythm,
and the flow of the language. It isn't just firing
out the most common word regardless of context.
So guessing the is extremely safe and gets a
higher raw accuracy percentage on paper, but
it's incredibly boring and completely useless.
The trigram model is actually far more sophisticated
because it's mapping the true complexity of human
language. It takes risks. Utilizing trigram statistics
fundamentally refines the prediction in a way
unigrams never could, even if the model is still
faced with 247 equivalent choices at every step.
That's the key. And obviously the field has moved
far beyond 1992 trigram models. Since around
2007, deep learning has taken over entirely.
Today, we have these dominant transformer models,
Google's BERT, OpenAI's GPT -4, all these massive
large language models. And token normalized perplexity
is still a central tool for evaluating them.
Engineers use this metric to compare different
models on the exact same data set and to mathematically
guide the optimization of the AI's internal settings,
its hyperparameters. But as these models have
gotten larger and more complex, haven't researchers
discovered limitations to using perplexity as
the ultimate North Star? I mean, it can't be
perfect. They absolutely have. It is a powerful
tool, but it is deeply flawed if used in a vacuum.
First off, it's highly sensitive to linguistic
features and sentence length. You cannot just
blindly compare a perplexity score from a dataset
of medical journals to a perplexity score from
a dataset of Twitter posts and expect a perfect
apples -to -apples comparison. The intrinsic
entropy of those two environments is entirely
different. That makes total sense. The baseline
surprise of a dense medical journal is obviously
going to be different from a random tweet. Furthermore,
it turns out perplexity is an inadequate predictor
of actual performance in the real world, particularly
in tasks like speech recognition. Wait, really?
Yeah. You might engineer a model that achieves
a mathematically stunning ultra -low perplexity
score. The KL divergence is incredibly low on
the test set. But when you hook it up to a microphone
and have a real hewn speak to it, it might still
hallucinate, misinterpret accents, or just fail
to transcribe accurately. So how do they reality
check the math, then, if perplexity isn't enough?
Researchers often have to rely on a simpler,
alternative evaluation metric called word error
rate. or where. Word error rate. Okay, that sounds
much more grounded. It's simply taking the percentage
of erroneously recognized words, so deletions,
insertions, substitutions, and dividing it by
the total number of words spoken. Exactly. It's
a harsh reality check against the purely theoretical
clean math of perplexity because there is a massive
danger in AI development. If you blindly optimize
your system just to achieve the absolute lowest
perplexity score possible, you run into severe
issues with overfitting. Overfitting, meaning
the AI essentially just memorizes its training
data. Going back to our student analogy, it's
like a student who memorized the answer key to
the practice test without actually understanding
the underlying subject. They get zero perplexity
on the practice exam, but fail miserably when
they face a question worded slightly differently
in the real world. Right. They completely lose
their ability to generalize to new unseen information.
The AI becomes brilliant at predicting the past
and totally useless at navigating the future.
Wow. So to wrap all of this up, you have gone
on quite a mathematical journey with us today.
We started with the simple uncertainty of rolling
a fair six -sided die. We untangled the paradox
of why predicting a 90 % probability still carries
a mathematical penalty of surprise due to variable
length coding. We learned how KL divergence measures
the distance between a model's guess and true
reality. And finally, we scaled all that math
up to see how the most advanced large language
models in the world measure their own confusion
on a per -word basis. Next time you see an article
or a headline about a brand new AI model, and
the companies boasting about their groundbreaking
new benchmark stats, you know exactly what perplexity
means. It is the AI's internal mathematical measure
of surprise. And more importantly, you know exactly
why a single metric, no matter how low it gets,
doesn't tell the whole story of how well an AI
actually understands human language. And this
raises an important question. We've established
that data scientists and engineers are constantly
trying to minimize an AI model's perplexity.
They are tweaking the hyperparameters so that
the AI is trained to never be surprised by a
sequence of human words. It should perfectly
anticipate everything we say, reducing its cross
-entropy to the absolute minimum. But if we mathematically
train a system to ruthlessly eliminate all surprise,
Are we inadvertently mathematically stripping
away the model's ability to ever generate truly
surprising original or creative thoughts? Wow,
that is definitely something to mull over. Because
while an AI might be rigorously engineered to
be a perfect unsurprised predictor of text, when
a person jumps out from behind a door, that subjective,
unpredictable human surprise is sometimes exactly
what makes life interesting.