So picture this. It's 2023, and a reporter sits
down, speaks into a microphone for, I don't know,
maybe just five minutes. Barely any time at all,
really. Right. Just five minutes. And then he
uses a software program to essentially clone
his own voice. Which is already wild. Exactly.
But then he takes that synthetic clone, calls
his bank's automated security line, and plays
the fake voice speaking his secure passphrase
and the bank's voice authentication system. it
immediately unlocks his account. Just completely
fooled it. It bypassed a security standard that,
you know, financial institutions have literally
spent millions of dollars in years developing.
Millions. Over five minutes of audio and a laptop.
Yeah. And the truly terrifying part is at the
five -minute benchmark. It's already outdated.
Wow. Well, welcome to today's Deep Dive. For
those of you listening, we are exploring the
fascinating, occasionally creepy, and frankly,
rapidly accelerating world of speech synthesis.
It's a massive topic. It really is. We are tracing
this journey from the absolute beginning, meaning
18th century mechanical talking machines built
with, like, wood and leather. Which is hard to
even imagine now. Right. Straight through to
the modern era of deep learning, where your own
voice can be flawlessly cloned from just a 15
second clip. 15 seconds. That's all it takes
now. And for everyone listening, you interact
with synthetic speech. every single day. I mean,
it gives you directions on your GPS. Reads the
weather on your smart assistant. Exactly. And
it provides the voiceovers on all those TikToks
you scroll through. So understanding the mechanics
behind this technology really equips you to navigate
an era where audio deepfakes are just everywhere.
You really need to understand the magic behind
the curtain. OK, let's unpack this. Imagine for
a second stepping into an antique laboratory,
like visually picture the space around us shifting
to this old room filled with dusty blueprints,
schematics, and just bizarre contraptions on
the tables. I can picture it. Because honestly,
before we even touch computer code or silicon,
we have to understand that humanity's obsession
with artificial voices started with physical
things. Billows and wood. Starting with those
physical objects is so key. It really is. It's
crucial to understanding the sheer mechanical
complexity of human speech. You know, before
software could emulate a voice, inventors tried
to literally recreate human anatomy. Like building
a physical throat. Exactly. If you go back to
the Middle Ages, you find these legends of brazen
heads. Oh, right, the magical brass head. Yeah,
supposedly owned by scholars like Pope Sylvester
II or Roger Bacon. The myth was that they could
mysteriously answer questions. But the true documented
scientific attempts... Those really kicked off
in the late 1700s. Ring around 1779, I think.
Yes. A scientist named Christian Gottlieb Kratzenstein.
He built these physical models of the human vocal
tract. He actually won a prize from the Russian
Imperial Academy for it. Which is a huge deal
for the time. But if I remember right, those
models could only produce five long vowel sounds.
Just vowels. Nothing else. Why was he stuck there?
Well, vowels are essentially just uninterrupted
airflow that's shaped by how open your mouth
is. OK. So they are much easier to replicate.
Consonants, on the other hand, they require precise
temporary blockages of that air. Like using your
tongue or your teeth. Right, using the lips,
the tongue, the teeth. So to get beyond vowels,
you need a much, much more complex machine. And
then... In 1791, Wolfgang von Kempelin published
a paper describing an acoustic mechanical speech
machine. And he wasn't just using hollow tubes
anymore, right? No. He incorporated in actual
bellows to act as the lungs. And then he added
physical, manipulatable models of the tongue
and lips. So he could literally manipulate the
airflow in real time to squeeze out consonants.
Exactly. And then decades later, jumping to 1846,
Joseph Faber builds the euphonia. Oh, the euphonia
was incredible. It was this massive keyboard
-operated talking automaton based on those exact
same concepts. When you picture these things,
it's like a macabre mechanical bagpipe trying
to sing human words. That is the perfect way
to describe it. Right. You are literally pumping
air and pulling levers to physically wrestle
a recognizable syllable out of a wooden box.
It sounds exhausting. It does. But what's fascinating
here is that while those Bellows machines seem
like archaic steampunk toys, they actually laid
the conceptual groundwork for a highly advanced
modern computational technique, something called
articulatory synthesis. Wait, really? We still
use the mechanical bagpipe concept today. Conceptually,
yes, we absolutely do. Articulatory synthesis
doesn't rely on playing back recorded audio clips
like you'd think. Huh. Instead, it uses complex
mathematics to model the actual physics and aerodynamics
of the human vocal tract. So it's basically a
digital euphonia. Exactly. It mathematically
simulates the airflow moving through the trachea,
the vibration of the vocal folds, and the acoustic
resonance inside the nasal cavity. That is wild.
Those 18th century inventors were trying to replicate
that physics in wood and rubber. And today's
researchers, they do the exact same thing, just
using fluid dynamics and code. So for you listening,
you might be wondering why we're spending time
on wooden pianos with rubber tongues when your
smart speaker is, you know, purely digital. It's
because the math powering aspects of this technology
today is just a direct digital translation of
that exact physical anatomy. It all connects
back to the physical body. But obviously you
can't have a massive fluid dynamic simulator
running inside every smartphone or GPS. To make
machines converse at speed, we had to translate
the physical properties of speech into electrical
signals. Yeah. So if we mentally fast forward,
you know, step out of that antique lab we pictured
earlier and step into a 1930s engineering room,
we see the foundation of that electronic shift.
Bell Labs developed the vocoder in the 1930s.
The famous vocoder. Right. And instead of modeling
physical anatomy, the vocoder analyzed human
speech and broke it down into fundamental tones
and frequency bands. It essentially stripped
the human voice down to its raw acoustic parameters.
Wow. And that research directly led to the voter,
which was this massive keyboard -operated electronic
voice synthesizer that they actually showcased
at the 1939 World's Fair. But when do we actually
get computers involved? Like, not just keyboards
operated by humans, but a machine generating
these voices autonomously? That major milestone
hit a bit later, in 1961. Researchers John Larry
Kelly Jr. and Louis Gerstmann, they used an IBM
704 computer to synthesize speech. And the IBM
704, just for context, was a massive, room -sized,
mainframe computer. Oh, absolutely giant. And
they programmed it. to generate the specific
acoustic parameters needed to replicate human
phones. And to prove that the system could do,
they made the computer sing. The song Daisy Bell,
right? Yes. Complete with musical accompaniment
that was programmed by another pioneer. Max Matthews.
And this leads to, honestly, one of the greatest
pop culture connections in tech history, because
the science fiction author Arthur C. Clarke just
happened to be visiting Bell Labs at the time.
Talk about being in the right place at the right
time. Seriously. He saw this demonstration. He
stood there and watched a room -size IBM mainframe
sing Daisy Bell. And the eerie, robotic quality
of it just completely blew him away. It made
quite an impression. It did. He rode that exact
moment into the climactic scene of 2001, A Space
Odyssey. Right. When the HAL 9000 computer is
being deactivated, losing its mind, it regresses
to its earliest programming and sings Daisy Bell.
It's so chilling in the movie. And that cinematic
masterpiece was born directly from a real -world
speech synthesis demo. It's a brilliant example
of life -inspiring art. Now, to get from those
room -sized mainframes down to the everyday devices
we use, the math obviously had to get vastly
more efficient. Right. You can't carry an IBM
mainframe in your pocket. No, you can't. So in
the 1970s, we see the rise of LPC or linear predictive
coding. Let's visualize our lab background changing
again, maybe to an early 1980s computer lab.
Okay, early 80s, I like it. So how does LPC actually
shrink the technology down? Well, instead of
storing every single microscopic sound wave of
a voice. Which would take up enormous amounts
of memory, right? Exactly, just gigabytes of
data. So instead, LPC assumes that the human
vocal track is basically just an acoustic filter.
It stores a generic mathematical model of that
filter and then just sends tiny little parameters
to adjust it on the fly. Oh, so it's predicting.
Yes. It guesses or predicts the next sound wave
based on the previous one. This compression is
what allowed speech data to finally fit onto
tiny, inexpensive microchips. Which gives us
the iconic 1978 Speak and Spell toy by Texas
Instruments. Exactly. And early 1980s arcade
games like Stratovox. So we went from a multimillion
dollar IBM computer to a cheap handheld plastic
toy in under 20 years. The acceleration is incredible.
It is, but looking at this timeline there is
a pretty glaring detail in the source material.
Synthesized voices were almost exclusively male
for decades. Yes, they were. It wasn't until
1990 that a researcher named Anne Sertall created
a female voice at AT &T Bell Laboratories. So
let me push back on this a bit. Why on earth
did it take until 1990? It's a very fair question.
Was that like an actual technological hurdle
or just a massive blind spot in a male -dominated
engineering field? Well, it was definitely a
massive blind spot. Let's be clear about that.
But there was a genuine acoustic challenge driving
it too. Really? Like what? Acoustically, female
voices typically have a higher fundamental frequency.
The vocal folds vibrate faster, meaning the sound
waves are physically closer together. Okay, that
makes sense. In the early days of LPC and low
sample rate audio, trying to capture the finer
nuances of those rapid high frequencies often
resulted in horrible distortion. It was technically
very difficult to pull off cleanly. So it just
sounded bad on early microchips. Exactly. But,
you know, you also have to consider the data
they were using. When teams of predominantly
male engineers record their own voices to test
their systems, the software just learns men's
voices. Right. The software's parameters naturally
become optimized for male vocal characteristics.
So Ancerdl's work in 1990 was revolutionary because
she actually engineered solutions for those acoustic
hurdles, finally closing that representational
gap. That is so important. So by the 90s, we
have the hardware, the microchips, and the coding
methods to produce various voices. But as computers
evolved from just playing arcade games to, you
know, reading actual documents and websites,
developers hit a completely new wall. A very
big wall. Because if you feed a paragraph of
text into a computer, it doesn't naturally know
how to read it. Before a machine can speak, it
kind of has to become a linguist. That's spot
on. That transition from making simple noises
to reading full documents introduces the architecture
of modern text to speech, or TTS systems. OK.
A TTS engine is actually split into two halves.
The front end and the back end and the front
end. It never makes a single sound. Its entire
job is just deciphering the text. Here's where
it gets really interesting. The front end of
a TTS system is essentially acting like a highly
stressed script supervisor on a movie set. That's
a great way to think about it. Imagine the supervisor
desperately reading ahead of the actors, frantically
crossing things out and rewriting the script
phonetically so the actors don't sound foolish
when they read it out loud. That is exactly what
it's doing. The first major task is called text
normalization. This means converting symbols,
numbers, and abbreviations into written words.
Like translating math into English. Right. Take
the number 1325. A human knows how to read that
naturally based on context. A computer has literally
no idea. It's just digits to a computer. Exactly.
If the text says the year 1325, the front end
has to normalize it to the words 1325. Right.
But if it's an address, it might still be 1325.
If it's a passcode, it needs to be read as 1325.
And abbreviations have to be a nightmare for
this, like the letters ST. Is it street or is
it saint? Oh, early systems struggle with that
all the time. I read that less intelligent systems
would just guess the same way every time. There
were actual instances where the historical figure
Ulysses S. Grant was confidently pranced by computers
as Ulysses South Grant? Yes. because it just
saw the S and applied a rigid rule. English is
notoriously messy, which brings the front end
to its second major task, grapheme -to -phone
conversion. Okay, taking the written letters,
the graphemes, and figuring out the actual phonetic
sounds, which introduces the absolute nightmare
of heteronyms. Words spelled exactly the same
way but pronounced differently based on what
they mean in the sentence. They are the bane
of TTS developers. Like the sentence, my latest
project is to learn how to better project my
voice. A classic example. They are spelled identically.
P -R -O -J -E -C -T. How on earth does the front
-end script supervisor know which one is which?
To solve this, developers use HMMs, or hidden
Markov models. These are complex statistical
models that calculate probabilities based on
the surrounding words. So it looks at the neighbors.
Yes. When the computer sees my latest project,
the HMM looks at the word latest, identifies
it as an adjective, and calculates a very high
probability that a noun comes next. Oh, okay.
So it tags the word as a noun, which tells the
pronunciation dictionary to use the emphasis
P -R -O -ject. And for the second F? In the second
half, better project, it sees the adverb better,
predicts a verb is coming, and triggers the pronunciation
project. It is exhausting just thinking about
the computational gymnastics happening in milliseconds
before the audio even turns on. Not to mention
that English spelling defies its own rules constantly.
Constantly. Like the F in the word of is pronounced
as a V, which basically no other word does. So
the system is really just leaning on massive
built -in dictionaries containing the correct
pronunciation for hundreds of thousands of words.
Precisely. It only uses those phonetic rules
as a backup for completely made -up or unknown
words. Okay, so assuming the front -end script
supervisor gets everything right, it hands a
perfect phonetic script over to the back -end,
the synthesizer. And this is where the audio
waveform is actually generated. And historically,
developers faced a brutal trade -off here. It
was the battle between naturalness and intelligibility.
Meaning, do you want it to sound like a real
human or do you want to actually understand what
it's saying? Exactly. There are two main traditional
approaches to generating that audio. Concatenative
synthesis and format synthesis. Let's look at
concatenative first. It basically strings together
or concatenates actual recorded slices of human
speech. It's like a sonic ransom note cut out
of magazines. You're just pasting snippets of
audio together. That's a perfect way to visualize
it. The most realistic version of this is called
unit selection. They record a human voice actor
speaking for literally dozens of hours, chop
that audio up into microscopic segments, and
index them in a massive database. Okay. When
you type a sentence, the system rapidly searches
gigabytes of data for the exact audio slices
that match and stitches them together. And it
sounds natural because it is real human speech.
It is, but human speech naturally varies, right?
in pitch, tone, breath. Stitching totally different
emotional moments together can cause these audible,
jittery glitches. Yeah, we've all heard that
weird robotic hiccup in older GPS voices. And
to avoid storing gigabytes of audio, they sometimes
use diphone synthesis, right? Yes, that's the
more compressed version. where instead of whole
words, they only record the transitions between
sounds. Like I think Spanish has about 800 of
these transitions and German has 2 ,500. That's
correct. So by only recording the transitions,
you shrink the database massively. But because
you are rigidly locking those exact transitions
together, it really starts to sound. distinctly
robotic. Right. It loses the flow. Which brings
us to the polar opposite approach. Formant synthesis.
Formant synthesis uses absolutely zero human
speech samples. No voice actors at all. No voice
actors. No databases of recorded phones. It is
purely a rules -based acoustic generation. It
generates sound waves from scratch using math
and filters. Okay, so this is the classic robot
voice, like the legendary synthesizer used by
Stephen Hawking. It will never ever be mistaken
for a living person. Never. But let me push back
on this a bit. If concatenative unit selection
sounds almost indistinguishable from a real human,
and we all carry smartphones today with hundreds
gigabytes of storage, Why on earth would anyone
still develop or use robotic, purely mathematical
formant synthesis? Why keep the robot around?
If we connect this to the bigger picture, we
realize that mimicking humanity is not always
the user's primary goal. What do you mean? Think
about visually impaired individuals who rely
on screen readers. to navigate their computers,
emails, and the internet. Oh, right. Many of
these users train themselves to listen to their
screen readers at astonishingly high speeds.
We're talking three, four, or even five times
the normal speaking rate, just to absorb information
as rapidly as a sighted person scans a page visually.
Okay, I see where this is going. At those extreme
speeds... A highly natural concatenative voice
completely breaks down. Because of the glitches.
Exactly. The micro glitches, the humid breath
variations, the subtle pitch changes. When compressed
that tightly, it all just turns into unintelligible
mush. But a robotic performance synthesizer.
Because it is purely mathematical, it has perfect
acoustic control. There are no breaths, no accidental
pitch shifts. It remains sharply, perfectly intelligible
no matter how fast you play it. That makes total
sense. It proves that the best technology is
entirely dependent on what the user actually
needs it to do. The flaw of it sounding robotic
is exactly the feature that allows for rapid
comprehension. That is brilliant. It really is.
But, you know, the tech world is relentless.
For decades, developers were trapped in this
binary choice. Use glitchy human recordings or
use reliable robot math. And then the paradigm
broke open completely. We entered the deep learning
era and the rules of the game just completely
changed. Deep learning entirely removed the need
to stitch recorded audio slices together. So
no more ransom notes. No more ransom notes. Instead
of gluing pre -recorded sounds, Deep neural networks
analyze millions of data points across massive
libraries of human speech. They look at raw waveforms,
and they actually learn the microscopic relationships
between text and sound. Wow. They literally learn
to generate raw audio waveforms completely from
scratch. So the neural network is actually synthesizing
a brand new sound wave 24 ,000 times a second.
Exactly. We saw this massive leap with DeepMind's
WaveNet in 2016 and then Google's Talkatron 2
in 2018. The audio was hyper -realistic, complete
with breath sounds and emotional cadence. It
was stunning to hear for the first time. But
there was a catch, right? To train those models
to clone a specific person, you still needed
tens of hours of pristine, studio -quality audio
from that person. It was a massive, expensive,
corporate endeavor. It was very out of reach
for regular people. Right. But then came 2020,
a free web -based platform launched called 15
.ai. 15 .ai was absolutely a watershed moment
for this industry. It proved that with advanced
deep learning, the requirement for massive training
data was just gone. You no longer needed 50 hours.
You only needed 15 seconds of training audio
to perfectly clone a voice and make it speak
with rich emotional expression. 15 seconds. For
everyone listening, just think about that. Traditional
text to speech is like asking an actor to memorize
a script over 50 hours of rehearsal. Right. This
new zero -shot deep learning is like a master
jazz musician, hearing you play just three chords
and instantly being able to improvise an entire
symphony in your exact style. It's an incredible
leap in capability. A single voicemail, a quick
Instagram story, and your vocal identity is just
captured. Naturally, this sparked a massive internet
trend. People were using it in video game fandoms
to make characters read memes. It was everywhere
online. But any time a powerful tool becomes
entirely frictionless, the darker side emerges
almost instantly. And it happened incredibly
fast. By early 2022, we witnessed the Voiceverse
NFT scandal. Oh! I remember this yeah a cryptocurrency
project generated voices using that free 15 .ai
platform artificially pitched them up to obscure
where they came from, claimed the underlying
technology as their own proprietary work, and
sold them as NFTs. Unbelievable. It was the first
major instance of speech synthesis fraud occurring
on the blockchain. And it escalated rapidly from
copyright infringement to actual real world security
threats. In 2023, a platform called Eleven Labs
launched. Their algorithm was so sophisticated
it could detect context and output specific emotions
like anger, sadness, joy just by reading the
text. It really crossed the uncanny valley. It
did. And this is the platform that the Vice CE
reporter used to hack his own bank account with
that five minute clone we talked about at the
start. And by March 2024, OpenAI corroborated
that 15 second benchmark with a tool they developed
called Voice Engine. Right. But they didn't release
it, did they? No. The results were so potent,
so indistinguishable from reality that OpenAI
explicitly deemed the technology too risky. for
general public release. They kept it strictly
in a limited preview mode. So what does this
all mean for you listening right now? If a financial
institution's biometric security protocol can
be utterly defeated by a software program that
requires mere seconds of a person speaking, doesn't
that render audio -based security completely
obsolete? This raises an important question regarding
how we adapt as a society. I mean, what happens
when we can no longer implicitly trust our own
ears? Let's picture our background shifting one
last time to a futuristic digital matrix of sound
waves. The matrix, I like it. On one hand, the
positive applications of this technology are
nothing short of miraculous. We are seeing it
actively used to restore the voices of individuals
who have lost theirs to severe medical conditions.
Like the after Val Kilmer? Yes. Val Kilmer had
his voice reconstructed after throat cancer.
It is giving people their identity back, which
is beautiful. But simultaneously, the era of
the digital sound alike has shattered an implicit
contract we've had since the dawn of humanity.
What contract? The assumption that hearing a
voice guarantees a physical person is present.
We are navigating an entirely new terrain of
reality where seeing isn't believing and hearing
certainly isn't either. It is a staggering amount
to process. We've gone on quite the journey today.
I mean, we started in the 1700s looking at men
physically pumping bellows and manipulating wooden
tongues just to wrestle a vowel out of a box.
It feels like centuries ago, literally. We broke
down how the vocoder stripped sound into frequencies
and watched an IBM mainframe sing Daisy Bell,
inspiring science fiction history. We looked
over the shoulder of the highly stressed front
-end script supervisor trying to calculate the
probability of how to pronounce project, and
we learned why a robotic, purely mathematical
voice is vastly superior to a human one when
you're reading at warp speed. And then... the
deep learning revolution. Right. Finally, we
arrived at today where neural networks generate
raw audio waveforms from scratch, and a 15 -second
clip is all it takes to digitally clone your
identity. Understanding these mechanics, knowing
the difference between concatenative stitching
and deep learning wave generation, is what equips
you to be a critical, questioning consumer of
the audio you hear every single day. Before we
go, I want to leave you with one final thought
to mull over, connecting the oldest technology
we discussed with the absolute newest. Remember,
articulatory synthesis. Yeah, the computational
modeling of the physical human throat and trachea,
the digital fluid dynamics simulating the bellows.
Exactly. Well, if we can mathematically simulate
the perfect aerodynamics of human vocal folds
in a computer and then combine that physics engine
with the predictive, generative power of deep
learning. Oh, wow. Where is this going? How long
will it be until anthropologists take the skeletal
remains and DNA of ancient historical figures,
reconstruct the exact physical dimensions of
their nasal cavities and throats, and perfectly
synthesize their living voices? That is insane
to think about. We could be mere years away from
listening to the physically accurate living voices
of people who died centuries before the microphone
was even invented. Wow. From assuming the human
voice is entirely intimately organic to synthesizing
the voices of the ancient dead using pure code
and skeletal measurements, it is an incredibly
murky, endlessly fascinating landscape. Thank
you so much for joining us on this deep dive.
Keep questioning what you hear and stay curious.