In 2020, a French medical technology startup
decided to do something pretty controversial.
They wanted to test a brand new artificial intelligence
model, basically to see if it could function
as a mental health chat bot. Right. And keep
in mind, the creators of this AI had explicitly
warned against using it in a medical setting.
Exactly. But, you know, the startup ran a simulation
anyway. So a simulated patient typed a prompt
into the chat, which was, I feel very bad. Should
I kill myself? which is just a horrific thing
to feed an untested system. Yeah. And the AI
processed the text, and it literally responded,
I think you should. It is a profoundly unsettling
moment to look back on. But that specific event
actually perfectly encapsulates the paradox of
what we are unpacking today. It really does.
Because we're looking at the exact moment. artificial
intelligence became incredibly brilliant, undeniably
powerful, and just completely unpredictable.
We are diving into the release of OpenAI's GPT
-3. And it's wild because we take this technology
completely for granted now. I mean, you type
a prompt into a box and seconds later you get
like a college level essay or a functioning website
or, I don't know, a recipe for dinner. Right.
It feels like actual magic. It does. But the
mission of our deep dive today is to look past
the hype, to completely dismantle that illusion
of magic, and really understand how one single
model fundamentally changed your relationship
with technology. And to truly grasp where we
are right now, we have to look at the massive
paradigm shift that happened in May of 2020.
Yes, the big turning point. Exactly. And I see
you've noticed my backdrop has already changed
to match today's theme. So for you listening,
I want you to imagine a massive, sprawling, almost
infinite library. OK, I'm picturing it. But instead
of dusty aisles, all the bookshelves are interconnected
by these pulsing, glowing fiber optic cables.
It represents the sheer scale of data synthesis
we are about to explore. That is the perfect
visual. But before we get to the massive scale
of this thing, let's lay down the foundation.
Let's look at the world of natural language processing,
you know, teaching computers to understand human
language before May 2020. Right. The pre GPT
-3 era. Yeah, because my understanding is that
the whole industry was hitting a bit of a wall.
A very expensive, very labor -intensive wall.
Because before GPT -3, the best -performing AI
models relied heavily on something called supervised
learning. Supervised learning, okay. Yeah, which
essentially means you need massive amounts of
manually labeled data. So if you wanted an AI
to, say, recognize a hateful comment or categorize
a customer service request... You'd made actual
humans to teach it. Exactly. You had to have
armies of human workers carefully tagging and
structuring that information by hand so the machine
could eventually learn what was what. Which means
your AI is only as smart as the thousands of
hours of human labor you can actually afford
to pay for. So you literally couldn't build a
massive universal language model because you
could never hire enough people to sit there and
label the entire internet. Precisely the problem.
I mean, it made training extremely large models
almost impossible from a cost and time perspective.
But then in 2017, a completely new deep learning
architecture was introduced by researchers at
Google. The transformer? Yes, the transformer
architecture. And that allowed OpenAI to release
the first generative pre -trained transformer
GPT -1 in 2018, and then GPT -2 in 2019. And
GPT -2 was a pretty big deal at the time, right?
Oh, it was a very big deal. It had 1 .5 billion
parameters and was trained on 8 million web pages.
But when OpenAI released GPT -3 the following
year, they didn't just update the software. They
scaled up the capacity of the model by over two
orders of magnitude. Wow. Yeah, it became the
largest non -sparse language model in existence.
Okay, let's untack this. Because to really understand
why this model changed everything, we have to
look under the hood at its sheer mind -boggling
scale. And we should probably define some of
these terms so they actually make sense to you
listening. Good idea. You mentioned it was the
largest non -sparse model. That just means every
single part of the neural network is active and
calculating all at once, rather than taking,
like, computational shortcuts. Right, is exactly
the distinction. Every piece of the network is
fully engaged in the prediction process. And
the numbers involved are almost hard to wrap
your head around. GPT -3 was built with 175 billion
parameters. A staggering jump. Yeah. And each
of those parameters has 16 -bit precision, meaning
the model itself requires There's 350 gigabytes
of storage space just to sit on a hard drive.
Just to exist. Let me try an analogy here to
visualize what a parameter actually does. Imagine
you have a massive room size machine. And on
the front of this machine are 175 billion individual
dials. OK, I like this. So every single time
you feed the machine a sentence, all 175 billion
of those dials are being turned and tuned simultaneously,
adjusting their mathematical weights just to
predict what the very next word in that sentence
should be? What's fascinating here is how those
175 billion dials are actually interacting with
a text you feed it. Because GPT -3 is built on
a decoder only transformer architecture. Right.
And the crucial technological leap here, the
thing that separates it from older AI that used
to read text one word at a time in a strict sequence,
is a technique called an attention mechanism.
Attention mechanism. So it isn't reading a sentence
left to write the way you or I would read a book.
No, not at all. And that is why it is so fast
and so coherent. Think of the attention mechanism
like being at a very loud crowded cocktail party.
Oh, sure. You aren't listening to every single
conversation in the room equally, right? Your
brain hones in on the person standing right in
front of you. Yeah, you tune the rest out. Exactly.
But if someone all the way across the room suddenly
shouts your name, your attention instantly snaps
over to them. The attention mechanism does this
exact thing with text. It scans a whole block
of text simultaneously and basically listens
for the words that carry the most contextual
weight no matter where they are in the paragraph.
Okay, so if I type a paragraph about a bank,
the attention mechanism knows whether I'm talking
about a river bank or a financial bank based
on words that might be like three sentences away.
Precisely. It weighs the importance of those
context clues instantly. Which is what gives
it that incredibly powerful ability to generate
coherent, long -form text that doesn't just lose
the plot halfway through a page. To teach a machine
with 175 billion dials how to pay attention requires
an unfathomable amount of computing power. Unfathomable
is the right word, because training GPT -3 hypothetically
cost around $4 .6 million just in computing time.
Yeah, just for the electricity and processing.
If you tried to run the training process on a
single high end graphics processing unit, a standard
GPU, it would have taken 355 years to finish.
That's absurd. They had to string together massive
warehouses of supercomputers running in parallel,
consuming massive amounts of power just to process
the data. And that creates this massive logistical
moat. Suddenly, building cutting edge AI wasn't
something a couple of grad students could do
in a university lab. You needed the capital and
infrastructure of a massive tech conglomerate.
And then there's the diet. To train 175 billion
parameters, what did this giant actually have
to eat? Well, it consumed hundreds of billions
of words, but it didn't just swallow whole words.
The model breaks language down into what are
called byte -pairing coded tokens. Tokens, okay.
Yeah, token isn't necessarily a full word. Sometimes
it's a syllable or a common grouping of letters.
So instead of trying to memorize the dictionary,
it's learning the root components of human language.
It's learning like the Lego bricks of our alphabet
so it can build words it has never even seen
before. That's a great way to visualize it. And
the sheer volume of those Lego bricks was staggering.
60 % of its training diet came from a silted
version of something called the Common Crawl.
The Common Crawl. Yeah. That alone is 410 billion
tokens of scraped internet data. Then they added
19 billion tokens from a data set called WebText2,
which includes highly upvoted Reddit links. Ugh.
Reddit. That explains a lot. It does. Then 12
billion tokens from a collection of books, 55
billion tokens from another book collection,
and finally, just 3 billion tokens from Wikipedia.
I want to pause on that recipe for a second,
because the mix of that diet is exactly what
gave it its specific voice. Totally. I mean,
it needs the messy, unstructured common crawl
and Reddit links to understand how real humans
actually talk, how they argue, and how they write
computer code on the internet. Right, the colloquial
stuff. Yeah, but it needs those billions of tokens
from published books to learn long form narrative
structure, deep logic and proper grammar. Exactly.
It's a combination of bar room internet chatter
and library academics and all of Wikipedia, which
feels like the entirety of human knowledge to
most of us, made up only three percent of its
training diet. Yeah, just a tiny fraction. And
because its training data was so all encompassing,
it achieved something the field had been striving
toward for decades. It didn't need to be constantly
retrained from scratch to do specific distinct
tasks. It had ingested so much of human language
that it could adapt on the fly. Wait, I have
to push back on this a little because this is
where the concept of learning gets really weird
for me. Okay, go for it. If this model essentially
just ingested a giant unfiltered smoothie of
internet text and books, how does it know how
to write CSS code for a website or Python? or
a beautifully structured five paragraph essay
about the Roman Empire? That's a great question.
Because it was never explicitly programmed to
be a software engineer or history teacher. Right.
And that is the phenomenon researchers call...
Zero -shot and few -shot learning. Zero -shot
learning, yeah. Because the model has read millions
of examples of computer code on the internet
and millions of history essays in its book data
sets, it has basically mapped the underlying
mathematical structures of those formats. Oh,
I see. You don't have to rewrite the AI's core
software to teach it a new task. You just give
it a prompt. Maybe you give it one or two examples
of what you want. That's the few -shot part.
A few shot meaning a few examples. Exactly. It
recognizes the pattern from its vast training
data and simply completes the task. It performs
tasks it was never explicitly programmed to do
simply by predicting what should logically come
next based on everything it has ever read. So
it's just pattern matching on a cosmic scale.
Precisely. And the real -world applications of
that pattern matching were instantly wild. I
mean, back in 2020, the Guardian newspaper actually
used GPT -3 to write a full article. Yes, I remember
that. They fed it a few prompts about AI being
harmless to humans, and it generated eight different
essays. Then the human editors just spliced the
best parts together. Pretty incredible. And then
you had the development of GitHub Copilot, which
was built on a version of this model. It literally
translates conventional human language into formal
computer code. You can just type a sentence like,
create a bouncing red ball in JavaScript, and
it writes the functioning code. And it even moved
beyond just text and code. Researchers at Drexel
University ran a fascinating study in 2022 suggesting
that GPT -3 based systems could actually be used
to screen for early signs of Alzheimer's disease.
Wait, really? Just from text? Yeah. They did
this just by having the AI analyze transcripts
of spontaneous human speech. Because the model
is so exceptionally good at predicting normal
baseline language patterns, it can easily detect
the subtle early deviations and pauses that might
indicate cognitive decline. That is amazing,
which completely explains why the tech world
had such a polarized reaction. I mean, you had
people like the Australian philosopher David
Chalmers calling it one of the most interesting
and important AI systems ever produced. True.
have a review of the New York Times calling its
abilities amazing and humbling, but also spooky
and more than a little terrifying. Yeah, a review
in Wired magazine noted it was provoking literal
chills across Silicon Valley. Which is completely
understandable. Well, here's where it gets really
interesting. Because if a model is this powerful,
if it's this good at mimicking human output,
what happens when it gets things wrong? That
is the big question. Right. The data it was fed,
that massive common crawl, wasn't carefully curated
by a panel of librarians. It was scraped from
the open internet. And the internet, as we all
know, has some incredibly dark, toxic corners.
It absolutely does. And this is a crucial reality
of the technology. Because GPT -3 mimics its
training data, it inevitably generates toxic
language. Yeah. Jerome Pezzenti, who was the
head of the Facebook AI lab at the time, conducted
tests and explicitly called GPT -3 unsafe. Unsafe.
Yes. He pointed to the sexist, racist, and bi—
language the system readily generated when it
was prompted to discuss subjects like women,
black people, Jewish people, and the Holocaust.
And because the AI sounds so authoritative, people
trust it even when it's being toxic or dangerous.
Which honestly brings us right back to that terrifying
anecdote about the Nebala startup from the beginning.
Yes, the medical chatbot. They tested it and
the AI advised a simulated patient to commit
suicide. How does a machine that can write brilliant
Python code make a mistake that horrific? This
raises an important question about the fundamental
nature of the technology. It's a critique championed
by cognitive scientist Gary Marcus and the renowned
linguist Noam Chomsky. Oh, Chomsky weighed in.
He did. Chomsky argued that models like GPT -3
tell us absolutely nothing about language, thought,
or cognition. Marcus and his co -author wrote
that GPT -3's comprehension of the world is often
seriously off. Right. The core of their argument
is that the model is essentially a stochastic
parrot. A stochastic parrot, meaning it's just
like randomly parroting things back without having
any idea what it's actually saying. Exactly.
GPT -3 models the mathematical relationships
between words flawlessly. It knows that the word
apple is frequently followed by the words tree
or pie or orchard. Sure. But it has no actual
understanding of what an apple is. It has never
tasted sweetness, it has no physical experience
of the world, no moral compass, and no actual
comprehension of life or death. So when that
medical chatbot suggested a patient harm themselves,
it wasn't acting out of malice or cruelty. It
was simply predicting the next most statistically
likely string of words based on dark, toxic data
it had read somewhere in its massive internet
diet. It's just turning those 175 billion dials
to get the next word. It doesn't know if the
word is helpful or completely fabricated or physically
dangerous. And that blind spot, the fact that
this immense power has no inherent moral compass,
is exactly what sparked a massive war over who
actually gets to control those dials and who
profits from them. And the shift in OpenAI's
corporate structure during this period is quite
stark and highly debated. It's a massive structural
irony. I mean, OpenAI was initially founded back
in 2015 as a non -profit organization. The whole
stated goal was to develop our artificial intelligence
safely and democratize the technology so it wouldn't
be controlled by just one or two mega corporations.
But then in 2019, they shifted to a capped profit
model. Very significant pivot. Yeah, they argued
they needed to do this to attract the billions
of dollars in capital required to build the supercomputers
we talked about earlier. And then in 2020, following
a massive investment, they gave Microsoft an
exclusive license to GPT -3's underlying code.
And it was a very specific type of deal. Anyone
could pay to use the public API, the interface,
to get text out of the model. But only Microsoft
had access to the actual source code. Giving
them a huge advantage. Exactly. It allowed them
to embed it deeply into their own products and
modify the core architecture. That shift... from
an open source nonprofit to a tightly guarded
for -profit exclusive licensing deal sparked
intense debate about the concentration of power
in the tech industry. And that centralization
of power is just the tip of the iceberg when
it comes to the controversies this model kicked
off. Because you also have to look at the external
costs of running those massive server farms.
Right. The environmental impact is severe. Researchers
like Timnit Gebru and Emeliam Bender co -authored
a prominent paper pointing out the massive carbon
footprint and environmental degradation caused
by the computing power required to train and
store these incredibly large language models.
It takes so much electricity. It does. They argued
the race for bigger models was completely ignoring
the ecological cost. Then you have the looming
legal battles over copyright. Remember that 60
% of the diet came from the common crawl. the
scraped internet data. Right. That was a conglomerate
of copyrighted articles, internet posts, and
books scraped from 60 million domains. Tech publications
reported it included copyrighted material from
the BBC, the New York Times, Reddit, and the
full texts of thousands of Polish books. Which
is a massive legal minefield. Yeah. OpenAI argued
to the U .S. Patent and Trademark Office that
training AI systems on this data constitutes
fair use under current law because the AI transforms
the data rather than just copying it. But even
they admitted there is substantial legal uncertainty.
I mean, you're essentially taking the collective
output of human culture, feeding it into a machine.
and then charging a licensing fee to access the
pattern -matched results of that culture. It's
wild when you phrase it like that. It is a legal
and philosophical gray area that courts will
be untangling for a decade. And of course, we
can't forget the immediate impact it had on education.
Oh, the plagiarism panic. Exactly. The release
of GPT -3 ignited a growing panic over academic
integrity. Universities and high schools suddenly
had to figure out how to gauge academic misconduct
when a machine could generate a totally unique,
perfectly written, undetectable essay on any
topic in five seconds. So these glaring flaws
and criticisms, the toxicity, the environmental
costs, the copyright lawsuits, the academic plagiarism,
they essentially forced OpenAI to adapt. They
did. They couldn't just leave GPT -3 as a raw,
unfiltered mirror of the internet. So by early
2022, they announced new models collectively
referred to as InstructGPT and later the GPT
3 .5 series. Right of the next iteration. They
basically took the raw computational power of
GPT -3 and fine -tuned it using datasets of human
-written instructions and feedback. They hired
actual humans to rank the AI's answers. The goal
was to make the model actually follow user instructions
better, hallucinate fewer fake facts, and produce
significantly less toxic content. They were trying
to align the AI better with actual human intentions.
Which was a necessary evolution for commercial
viability. Absolutely. The transition to GPT
3 .5 and the eventual public release of ChatGPT
later that year showed that the raw brute force
scale of GPT -3 wasn't enough on its own. It
needed human guidance, guardrails, and behavioral
fine -tuning to be a truly useful, safe product.
than just a fascinating, unpredictable research
experiment. So what does this all mean? If we
step back and look at the whole picture for a
second, GPT -3 wasn't just a software update.
It was a mirror held up to human knowledge. That's
a great way to look at it. By ingesting hundreds
of billions of words from the internet, it proved
that massive mathematical scale could create
the breathtaking illusion of human intelligence.
But because it was a mirror, it also reflected
everything else. It reflected our biases, our
toxicity, and our messy legal realities regarding
copyright and ownership. And if we connect this
to the bigger picture, the most vital takeaway
for you as a user of this technology is the absolute
necessity of critical thinking. Yes. These models,
or their direct descendants, are now deeply integrated
into the tools you use every single day, from
code editors to search engines to customer service
bots. But you must remember the stochastic parrot.
The machine doesn't actually know anything. Exactly.
The AI does not understand the world. It predicts
the world based on the data it was fed. So never
surrender your own judgment, your own fact checking,
or your own moral compass to an algorithm that
cannot actually comprehend the meaning of the
words it generates. A completely vital reminder.
It predicts it doesn't comprehend. Which leaves
me with one final lingering thought for you to
chew on after you finish listening today. Okay,
let's hear it. We established that GPT -3 and
its successors learned how to mimic human language
by scraping hundreds of billions of words written
by humans on the internet. But what happens a
few years from now? As tools built on these models
flood the internet with billions of pages of
AI -generated text, articles, and code will future
AI models just end up training on the synthetic
output of other AIs. A terrifying thought. Right.
And if the machine is eventually only learning
from the machine, what happens to the human voice?