So every time you type a prompt into, you know,
chat GPT or Gemini or Claude, you are essentially
interacting with the ghost of the internet. Right.
Yeah. It's a vast echo. Yeah. Behind the magic
of all this modern artificial intelligence, there
isn't just like a clever algorithm. There is
a massive, totally invisible and highly controversial
database. Run by a quiet nonprofit. Exactly.
One that you, the listener, have likely never
even heard of. So today, for you, we're taking
a deep dive into the Common Crawl Foundation.
Which is such an important topic right now. It
really is. Our mission today is to unpack how
this completely obscure organization became well.
the foundational engine powering the entire AI
revolution. And how it suddenly found itself
right at the center of this massive war over
the future of the internet itself. It's wild.
OK, let's unpack this. Because before we get
into the modern AI drama and all the lawsuits,
we really need to understand what Common Crawl
actually is. Yeah, we have to go back a bit.
Because if you use AI tools today, you're literally
querying the echoes of Common Crawl's work. Right.
So. Back in 2007, an entrepreneur named Gil El
-Baz founded this organization as a 501c3 nonprofit.
And the stated mission back then was entirely
focused on the public good. Just to crawl the
web and freely provide its archives and data
sets to the public, essentially like they wanted
to make the Internet downloadable. Which, you
know, doing that requires immense infrastructure
and a very specific technical approach. So to
visualize this web crawling, it's kind of like
imagine a massive fleet of digital Roombas. Right.
That's a great way to put it. Yeah. These automated
software bots that start their journey on a few
highly connected web pages. And they read the
raw HTML code of a page. They vacuum up all the
text, the images, the metadata. And then they
look for any hyperlinks. Exactly. They click
those links, travel to the next set of pages,
and just repeat the whole process. They just
relentlessly moving across the open web, copying
everything they touch, and dumping it into this
massive searchable repository. Right. And they
mirrored all the data and made it available online
through the Internet Archive's Wayback Machine.
Which is incredible. And they brought in some
serious tech heavyweights as advisors early on.
Yeah. People like Peter Norvig and Joy Ito. Big
names. But I guess my question is, why do this
as a nonprofit? Like, was the original goal just
to be a giant backup hard drive for humanity?
Well, no, it was fundamentally about democratizing
research and development. OK. I mean, think about
the landscape of the tech world in like the late
2000s and early 2010s. If you were an academic
wanting to analyze web scale data. Like tracking
how slang spreads across internet forums or something.
Exactly. Or studying search engine optimization
trends or mapping misinformation. You simply
couldn't do it. Because you need Google servers.
Right. You needed the resources of a massive
corporation like Google or Yahoo. You needed
server farms just to gather the raw data before
you could even begin your analysis. So the barrier
to entry was what millions of dollars. and computing
power just to get a snapshot of what people were
talking about online? Yeah, and Common Crawl
totally removed that barrier by doing the really
expensive labor -intensive vacuuming and then
giving the resulting data set away for free.
Which meant academics, independent researchers,
small startups. They suddenly had access to the
exact same scale of internet data. that the tech
giants had. And it worked perfectly for its intended
purpose. I mean, long before AI chatbots were
a thing, people built real, tangible tools with
this. Like Tiny, right? Yes. By 2013, the reverse
image search site Tiny built its products off
common crawls data. That's amazing. And academically,
by 2024, the data set had been cited in over
10 ,000 academic studies. It's a staggering foundational
impact. It proved that if you provide the raw
material of the internet for free, human ingenuity
will find and incredible ways to use it. It was
almost this utopian vision of the internet. Open
data leading to open innovation. But, you know,
the nature of innovation is unpredictable. And
the way human ingenuity ended up using this data
shifted the entire global economy. Right, which
brings us to the AI boom. The big pivot. Because
you have this massive free digital library of
human knowledge. That's great for academics writing
papers. Yeah. But over the last few years, the
tech landscape just like shifted beneath our
feet. Completely. Artificial intelligence companies
realized they desperately needed an unprecedented
volume of text to train their new models. And
they looked around and sitting right there was
this colossal prepackaged snapshot of the internet.
And this is where the paradigm completely chefs,
because the invention of large language models
or LLMs changed everything about how software
is developed. Because they don't learn like traditional
software. No, they don't learn by being programmed
with rigid grammar rules by a human engineer.
They learn through a process called next word
prediction. OK, so you feed the model a sequence
of words. and it just guesses the next word based
on mathematical probabilities. Exactly. And to
get good at that, like to converse fluidly like
a human, to write code, to compose poetry, it
requires a volume of text that we can barely
comprehend. Right, you can't just feed an AI...
a thousand books or even a hundred thousand Wikipedia
articles and expect it to pass the bar exam.
Not at all. You need billions, if not trillions,
of words to understand language context, subtle
jokes, coding syntax, historical facts. The AI
has to map the statistical relationships between
tokens across a truly massive corpus of text.
Tokens being like... Pieces of words. Yeah, pieces
of words. And Common Crawl was simply the right
data set at the exact right time. It transitioned
almost overnight from an academic resource to
the foundational infrastructure of a trillion
dollar industry. I mean, a filtered version of
Common Crawl was heavily utilized to train OpenAI's
GPT -3, which was announced back in 2020. Yep.
and it was used to train Google DeepMind's Gemini.
Google even created its own specific, highly
-filtered version of Common Crawl back in 2019.
They called it the Colossal Clean Crawl Corpus,
or C4 for short. And they used that to train
their T5 language model series. Without Common
Crawl's historical archiving, the rapid advancement
of these language models would have been severely
bottlenecked. Because the data was the missing
piece of the puzzle. Exactly. The tech companies
had the massive computing power, and they had
the brilliant algorithms, but they desperately
needed the raw text to feed the machine. And
CommonCrawl provided it on a silver platter.
Yeah, completely free. But I have to push back
here, though, because this dynamic feels incredibly
tilted. It's so. Well, you have a 501C nonprofit.
funded originally by the Elbez Family Foundation
Trust, giving away its data for free out of a
sense of public good. Right. And that free data
becomes the absolute secret sauce for massively
profitable multi -billion dollar AI companies.
It does. And then, in 2023, Common Crawl begins
receiving significant financial support directly
from the AI industry. We're talking $250 ,000
donations each from OpenAI and Anthropic. Yeah,
those are big numbers. Doesn't that compromise
their independent mission? I mean, it feels like
the digital Roomba suddenly works for a very
specific corporate master. It absolutely highlights
a massive wealth transfer. And that is the exact
tension the tech world is grappling with right
now. OK. Because from one perspective, the nonprofit
is just receiving necessary funding to continue
its incredibly expensive mission. I mean, archiving
the web at that scale. The server costs and bandwidth
are astronomical. Right. So who better to fund
it than the companies relying on it? Exactly.
And the data is still technically free for anyone
else to use. But you are absolutely right to
point out the friction there. Yeah. The nonprofit
does the heavy lifting of gathering the world's
knowledge and for -profit companies to synthesize
it into closed products that generate massive
exclusive revenue. So you have AI companies turning
credible profits off free data. But there's a
huge missing link here, which is the people who
actually wrote that data in the first place.
The creators. Right. When OpenAI or Google makes
millions of dollars off a language model, the
journalist, the blogger, or the novelist who's
writing trained that model doesn't see a dime.
Not a single dime. And that brings us to the
massive copyright collision. Because when you
scrape the entire open web, you inevitably scrape
things that people want protected. Oh, totally.
You capture copyrighted books, paywalled investigative
articles, private blogs, recipe sites, proprietary
code repository, everything. And as far back
as 2016, it was well documented that the Common
Crawl data set included copyrighted work. But
they were distributing it from the United States
under fair use claims. Yes. So let's break down
that fair use argument mechanically. How does
vacuuming up someone's copyrighted book legally
fly? Well, the concept of fair use in the US
is pretty legally flexible. It generally allows
for the use of copyrighted material without permission
if the use is highly, quote unquote, transformative.
OK, transformative. Like what? For example, a
search engine indexing a web page and displaying
a snippet of text. So you can find the site that's
considered fair use. Right. So AI companies argue
that training a machine learning model is also
transformative. They argue they aren't republishing
a copyrighted book to compete with the author.
They're just analyzing the book. to learn the
underlying mechanics of human language. Exactly.
Okay, but the internet isn't governed solely
by US law. What happens when researchers or AI
developers in Europe, where copyright laws are
often much stricter, want to use this data? That
is where we see some incredible legal and technical
gymnastics. Oh boy. Because researchers outside
the US face different liabilities, so they've
had to invent these wild workarounds to interact
with the data set without technically hosting
copyrighted works. Wait, like what kind of workarounds?
One prominent technique is literally shuffling
the sentences of a document before analyzing
it. Shut up. Really? So they take a novel or
a deeply researched article and just put it into
a digital blender? Precisely. You are literally
destroying the artistic expression, the specific
way the author ordered their thoughts, built
their narrative, conveyed emotion. Which is what
copyright actually protects. Exactly. But by
keeping the words and sentences intact, just
out of order, you preserve the statistical relationships
between the individual words. Wow. And that statistical
math is all the AI or the researcher actually
cares about anyway. Right. Alternatively, developers
would just reference the common crawl datasets
remote location rather than hosting the data
themselves, kind of shifting the legal liability
away from their own servers. So they basically
found a legal loophole. They destroy the art
to extract the data. Essentially, yes. Let's
look at this through a different lens. Imagine
someone going to a massive public library and
taking millions of free books. Okay. They feed
all those books into a giant industrial processor
and out comes this omniscient Robot. And that
robot then stands right outside the library doors
and charges people a fee to answer any question
they have. Perfectively ensuring those people
never need to go inside, check out a book, or
support the author. Exactly. The original authors
are going to look at that robot and say, hey,
wait a minute. You used my life's work to build
my replacement. And that captures the existential
threat. to creators perfectly. I mean, the internet
was built on the premise of open sharing and
indexing. When a writer put an article online
in, say, 2010, they wanted a crawler like Google
to index it so humans could find it, read it,
and maybe click and add or subscribe. But AI
training wasn't the intended use. No. They never
consented to a machine ingesting their work to
learn how to mimic their writing style. And then
just... answer user queries directly, bypassing
the author's website entirely. And creators are
actively waking up to this and fighting back.
Big time. There was a 2024 New York Times study
by Kevin Ruse that revealed a staggering statistic.
45 % of content is now explicitly restricted
by websites. That's nearly half the internet.
Yeah. They are actively putting up digital do
not enter signs because they refuse to be scraped
without compensation. We also saw major concerns
raised over copyrighted content specifically
inside Google's C4 data set, which was thoroughly
reported by The Guardian in 2023. The tension
is just palpable. It is. The core question society
is wrestling with right now is what fair use
really means when a machine rather than a human
is doing the reading at a scale of billions of
pages a day. Which means all of this underlying
friction eventually hits an absolute boiling
point. And it did. November 2025. Technology
journalist Alex Reisner publishes this massive
explosive investigation in the Atlantic. Yes.
And the allegations leveled against common crawl
are severe. The investigation claimed that Common
For All was explicitly bypassing publisher requests
to have their content removed from its databases.
And to understand the gravity of that, we have
to explain how those removal requests usually
work. Right. Let's get into the technical side.
When a website wants to block a web crawler,
they typically use a file called robots .txt.
It's essentially a simple text file sitting on
the website server that acts as a digital traffic
cop. It tells incoming bots, you are allowed
here, but you are not allowed there. Okay, so
a publisher puts up the stop sign. Right. And
the allegation in the Atlantic was that Common
Crawl's bots were simply ignoring these standard
digital traffic lights or actively bypassing
paywalls to get to the text underneath. And there's
a highly specific technical piece of this allegation
that really stands out. The piece claimed that
the public search function on Common Crawl's
own website was fundamentally misleading. And
this is the part that caused massive, massive
outrage. Explain how that worked. So if you are
a publisher and you add that robots .txt file
to block Common Call, you might go to Common
Call's website a few weeks later and search for
your domain just to verify they complied. Makes
sense. You want to check their work. Exactly.
And according to the investigation, the public
-facing search tool would show zero entries for
your site. So you'd look at that and think you
were safe. Right. You'd think, OK, they respected
my request. But the allegation was that while
the public search index hid your site, your data
was actually still included in the underlying
raw scraped data dumps. The WarRC files. Yes,
the WarRC files. And those are the files being
handed over in bulk to the AI companies. Wow.
If true, that is a massive breach of trust. It
suggests this two -tiered system, like a sanitized,
compliant public face to appease angry publishers,
and then a backend data pipeline that quietly
ignores restrictions to just keep feeding the
AI industry's insatiable appetite for fresh data.
And publishers did not take this lightly at all.
A report by Wired noted that publishers are now
actively targeting common crawl in these fights
over AI training data. They're realizing that
if they want to stop AI companies from using
their work, suing the AI companies might not
be enough. Right, they have to go directly after
the supplier of the raw material. But Comic Roll
didn't just absorb the blow. No, they fired back.
Rich Screnta published a formal public reply
titled, Setting the Record Straight. vigorously
defending the organization. And I really want
to break down both sides of this battlefield
impartially, because this isn't just a technical
dispute. No, it's a fundamental clash over who
owns the information on the internet. Exactly.
So let's start with the publishers. Their argument
is rooted purely in commercial survival. Absolutely,
because investigative journalism, high quality
literature, detailed research, that stuff costs
real money to produce. Yeah, you have to pay
writers. Paywalls and ad revenue are how these
entities survive in the digital age. So from
the perspective of the Atlantic and the wider
publishing industry, if an organization bypasses
those paywalls, copies the proprietary content,
and feeds it into an AI. An AI that can then
just summarize that exact content for an end
user. Then the original publisher loses their
traffic. They lose their subscriber revenue.
Right. To them, bypassing a paywall to scrape
data is theft, plain and simple, and their intellectual
property absolutely must be protected for their
industries to survive. It's a very clear line.
If you fund a six -month journalistic investigation,
you deserve to get paid for it, not have a machine
ingest it for free and spit out the bullet points
to millions of people. Exactly. But then we have
common -crawls defense, which is equally foundational
to how the architecture of the internet actually
works. Yeah, Common Crawl's defense, as outlined
in Screnta's Setting the Record Straight, rests
on their long -standing commitment to transparency
and their belief in the public good. They really
push back on the idea of malicious intent, right?
They do. They defend their technical processes,
noting that web crawling at the scale of billions
of pages is an imperfect science. It's not some
malicious conspiracy to steal data. And their
perspective is that locking down the internet
behind paywalls and opting out of historical
archives fundamentally damages the open web.
Because if everything is locked down, only the
richest companies can afford to buy access to
the data. through licensing deals. Exactly. And
then we are right back to the 2007 problem where
only a massive tech giant can afford to do research.
Wow. Full circle. Precisely. To common crawl,
indexing the web even for AI training is a transformative
use that ultimately benefits society by driving
massive technological progress. They sort of
view themselves as digital librarians, don't
they? They do. And a librarian doesn't pay a
royalty every time someone reads a book in the
library to learn something new or be inspired
to write their own book. Right. They believe
that archiving the public web is a vital service
to humanity and restricting that archive actively
hinders innovation. It's the classic unstoppable
force meeting an immovable object. The commercial
necessity for human creators to survive versus
the technological drive to organize and process.
all human knowledge. And caught right in the
middle of it is a 501c3 nonprofit that started
with the simple goal of giving researchers a
downloadable backup of the internet. It perfectly
illustrates how quickly technology outpaces our
legal and ethical frameworks. I mean, the infrastructure
of the open web was built for human readers in
one era, and it has been entirely co -opted by
machine readers in another. It really has. It's
a profound journey when you step back and look
at it. We started this deep dive looking at Gill
Elbaas in 2007, building digital Roombas to help
academics spot trends. And we've ended up in
a landscape where those same Roombas are funded
by massive tech conglomerates, accused of sneaking
past paywalls, and serving as the primary battleground
for the future of copyright law. The most crucial
takeaway for you, the listener, is to remember
that every single time you prompt an AI, you
are not just talking to a clever algorithm. No,
you are querying the entire archived history
of the open web, vacuumed up and processed by
common crawl. But that open web is closing. As
we discussed, nearly half of the internet's content
is now explicitly restricted. Yeah, with these
massive controversies, publishers, journalists,
and creators are building digital fences faster
than ever. They are locking their doors. Which
leaves you with a really profound question to
mull over. If the open web continues to build
these fences, what happens to the next generation
of artificial intelligence? Will future models
only be able to learn from a closed -off, highly
commercialized, sanitized version of the Internet?
The AI magic trick we enjoyed today relies entirely
on a massive, hidden engine of free human knowledge.
But if those raw materials dry up or get locked
behind iron -clad legal vaults, what does a post
-common -crawl Internet look like for a learner
seeking the truth?