WEBVTT 00:00:00.000 --> 00:00:02.379 Every single time you rate a movie on your TV 00:00:02.379 --> 00:00:06.639 or fire off a quick, frustrated email at work 00:00:06.639 --> 00:00:09.199 or even just walk down a busy city street with 00:00:09.199 --> 00:00:11.519 your smartphone sitting in your pocket, you are 00:00:11.519 --> 00:00:15.519 leaving behind this massive, invisible trail 00:00:15.519 --> 00:00:17.989 of digital exhaust. Yeah, you really are. And 00:00:17.989 --> 00:00:20.870 it's constant. Right. Most of the time, you probably 00:00:20.870 --> 00:00:23.250 don't think twice about it. It just feels like, 00:00:23.250 --> 00:00:25.370 you know, the byproduct of living in the modern 00:00:25.370 --> 00:00:27.989 world. Just background noise, essentially. Exactly. 00:00:28.449 --> 00:00:30.969 But what if I told you that this digital exhaust 00:00:30.969 --> 00:00:33.770 doesn't just evaporate into the ether. It gets 00:00:33.770 --> 00:00:36.950 captured, meticulously packaged, and fed directly 00:00:36.950 --> 00:00:39.030 into the minds of the machines that are slowly 00:00:39.030 --> 00:00:41.490 learning to run our reality. Which is a pretty 00:00:41.490 --> 00:00:43.570 wild thing to wrap your head around. It really 00:00:43.570 --> 00:00:46.799 is. Welcome to the deep dive. Today our mission 00:00:46.799 --> 00:00:49.420 is to crack open the digital diet of artificial 00:00:49.420 --> 00:00:52.140 intelligence. We are looking at a master list, 00:00:52.340 --> 00:00:55.079 a massive open source compendium compiling the 00:00:55.079 --> 00:00:57.500 most critical data sets used for machine learning 00:00:57.500 --> 00:00:59.979 research. And it is a truly massive list. Oh 00:00:59.979 --> 00:01:02.979 yeah. Now our goal here isn't just to read off 00:01:02.979 --> 00:01:05.920 a table of contents. We are looking for the invisible 00:01:05.920 --> 00:01:08.980 scaffolding of AI. We want to understand the 00:01:08.980 --> 00:01:11.780 exact foundational raw materials that machines 00:01:11.780 --> 00:01:14.400 are fed to learn how to think, speak, and understand 00:01:14.400 --> 00:01:16.569 the universe and I live in. Which is the stuff 00:01:16.569 --> 00:01:19.030 people rarely talk about. Right. OK, let's unpack 00:01:19.030 --> 00:01:20.769 this. Because whenever people talk about AI, 00:01:21.129 --> 00:01:23.129 all the glory usually goes to the flashy stuff, 00:01:23.489 --> 00:01:26.409 right? The massive neural networks, the billion 00:01:26.409 --> 00:01:28.650 dollar data centers, the algorithms that seem 00:01:28.650 --> 00:01:33.109 like magic. Major advances in this field actually 00:01:33.109 --> 00:01:36.030 rely just as heavily on the unsung heroes, which 00:01:36.030 --> 00:01:38.609 are high quality training data sets. It really 00:01:38.609 --> 00:01:41.409 is the great untold story of the entire industry. 00:01:41.890 --> 00:01:44.769 I mean, an algorithm, no matter how incredibly 00:01:44.769 --> 00:01:47.430 sophisticated or well designed it is, is completely 00:01:47.430 --> 00:01:49.930 useless without data. It's an empty shell. Exactly. 00:01:50.049 --> 00:01:52.109 It is an engine without fuel. And the reality 00:01:52.109 --> 00:01:54.870 of acquiring, refining and storing that fuel 00:01:54.870 --> 00:01:57.030 is what actually dictates the pace of innovation. 00:01:57.159 --> 00:01:59.819 To train a machine effectively, especially in 00:01:59.819 --> 00:02:03.180 what we call supervised learning, you need high 00:02:03.180 --> 00:02:06.799 quality labeled data. Meaning humans are doing 00:02:06.799 --> 00:02:09.319 the heavy lifting behind the scenes. Oh, absolutely. 00:02:10.099 --> 00:02:12.319 That means humans have had to actually sit down 00:02:12.319 --> 00:02:15.379 and tag the information. They have to explicitly 00:02:15.379 --> 00:02:18.219 tell the computer, you know, this collection 00:02:18.219 --> 00:02:21.939 of pixels is a picture of a cat. Or, this specific 00:02:21.939 --> 00:02:24.719 string of words is a sarcastic sentence. Or like, 00:02:24.740 --> 00:02:27.400 this cluster of pixels is a healthy cell. Right, 00:02:27.699 --> 00:02:31.180 exactly. And that process is painstakingly slow. 00:02:31.479 --> 00:02:34.800 Not to mention, astronomically expensive. And 00:02:34.800 --> 00:02:37.099 even if you shift to unsupervised learning, where 00:02:37.099 --> 00:02:39.479 the machine just looks at raw, unlabeled data 00:02:39.479 --> 00:02:42.039 to find its own patterns, just the sheer act 00:02:42.039 --> 00:02:44.060 of gathering, cleaning, and storing those massive 00:02:44.060 --> 00:02:46.939 data sets carries an enormous logistical cost. 00:02:47.060 --> 00:02:49.759 So it's never really free or easy? No, never. 00:02:49.879 --> 00:02:52.300 The files on this list we are looking at today 00:02:52.300 --> 00:02:54.620 are the bedrock that makes modern artificial 00:02:54.620 --> 00:02:57.180 intelligence possible. Okay, so before we look 00:02:57.180 --> 00:02:59.639 at what exactly the machines are eating, we really 00:02:59.639 --> 00:03:01.580 need to understand the logistics of how it's 00:03:01.580 --> 00:03:03.659 being served to them. Yeah, the delivery mechanism 00:03:03.659 --> 00:03:06.150 is key. Because you don't just dump a billion 00:03:06.150 --> 00:03:09.030 random text messages into a server, hit enter 00:03:09.030 --> 00:03:11.569 and expect an artificial intelligence to suddenly 00:03:11.569 --> 00:03:13.949 wake up and become a poet. Well, exactly. You 00:03:13.949 --> 00:03:16.389 could dump a billion texts into a server, but 00:03:16.389 --> 00:03:19.030 it would be completely useless without structure. 00:03:19.330 --> 00:03:20.930 The machine wouldn't even know what it's looking 00:03:20.930 --> 00:03:23.669 at. It'd just be digital noise. Right. That is 00:03:23.669 --> 00:03:26.389 why the infrastructure of open data is so critical. 00:03:26.889 --> 00:03:29.569 Many organizations, from global universities 00:03:29.569 --> 00:03:32.550 to local governments, share their data on massive 00:03:32.550 --> 00:03:35.509 open portals. Like... The ones in the compendium. 00:03:35.710 --> 00:03:38.129 Yeah. We are talking about platforms like OpenML, 00:03:38.449 --> 00:03:41.930 Kaggle, Dataverse, and even decentralized networks 00:03:41.930 --> 00:03:45.289 like Academic Torrents. And to ensure that a 00:03:45.289 --> 00:03:47.729 researcher in Tokyo and a researcher in Toronto 00:03:47.729 --> 00:03:50.689 can both understand the exact same data set, 00:03:51.090 --> 00:03:54.229 they rely on common metadata formats, like the 00:03:54.229 --> 00:03:56.460 Croissant standard, for instance. croissant, 00:03:56.599 --> 00:03:59.020 like the buttery French pastry? Just like the 00:03:59.020 --> 00:04:01.099 pastry, yeah. It's an open standard that helps 00:04:01.099 --> 00:04:03.500 describe what the data actually is. It tells 00:04:03.500 --> 00:04:05.740 the machine and the researcher the layout of 00:04:05.740 --> 00:04:08.360 the information. Oh, OK. That's a fun name for 00:04:08.360 --> 00:04:11.699 something so technical. It is. And these portals 00:04:11.699 --> 00:04:15.099 meticulously sort the data using these standards. 00:04:15.219 --> 00:04:17.879 They organize it by scope. So is this a data 00:04:17.879 --> 00:04:20.699 set covering a single local municipality or is 00:04:20.699 --> 00:04:23.860 it supranational, covering global metrics? They 00:04:23.860 --> 00:04:26.259 organize it by language. They sort it by license 00:04:26.259 --> 00:04:28.720 type, which is huge. Why is the license type 00:04:28.720 --> 00:04:31.800 so huge? Well, you have Creative Commons for 00:04:31.800 --> 00:04:33.860 general open use, and then you have things like 00:04:33.860 --> 00:04:36.379 the AGPL license, which is a strict open source 00:04:36.379 --> 00:04:39.100 agreement, basically saying, you know, if you 00:04:39.100 --> 00:04:41.300 use this data to build something, you have to 00:04:41.300 --> 00:04:43.600 share your homework and make your new code open 00:04:43.600 --> 00:04:46.209 to everyone else. I see. So it's heavily regulated 00:04:46.209 --> 00:04:48.829 and categorized. Very much so. It's like training 00:04:48.829 --> 00:04:51.250 in AI isn't really like filling a gas tank with 00:04:51.250 --> 00:04:53.430 ultra -refined fuel, right? It's more like raising 00:04:53.430 --> 00:04:57.009 a child in a completely sealed library. If you 00:04:57.009 --> 00:04:59.089 are going to lock them in there to learn about 00:04:59.089 --> 00:05:01.610 the world, the books better be categorized perfectly. 00:05:01.790 --> 00:05:03.370 That's a great way to put it. But looking at 00:05:03.370 --> 00:05:05.829 this list, the sheer variety is staggering. I 00:05:05.829 --> 00:05:07.970 mean, there is everything from last hour updates 00:05:07.970 --> 00:05:12.189 on global weather patterns to decades old, incredibly 00:05:12.189 --> 00:05:14.329 dry text files. Yeah, it's a massive spectrum. 00:05:14.589 --> 00:05:17.370 So if you just throw every single type of data 00:05:17.370 --> 00:05:19.329 into one of these portals, doesn't that just 00:05:19.329 --> 00:05:21.970 make an AI incredibly confused? Like, if I'm 00:05:21.970 --> 00:05:23.910 trying to teach a machine to recognize human 00:05:23.910 --> 00:05:26.040 faces and it accidentally reads a spread sheet 00:05:26.040 --> 00:05:28.639 about municipal water pressure, does it break? 00:05:28.860 --> 00:05:31.079 What's fascinating here is that the structural 00:05:31.079 --> 00:05:34.199 taxonomy of the portal is exactly what prevents 00:05:34.199 --> 00:05:37.160 that kind of systemic confusion. Oh really? How 00:05:37.160 --> 00:05:40.139 so? The data is strictly categorized into specific 00:05:40.139 --> 00:05:43.040 types. You have tabular data, which is your standard 00:05:43.040 --> 00:05:46.319 rows and columns, then text data, image data, 00:05:46.500 --> 00:05:48.720 sound, and signal data. So they live in completely 00:05:48.720 --> 00:05:51.439 different buckets. Exactly. This structure is 00:05:51.439 --> 00:05:53.579 what allows researchers to pull precisely the 00:05:53.579 --> 00:05:56.360 right context. If you are training a computer 00:05:56.360 --> 00:05:59.300 vision model, feeding it audio data isn't just 00:05:59.300 --> 00:06:01.759 confusing. It's incompatible with the mathematical 00:06:01.759 --> 00:06:03.879 architecture the researcher is building. It physically 00:06:03.879 --> 00:06:06.879 can't process it. Right. The portal acts as a 00:06:06.879 --> 00:06:09.240 highly organized librarian, ensuring that if 00:06:09.240 --> 00:06:11.439 an algorithm is designed to find visual patterns, 00:06:12.000 --> 00:06:14.620 it is only handed visual data. All right. So 00:06:14.620 --> 00:06:16.680 if these machines are going to eventually interact 00:06:16.680 --> 00:06:20.259 with you and me, their first major hurdle is 00:06:20.259 --> 00:06:23.610 figuring out how humans communicate. And we are 00:06:23.610 --> 00:06:26.050 not easy to figure out. No, we are incredibly 00:06:26.050 --> 00:06:29.649 messy, contradictory creatures. So researchers 00:06:29.649 --> 00:06:32.290 had to find data sets that captured the absolute 00:06:32.290 --> 00:06:35.269 chaos of human interaction. Let's look at how 00:06:35.269 --> 00:06:38.389 they teach machines to read and speak. Here's 00:06:38.389 --> 00:06:41.029 where it gets really interesting. There is a 00:06:41.029 --> 00:06:43.569 data set on this list called the Enron Corpus. 00:06:43.829 --> 00:06:46.129 Oh, yeah. That one is famous in the industry. 00:06:46.350 --> 00:06:49.290 It contains around 500 ,000 emails from employees 00:06:49.290 --> 00:06:51.990 at Enron. Yes, the infamous energy company that 00:06:51.990 --> 00:06:54.110 famously collapsed under the weight of massive 00:06:54.110 --> 00:06:57.209 corporate fraud back in 2001. Right, so researchers 00:06:57.209 --> 00:06:59.170 took half a million of their internal emails, 00:06:59.550 --> 00:07:01.350 scrubbed the attachments, removed some of the 00:07:01.350 --> 00:07:03.810 personal identifiers, and now this data set is 00:07:03.810 --> 00:07:06.870 heavily used to train AI in network analysis 00:07:06.870 --> 00:07:09.209 and sentiment analysis. It's a goldmine of data. 00:07:09.329 --> 00:07:11.910 But wait, if the data set is their entire universe, 00:07:12.230 --> 00:07:14.389 aren't we just building a machine that mim - 00:07:13.899 --> 00:07:16.579 paranoid corporate speak. That's a common concern 00:07:16.579 --> 00:07:19.360 actually. Like if I'm using an AI to help draft 00:07:19.360 --> 00:07:22.180 an email or to analyze my company's communication, 00:07:22.699 --> 00:07:25.180 I really don't want it sounding like an executive 00:07:25.180 --> 00:07:27.720 who is two days away from a federal indictment. 00:07:28.199 --> 00:07:30.660 Doesn't this bake a lot of weird human flaws 00:07:30.660 --> 00:07:33.459 directly into the software? I mean yeah that 00:07:33.459 --> 00:07:35.949 totally makes sense on paper. Yeah. But it kind 00:07:35.949 --> 00:07:38.370 of misunderstands how the data is being applied. 00:07:38.610 --> 00:07:40.389 OK. Correct me. What are they doing with it? 00:07:40.529 --> 00:07:42.550 The researchers aren't using the Enron corpus 00:07:42.550 --> 00:07:45.790 to teach an AI what to say. They are building 00:07:45.790 --> 00:07:48.810 a chat bot out of it to generate new Enron emails. 00:07:48.990 --> 00:07:51.329 Oh, OK. They're using it to teach the AI how 00:07:51.329 --> 00:07:53.810 to measure sentiment. You see, a machine doesn't 00:07:53.810 --> 00:07:56.129 inherently know what the concept of stress or 00:07:56.129 --> 00:07:58.850 panic is. It's just math. Right. It doesn't have 00:07:58.850 --> 00:08:01.620 emotions. Exactly. To an algorithm, the Enron 00:08:01.620 --> 00:08:05.160 Corpus is a pristine, incredibly rare map of 00:08:05.160 --> 00:08:08.240 power dynamics within a complex hierarchy under 00:08:08.240 --> 00:08:10.360 extreme pressure. Because things were falling 00:08:10.360 --> 00:08:13.819 apart over there. Precisely. The AI needs to 00:08:13.819 --> 00:08:16.100 see what a completely mundane Tuesday morning 00:08:16.100 --> 00:08:18.860 routine memo looks like and then mathematically 00:08:18.860 --> 00:08:22.079 compare it to an email sent at 2 .8 a .m. between 00:08:22.079 --> 00:08:24.579 two executives right before a bankruptcy filing. 00:08:24.800 --> 00:08:28.079 Oh wow, so it's looking for the delta. Yes. The 00:08:28.079 --> 00:08:30.019 contrast between those two emails is the data. 00:08:30.639 --> 00:08:33.279 It teaches the machine to detect urgency, defection, 00:08:33.899 --> 00:08:36.779 and shifting power dynamics in text. The goal 00:08:36.779 --> 00:08:39.470 isn't teaching morality, it's just... modeling 00:08:39.470 --> 00:08:41.850 reality. Okay, so it's not learning how to commit 00:08:41.850 --> 00:08:44.649 fraud, it's learning the linguistic fingerprints 00:08:44.649 --> 00:08:46.950 of panic. That's a perfect summary, yes. That 00:08:46.950 --> 00:08:49.549 makes a lot of sense. And the sheer scale of 00:08:49.549 --> 00:08:51.710 the human emotion they are analyzing to find 00:08:51.710 --> 00:08:53.929 these fingerprints is staggering. The volume 00:08:53.929 --> 00:08:56.049 is hard to comprehend sometimes. Just looking 00:08:56.049 --> 00:08:58.009 at some of the others here, there's the Netflix 00:08:58.009 --> 00:09:01.570 Prize data set which contains 100 .4 million 00:09:01.570 --> 00:09:06.659 movie ratings from 480 ,000 users. How does a 00:09:06.659 --> 00:09:09.960 machine even process 100 million subjective opinions 00:09:09.960 --> 00:09:12.259 on movies? Through a process broadly known as 00:09:12.259 --> 00:09:15.320 collaborative filtering or matrix factorization. 00:09:15.480 --> 00:09:17.220 OK, break that down for me. Well, AI doesn't 00:09:17.220 --> 00:09:19.120 know what a comedy or a drama is. It doesn't 00:09:19.120 --> 00:09:21.039 care about the plot of the movie at all. It just 00:09:21.039 --> 00:09:23.340 creates a massive mathematical grid. Like a giant 00:09:23.340 --> 00:09:26.370 spreadsheet of preferences. Pretty much. If you 00:09:26.370 --> 00:09:28.750 and I both gave a five -star rating to the exact 00:09:28.750 --> 00:09:32.250 same five obscure movies, the AI plots us close 00:09:32.250 --> 00:09:34.850 together in a mathematical neighborhood. It realizes 00:09:34.850 --> 00:09:37.129 that our tastes share hidden variables. Even 00:09:37.129 --> 00:09:40.610 if we've never met. Exactly. So it can confidently 00:09:40.610 --> 00:09:42.789 predict that you will like a sixth movie that 00:09:42.789 --> 00:09:45.500 I have already rated highly. even if it has no 00:09:45.500 --> 00:09:47.539 idea what the movie is actually about. It maps 00:09:47.539 --> 00:09:51.320 human taste as pure geometry. That is wild. It 00:09:51.320 --> 00:09:53.980 turns our subjective opinions into coordinates 00:09:53.980 --> 00:09:56.460 on a map. It really does. But human communication 00:09:56.460 --> 00:09:59.259 isn't always straightforward, right? What happens 00:09:59.259 --> 00:10:01.759 when we don't mean what we say? There is a data 00:10:01.759 --> 00:10:04.200 set on this list called SPIRs, which stands for 00:10:04.200 --> 00:10:07.399 Sarcasm, Perceived and Intended by Reactive Supervision. 00:10:07.519 --> 00:10:10.100 Oh, yes. Sarcasm, the ultimate challenge for 00:10:10.100 --> 00:10:12.879 AI. It contains 30 ,000 intended and perceived 00:10:12.879 --> 00:10:15.940 sarcastic tweets. How on earth do you teach a 00:10:15.940 --> 00:10:17.940 machine sarcasm? I mean, I know humans who can't 00:10:17.940 --> 00:10:20.600 even detect sarcasm on the internet. It's notoriously 00:10:20.600 --> 00:10:23.500 difficult because sarcasm relies almost entirely 00:10:23.500 --> 00:10:26.659 on context, tone, and the aversion of expectation, 00:10:26.860 --> 00:10:28.820 things that are often completely absent in a 00:10:28.820 --> 00:10:31.279 vacuum of text. Right. It's not what you say, 00:10:31.419 --> 00:10:34.240 it's how you say it. Exactly. If I text you, 00:10:34.559 --> 00:10:37.919 great job. It could be genuine praise or it could 00:10:37.919 --> 00:10:39.960 mean you just spilled coffee all over my laptop. 00:10:40.059 --> 00:10:42.980 Yeah, context is everything. The brilliance of 00:10:42.980 --> 00:10:45.960 the Spires dataset is right there in the name 00:10:45.960 --> 00:10:49.379 reactive supervision. Reactive supervision. What 00:10:49.379 --> 00:10:52.279 does that mean in practice? It means the researchers 00:10:52.279 --> 00:10:54.700 didn't just capture the original tweet in isolation. 00:10:55.240 --> 00:10:57.240 They captured the tweet and the reactions to 00:10:57.240 --> 00:11:00.000 it. Oh, that's smart. Yeah, by showing the AI 00:11:00.000 --> 00:11:02.799 both the statement and how other humans organically 00:11:02.799 --> 00:11:05.019 reacted to that statement, it gives the machine 00:11:05.019 --> 00:11:07.259 a fighting chance to detect the subtle linguistic 00:11:07.259 --> 00:11:10.399 flips, the invisible markers that indicate a 00:11:10.399 --> 00:11:12.960 sentence means the exact opposite of its literal 00:11:12.960 --> 00:11:15.480 words. So it's looking at the ripples the tweet 00:11:15.480 --> 00:11:17.740 caused to understand the rock that was thrown. 00:11:18.120 --> 00:11:20.659 That's clever. It's incredibly clever. And it's 00:11:20.659 --> 00:11:23.440 not just text they are analyzing either. They're 00:11:23.440 --> 00:11:26.639 teaching machines to literally hear our physical 00:11:26.639 --> 00:11:29.440 vulnerabilities. The Parkinson's speech data 00:11:29.440 --> 00:11:31.919 set is on here. Right, moving into audio. It 00:11:31.919 --> 00:11:34.039 contains multiple voice recordings of people 00:11:34.039 --> 00:11:37.059 with and without Parkinson's disease. Physicians 00:11:37.059 --> 00:11:39.620 physically scored the disease severity of the 00:11:39.620 --> 00:11:42.340 patients and then fed the audio to the machine. 00:11:42.679 --> 00:11:46.399 So the AI learns to detect microscopic vocal 00:11:46.399 --> 00:11:48.879 tremors that human ears might completely miss. 00:11:49.419 --> 00:11:52.009 Precisely. The machine converts those audio waves 00:11:52.009 --> 00:11:54.769 into spectrograms, which are visual representations 00:11:54.769 --> 00:11:58.210 of sound frequencies, and analyzes the mathematical 00:11:58.210 --> 00:12:00.190 consistency of the voice. It's turning sound 00:12:00.190 --> 00:12:02.909 into an image, basically. Essentially, yes. And 00:12:02.909 --> 00:12:05.529 we see this massive variety across all human 00:12:05.529 --> 00:12:08.289 communication data. You have the Reddit all comments 00:12:08.289 --> 00:12:11.870 corpus, which has 1 .7 billion comments, teaching 00:12:11.870 --> 00:12:15.330 the machine the absolute raw, unfiltered breadth 00:12:15.330 --> 00:12:18.169 of casual human conversation. Which is terrifying 00:12:18.169 --> 00:12:20.730 in its own right. Truly. And then on the complete 00:12:20.730 --> 00:12:22.289 opposite end of the spectrum, you have PubMed 00:12:22.289 --> 00:12:25.409 Central with 35 million citations of dense, highly 00:12:25.409 --> 00:12:27.669 technical biomedical literature. So it gets the 00:12:27.669 --> 00:12:30.029 bar room chatter and the textbook science. Exactly. 00:12:30.230 --> 00:12:32.629 The machine needs both data sets to truly map 00:12:32.629 --> 00:12:34.970 the full landscape of human language. But human 00:12:34.970 --> 00:12:38.129 behavior is just one piece of the puzzle. Once 00:12:38.129 --> 00:12:40.590 an AI has mapped out our messy conversations 00:12:40.590 --> 00:12:43.409 and our movie tastes, researchers immediately 00:12:43.409 --> 00:12:45.490 point it toward the unforgiving laws of nature. 00:12:46.350 --> 00:12:49.240 We are moving from human feelings to physical 00:12:49.240 --> 00:12:51.960 realities. This is a profound shift in the data. 00:12:52.080 --> 00:12:54.240 It really is. This is where machine learning 00:12:54.240 --> 00:12:56.860 stops modeling human behavior and transitions 00:12:56.860 --> 00:12:59.659 into driving actual scientific discovery. We 00:12:59.659 --> 00:13:02.659 are feeding AI the fundamental building blocks 00:13:02.659 --> 00:13:05.080 of the universe. And it gets incredibly complex 00:13:05.080 --> 00:13:07.200 very quickly. Look at the chemistry data sets 00:13:07.200 --> 00:13:10.179 on this compendium. There is one from 2025 called 00:13:10.179 --> 00:13:13.940 the Open Riasity CH -NEFH data set. A bit of 00:13:13.940 --> 00:13:16.679 a mouthful, but very important. Right. It contains 00:13:16.679 --> 00:13:19.700 tens of thousands of atomic configurations, mapping 00:13:19.700 --> 00:13:21.860 exact energies and forces to train what they 00:13:21.860 --> 00:13:23.840 call machine learning interatomic potentials. 00:13:23.840 --> 00:13:26.480 Yes. It's literally teaching the AI how carbon, 00:13:26.720 --> 00:13:29.039 hydrogen, oxygen, and nitrogen interact at a 00:13:29.039 --> 00:13:31.419 quantum level. It's mapping molecular reality. 00:13:31.659 --> 00:13:34.539 And then, moving into biology, you have datasets, 00:13:34.799 --> 00:13:38.120 like the mushroom dataset. It has over 8 ,000 00:13:38.120 --> 00:13:40.600 instances used to classify if a wild mushroom 00:13:40.600 --> 00:13:43.179 is toxic or safe to eat based on its physical 00:13:43.179 --> 00:13:46.059 characteristics. But wait, how does a machine 00:13:46.059 --> 00:13:48.220 look at a mushroom? It doesn't have eyes. This 00:13:48.220 --> 00:13:50.419 goes back to the tabular data we discussed earlier. 00:13:50.799 --> 00:13:52.659 When we say the AI is looking at a mushroom, 00:13:52.879 --> 00:13:55.460 it's not analyzing a photograph. So what is it 00:13:55.460 --> 00:13:57.179 doing? It is ingesting a massive spreadsheet. 00:13:57.620 --> 00:13:59.779 It's looking at columns for specific traits, 00:14:00.059 --> 00:14:03.559 cap shape, cap color, gill spacing, odor, stock 00:14:03.559 --> 00:14:06.919 root. It takes all those variables and mathematically 00:14:06.919 --> 00:14:09.799 calculates the probability of that specific combination 00:14:09.799 --> 00:14:12.460 equaling poisonous. OK, so it strips away the 00:14:12.460 --> 00:14:14.559 visual of the mushroom and turns its biology 00:14:14.559 --> 00:14:17.460 into a pure statistical equation. Exactly. It 00:14:17.460 --> 00:14:19.220 doesn't see a mushroom. It sees a math problem. 00:14:19.419 --> 00:14:22.299 So what does this all mean? The AI is acting 00:14:22.299 --> 00:14:25.600 like a virtual biologist. a physicist. It is 00:14:25.600 --> 00:14:28.100 dissecting digital mushrooms, measuring the brain 00:14:28.100 --> 00:14:30.500 waves of alcoholics in the EEG database with 00:14:30.500 --> 00:14:33.480 64 electrodes, and running virtual particle accelerators. 00:14:33.480 --> 00:14:35.820 Taking on the role of the scientist. But I have 00:14:35.820 --> 00:14:37.820 to stop and ask a critical question about how 00:14:37.820 --> 00:14:40.000 this actually works. Sure. Go ahead. Earlier, 00:14:40.080 --> 00:14:42.440 we talked about the Netflix data set having 100 00:14:42.440 --> 00:14:45.179 million ratings. To learn heat and taste, it 00:14:45.179 --> 00:14:48.159 needed 100 million data points. Right. Massive 00:14:48.159 --> 00:14:51.120 volume. But looking at the Earth and space section 00:14:51.120 --> 00:14:54.399 here, there's the Challenger USA Space Shuttle 00:14:54.399 --> 00:14:57.500 O -Ring data set. It is used to predict O -Ring 00:14:57.500 --> 00:15:00.179 failures based on launch temperatures and past 00:15:00.179 --> 00:15:02.980 flight data. A very famous and tragic case study. 00:15:03.120 --> 00:15:07.470 Yeah. But it only has 23 instances. Just 23. 00:15:08.090 --> 00:15:10.710 How can an AI possibly learn anything meaningful 00:15:10.710 --> 00:15:13.850 or reliable from just 23 data points? Doesn't 00:15:13.850 --> 00:15:16.669 it need massive volume to learn? This raises 00:15:16.669 --> 00:15:18.649 an important question about the fundamental difference 00:15:18.649 --> 00:15:21.629 between behavioral data and physical data. You're 00:15:21.629 --> 00:15:24.049 comparing apples to gravity here. Apples to gravity. 00:15:24.230 --> 00:15:26.429 I like that. When an AI looks at movie ratings 00:15:26.429 --> 00:15:29.230 or Reddit comments, human behavior is noisy, 00:15:29.669 --> 00:15:32.009 contradictory, and highly subjective. You need 00:15:32.009 --> 00:15:34.409 tens of millions of examples to find a reliable 00:15:34.409 --> 00:15:36.549 statistical signal through all that human noise. 00:15:36.629 --> 00:15:38.629 Because we change our minds constantly. Right. 00:15:38.929 --> 00:15:41.690 But physical systems obey strict, unyielding 00:15:41.690 --> 00:15:44.429 laws. So because physics doesn't change its mind, 00:15:44.750 --> 00:15:47.590 the data is somehow cleaner. It's not just cleaner. 00:15:47.830 --> 00:15:50.470 It is... infinitely more dense. Yeah. Think about 00:15:50.470 --> 00:15:53.330 it this way. When a rubber O -ring freezes, its 00:15:53.330 --> 00:15:56.990 elasticity degrades the exact same way every 00:15:56.990 --> 00:15:59.710 single time based on the temperature. It's entirely 00:15:59.710 --> 00:16:02.529 predictable. Yes. Let's look at another physical 00:16:02.529 --> 00:16:04.350 data set on the list to illustrate this, the 00:16:04.350 --> 00:16:06.850 Musk data set, which is used in drug discovery. 00:16:07.450 --> 00:16:09.889 To predict if a new molecule act as a Musk scent, 00:16:10.490 --> 00:16:12.470 researchers didn't just give the AI the name 00:16:12.470 --> 00:16:14.470 of the molecule. What did they give it? They 00:16:14.470 --> 00:16:18.269 extracted 168 highly specific mathematical and 00:16:18.269 --> 00:16:20.710 chemical features for each individual molecule. 00:16:21.350 --> 00:16:23.730 When you have highly detailed, precisely measured 00:16:23.730 --> 00:16:26.049 features like the exact temperature, the atmospheric 00:16:26.049 --> 00:16:28.789 pressure, and the specific material stress in 00:16:28.789 --> 00:16:31.529 the Challenger O -Ring data, a very small number 00:16:31.529 --> 00:16:34.950 of instances can be incredibly dense with predictive 00:16:34.950 --> 00:16:37.250 value. Oh, I see. The algorithm doesn't need 00:16:37.250 --> 00:16:39.190 millions of guesses if it's handed a perfectly 00:16:39.190 --> 00:16:41.909 detailed blueprint. Exactly. Quality and depth 00:16:41.909 --> 00:16:43.909 over sheer volume. And this actually matters 00:16:43.909 --> 00:16:46.870 for the listener, right? Because the exact same 00:16:46.870 --> 00:16:50.070 mathematical logic that allows an AI to predict 00:16:50.070 --> 00:16:52.889 a space shuttle overing failure from a small 00:16:52.889 --> 00:16:55.509 data set is what's running in the background, 00:16:55.750 --> 00:16:58.149 predicting when your car's transmission is going 00:16:58.149 --> 00:17:00.889 to slip. Or when the commercial flight you are 00:17:00.889 --> 00:17:03.429 waiting for is going to be delayed due to a mechanical 00:17:03.429 --> 00:17:06.430 issue. Right. It's high stakes prediction based 00:17:06.430 --> 00:17:10.009 on dense physical rules. Exactly. Once you extract 00:17:10.009 --> 00:17:12.460 the right physical features, The machine can 00:17:12.460 --> 00:17:14.619 map the future of that object with terrifying 00:17:14.619 --> 00:17:17.960 accuracy. And that clicks. So the machine understands 00:17:17.960 --> 00:17:20.240 the natural world. It understands human behavior. 00:17:21.180 --> 00:17:23.619 The next logical step is that we immediately 00:17:23.619 --> 00:17:26.579 put it to work solving the incredibly complex 00:17:26.579 --> 00:17:28.579 artificial systems we've built for ourselves. 00:17:28.740 --> 00:17:31.220 Naturally. I'm talking about our games, our transit 00:17:31.220 --> 00:17:34.019 networks, our cyber battles. Yes, the modeling 00:17:34.019 --> 00:17:37.079 of complex human constructs. This requires a 00:17:37.079 --> 00:17:39.339 leap into what we call multivariate data sets. 00:17:39.500 --> 00:17:41.900 Meaning what, exactly? Meaning the AI isn't just 00:17:41.900 --> 00:17:44.500 looking at one column of data, like temperature, 00:17:44.799 --> 00:17:46.960 and trying to predict an outcome. It is forced 00:17:46.960 --> 00:17:50.160 to balance dozens of intersecting realities simultaneously. 00:17:50.380 --> 00:17:52.940 Like spinning a bunch of plates at once. Exactly. 00:17:53.400 --> 00:17:56.460 An outcome in a multivariate data set is dependent 00:17:56.460 --> 00:17:59.039 on many different variables constantly shifting 00:17:59.039 --> 00:18:01.259 and reacting to one another. The gaming data 00:18:01.259 --> 00:18:03.079 sets are a perfect example of this. You have 00:18:03.079 --> 00:18:05.819 the poker hand data set, which mathematically 00:18:05.819 --> 00:18:08.960 categorizes over one million five -card hands. 00:18:09.180 --> 00:18:11.160 Right. Tracking probabilities. And there's an 00:18:11.160 --> 00:18:13.960 entire data set dedicated just to tic -tac -toe 00:18:13.960 --> 00:18:16.880 end games mapping every possible logical outcome. 00:18:17.240 --> 00:18:20.319 The AI is learning strategy. But then you take 00:18:20.319 --> 00:18:23.000 that strategic logic and scale it up to the size 00:18:23.000 --> 00:18:24.920 of a whole city. Which is an incredible leap 00:18:24.920 --> 00:18:27.660 in complexity. The New York City taxi trip data 00:18:27.660 --> 00:18:30.690 is on this list. It tracks pictures drop -offs 00:18:30.690 --> 00:18:32.930 and fair amounts for every single yellow and 00:18:32.930 --> 00:18:36.109 green cab across six full years. That's millions 00:18:36.109 --> 00:18:38.029 and millions of rides. And in the cybersecurity 00:18:38.029 --> 00:18:41.130 realm there is the Witty Worm data set. It meticulously 00:18:41.130 --> 00:18:43.170 tracks the spread of a malicious computer worm 00:18:43.170 --> 00:18:47.789 across exactly 55 ,909 IP addresses. Think about 00:18:47.789 --> 00:18:50.509 the immense computational power required to understand 00:18:50.509 --> 00:18:53.519 systems of that scale. A city's traffic grid 00:18:53.519 --> 00:18:56.359 during rush hour or the infection vector of a 00:18:56.359 --> 00:18:59.579 global internet worm represents millions of micro 00:18:59.579 --> 00:19:01.980 decisions happening concurrently. It's just staggering 00:19:01.980 --> 00:19:05.200 to think about. It is. By feeding these massive 00:19:05.200 --> 00:19:07.539 multivariate data sets to machine learning models, 00:19:07.900 --> 00:19:11.279 we are asking the AI to find the macro patterns 00:19:11.279 --> 00:19:14.140 that our human brains simply do not have the 00:19:14.140 --> 00:19:16.720 processing power to see all at once. It's basically 00:19:16.720 --> 00:19:20.099 like playing SimCity but on god mode. The AI 00:19:20.099 --> 00:19:22.720 has the entire history historical flow of metropolis 00:19:22.720 --> 00:19:24.900 mapped out in its brain. That's a great analogy. 00:19:25.039 --> 00:19:26.940 But wait, here's where my logic gets a bit stuck. 00:19:27.119 --> 00:19:29.779 Uh -oh. Where's the snag? If an AI learns how 00:19:29.779 --> 00:19:32.660 to optimize a city's traffic patterns based entirely 00:19:32.660 --> 00:19:35.720 on a six -year block of NYC taxi data from the 00:19:35.720 --> 00:19:38.579 past, what happens when the city builds a massive 00:19:38.579 --> 00:19:41.420 new network of bike lanes? Or what happens when 00:19:41.420 --> 00:19:43.720 a global pandemic hits, nobody commutes to the 00:19:43.720 --> 00:19:46.940 office, and the entire concept of rush hour disappears 00:19:46.940 --> 00:19:49.539 overnight? Aren't these data sets essentially 00:19:49.539 --> 00:19:52.339 just freezing human society in amber? If we connect 00:19:52.339 --> 00:19:54.660 this to the bigger picture, you have just hit 00:19:54.660 --> 00:19:57.539 on one of the most vital existential challenges 00:19:57.539 --> 00:20:01.059 in machine learning today. Really? Yes. Datasets 00:20:01.059 --> 00:20:04.119 are, by their very definition, historical artifacts. 00:20:04.380 --> 00:20:07.059 They are fossils. Look closely at the financial 00:20:07.059 --> 00:20:09.799 and weather data on this list. The Dow Jones 00:20:09.799 --> 00:20:12.380 index dataset they used to train financial models 00:20:12.380 --> 00:20:14.900 is specifically from the first and second quarters 00:20:14.900 --> 00:20:18.039 of 2011. Oh, wow. That's over a decade ago. Right. 00:20:18.279 --> 00:20:20.960 And the El Nino dataset contains surface meteorological 00:20:20.960 --> 00:20:23.640 readings from ocean booties collected decades 00:20:23.640 --> 00:20:27.500 ago. These are exact, unmoving snapshots in time. 00:20:27.589 --> 00:20:30.769 So an AI trained to trade stocks based on 2011 00:20:30.769 --> 00:20:34.089 data might be completely, disastrously lost in 00:20:34.089 --> 00:20:36.410 today's market condition. Without a doubt, the 00:20:36.410 --> 00:20:38.690 synthesis of this entire field is that knowledge 00:20:38.690 --> 00:20:41.309 is most valuable and most dangerous when it is 00:20:41.309 --> 00:20:43.250 understood within its temporal context. Because 00:20:43.250 --> 00:20:45.710 the rules keep changing. You absolutely cannot 00:20:45.710 --> 00:20:48.369 expect a model trained on traffic data from 2015 00:20:48.369 --> 00:20:51.710 to seamlessly and safely navigate a city in 2026 00:20:51.710 --> 00:20:54.470 without retraining. The baseline rules of the 00:20:54.470 --> 00:20:55.829 environment have changed. And that makes total 00:20:55.829 --> 00:20:58.819 sense. This is why the curation, updating, and 00:20:58.819 --> 00:21:01.299 maintenance of these open data portals is an 00:21:01.299 --> 00:21:04.819 endless monumental task. The world changes every 00:21:04.819 --> 00:21:07.559 second, and the fuel we feed the machine has 00:21:07.559 --> 00:21:09.799 to constantly change with it. Or else it's running 00:21:09.799 --> 00:21:12.460 on obsolete logic. Otherwise, the machine will 00:21:12.460 --> 00:21:15.099 confidently give you an answer that is mathematically 00:21:15.099 --> 00:21:18.059 perfectly correct for a world that no longer 00:21:18.059 --> 00:21:22.279 exists. Wow. That is a sobering thought. To recap 00:21:22.279 --> 00:21:24.599 the incredible journey we've just been on, we 00:21:24.599 --> 00:21:27.200 started by looking at the massive open data portals, 00:21:27.279 --> 00:21:29.799 the carefully regulated gas stations of the AI 00:21:29.799 --> 00:21:32.900 industry. The unsung heroes. We saw how algorithms 00:21:32.900 --> 00:21:35.960 learn the messy reality of human emotion, not 00:21:35.960 --> 00:21:38.619 by reading text, but by parsing the invisible 00:21:38.619 --> 00:21:41.480 markers of sarcasm in the spires data set and 00:21:41.480 --> 00:21:43.960 measuring the panic in the Enron emails. Modeling 00:21:43.960 --> 00:21:46.680 our chaos. We watched them shift from human noise 00:21:46.680 --> 00:21:49.579 to the unforgiving laws of physics. Using highly 00:21:49.579 --> 00:21:52.000 dense data to predict chemical reactions in the 00:21:52.000 --> 00:21:54.539 Open Riasi -T dataset and space shuttle O -ring 00:21:54.539 --> 00:21:57.630 failures. Acting as virtual scientists. And finally, 00:21:57.829 --> 00:22:00.950 we saw them tackle our massive modern constructs, 00:22:01.029 --> 00:22:03.730 tracking millions of New York taxi rides and 00:22:03.730 --> 00:22:06.329 predicting the spread of global cyber pathogens. 00:22:06.849 --> 00:22:09.529 It really drives home the reality that every 00:22:09.529 --> 00:22:12.329 single digital footprint you leave behind, every 00:22:12.329 --> 00:22:14.430 rating you give, every route you take on your 00:22:14.430 --> 00:22:17.809 GPS, every email you send, might end up in one 00:22:17.809 --> 00:22:20.769 of these repositories actively teaching the next 00:22:20.769 --> 00:22:23.789 generation of machines how to perceive your world. 00:22:24.170 --> 00:22:26.269 And that leads to a rather profound final thought. 00:22:26.269 --> 00:22:28.750 Let's hear it. If you look at the entire timeline 00:22:28.750 --> 00:22:31.089 of this compendium, it spans across generations 00:22:31.089 --> 00:22:33.890 of human history. On one end, we have the iris 00:22:33.890 --> 00:22:36.390 plant data set containing fundamental biological 00:22:36.390 --> 00:22:38.410 measurements that were literally collected by 00:22:38.410 --> 00:22:41.490 hand back in 1936. 1936. That's almost a century 00:22:41.490 --> 00:22:43.910 off. Exactly. And on the other end, we have the 00:22:43.910 --> 00:22:47.670 open reasset chemistry benchmark forged in 2025. 00:22:48.250 --> 00:22:49.910 Machine learning models are continually trained 00:22:49.910 --> 00:22:51.950 on these frozen moments in time, which makes 00:22:51.950 --> 00:22:54.970 you wonder. Are our future artificial intelligences 00:22:54.970 --> 00:22:58.170 secretly carrying the ghost of our past? How 00:22:58.170 --> 00:23:01.190 deeply does the age, the era, and the historical 00:23:01.190 --> 00:23:04.029 context of a data set shape the literal mind 00:23:04.029 --> 00:23:06.549 of the AI that is currently making decisions 00:23:06.549 --> 00:23:09.009 about your life today? That is definitely a lot 00:23:09.009 --> 00:23:10.910 to think about the next time you leave a trace 00:23:10.910 --> 00:23:13.269 of digital exhaust behind. Thank you for joining 00:23:13.269 --> 00:23:15.250 us on this deep dive into the source material. 00:23:15.450 --> 00:23:16.309 We'll catch you next time.