WEBVTT 00:00:00.000 --> 00:00:02.160 You know, when we normally think about mathematics, 00:00:02.439 --> 00:00:06.019 we usually picture something inherently rigid. 00:00:06.280 --> 00:00:08.220 Oh, absolutely, like very black and white. Right. 00:00:08.359 --> 00:00:11.099 We picture formulas on a dusty chalkboard or, 00:00:11.099 --> 00:00:13.240 I don't know, calculating the exact trajectory 00:00:13.240 --> 00:00:17.620 of a rocket. It's just a world of absolute undeniable 00:00:17.620 --> 00:00:20.460 precision. Yeah, it's the ultimate exact science. 00:00:20.699 --> 00:00:22.379 But then you look at what is happening today 00:00:22.379 --> 00:00:25.679 with technology. We have machines that are painting 00:00:25.679 --> 00:00:28.920 these like award winning pictures. They're composing 00:00:28.920 --> 00:00:32.000 original symphonies and writing incredibly nuanced 00:00:32.000 --> 00:00:34.899 essays. And we are told that beneath all of that 00:00:34.899 --> 00:00:39.780 profound chaotic creativity, it is still just 00:00:39.780 --> 00:00:43.340 math. It really is the ultimate paradox, isn't 00:00:43.340 --> 00:00:45.340 it? Yeah. Because we are taking the most rigid, 00:00:45.640 --> 00:00:48.380 definitive tool humanity has ever invented, and 00:00:48.380 --> 00:00:51.740 we are using it to simulate the most fluid, subjective 00:00:51.740 --> 00:00:53.759 thing humanity has ever experienced. Which is 00:00:53.759 --> 00:00:55.899 the act of creation itself. Exactly. And that 00:00:55.899 --> 00:00:58.420 is exactly why we are here today. I am speaking 00:00:58.420 --> 00:01:00.979 directly to you, the learner. We are tailoring 00:01:00.979 --> 00:01:03.159 this deep dive specifically for you to answer 00:01:03.159 --> 00:01:05.319 a question you've likely had while watching this 00:01:05.319 --> 00:01:07.560 whole technology boom. Right, because you hear 00:01:07.560 --> 00:01:11.299 the phrase generative AI constantly. It's in 00:01:11.299 --> 00:01:13.540 the news. It's transforming your workplace. It's 00:01:13.540 --> 00:01:15.680 sitting quietly on your phone right now. But 00:01:15.680 --> 00:01:18.359 what does the generative part actually mean on 00:01:18.359 --> 00:01:20.280 a purely mathematical level? On a mechanical 00:01:20.280 --> 00:01:23.040 level, yeah. Today's mission is to completely 00:01:23.040 --> 00:01:26.000 bypass the corporate tech jargon. We are going 00:01:26.000 --> 00:01:29.280 to decode the actual statistical engine driving 00:01:29.280 --> 00:01:32.299 this modern boom. And to do that, we're grounding 00:01:32.299 --> 00:01:35.799 our discussion. in a single, incredibly comprehensive 00:01:35.799 --> 00:01:38.739 source today. It's a slightly dense Wikipedia 00:01:38.739 --> 00:01:41.040 article on the genitive model. Slightly dense 00:01:41.040 --> 00:01:43.579 is putting it mildly. Fair enough. If you were 00:01:43.579 --> 00:01:45.680 to just pull this article up, you would see what 00:01:45.680 --> 00:01:48.560 looks like an intimidating, impenetrable wall 00:01:48.560 --> 00:01:51.319 of equations. Just heavy statistical notation. 00:01:51.379 --> 00:01:53.219 Yeah, it looks like a graduate level textbook. 00:01:53.319 --> 00:01:56.859 It does. But hidden inside that math is the actual 00:01:56.859 --> 00:01:59.640 source code. the fundamental architecture of 00:01:59.640 --> 00:02:02.180 how a machine learns to create. And we're going 00:02:02.180 --> 00:02:04.359 to translate all of that into plain English. 00:02:04.939 --> 00:02:06.980 And you are going to want to stick around for 00:02:06.980 --> 00:02:09.080 this because by the end of this deep dive, you 00:02:09.080 --> 00:02:12.740 will understand exactly why a somewhat obscure 00:02:12.740 --> 00:02:15.759 experiment from 1948. A great experiment, by 00:02:15.759 --> 00:02:18.120 the way. Oh, it's fascinating. An experiment 00:02:18.120 --> 00:02:23.020 that produced the absolute nonsense phrase. Representing 00:02:23.020 --> 00:02:26.419 and speedily is in good. Catchy. Super catchy. 00:02:26.800 --> 00:02:30.199 You'll learn why that exact gibberish is the 00:02:30.199 --> 00:02:34.539 direct undeniable ancestor to today's massive 00:02:34.539 --> 00:02:37.639 billion parameter music and image generators. 00:02:37.960 --> 00:02:40.360 It is a fascinating lineage. But you know, before 00:02:40.360 --> 00:02:42.680 we can understand how machines actually create 00:02:42.680 --> 00:02:45.360 data, we have to understand the baseline. Right. 00:02:45.379 --> 00:02:47.960 Where we started. Exactly. We have to look at 00:02:47.960 --> 00:02:50.080 the traditional way machines have historically 00:02:50.080 --> 00:02:53.069 categorized data. Because in statistical classification, 00:02:53.569 --> 00:02:55.610 there are two fundamentally different philosophies. 00:02:55.750 --> 00:02:57.810 OK, let's establish that baseline. Because most 00:02:57.810 --> 00:02:59.409 of the AI we've interacted with over the last 00:02:59.409 --> 00:03:01.389 decade, I mean, it hasn't been creating things, 00:03:01.530 --> 00:03:04.169 it's been sorting things. Yes, sorting and tagging. 00:03:04.210 --> 00:03:06.050 Like it's been looking at your email and deciding 00:03:06.050 --> 00:03:08.889 if it's spam or not spam, or it's looking at 00:03:08.889 --> 00:03:10.789 a photo on your phone and deciding if it's a 00:03:10.789 --> 00:03:14.370 cat or a dog. Precisely. And in the world of 00:03:14.370 --> 00:03:17.419 machine learning, That sorting process relies 00:03:17.419 --> 00:03:19.500 on what we call the discriminative approach. 00:03:20.219 --> 00:03:22.740 The other philosophy, the one driving today's 00:03:22.740 --> 00:03:26.199 creative boom, is the generative approach. And 00:03:26.199 --> 00:03:28.659 they view the exact same data in completely different 00:03:28.659 --> 00:03:31.039 ways. So let's break down the discriminative 00:03:31.039 --> 00:03:32.520 approach first, because I think that's the one 00:03:32.520 --> 00:03:35.120 most people implicitly understand. How does it 00:03:35.120 --> 00:03:37.979 work mathematically? Well, a discriminative model 00:03:37.979 --> 00:03:40.919 is incredibly focused. It learns the conditional 00:03:40.919 --> 00:03:43.960 probability of a target given an observation. 00:03:44.240 --> 00:03:46.259 Okay, so what does that mean in plain English? 00:03:46.800 --> 00:03:49.620 It means if you hand the model a piece of data, 00:03:50.020 --> 00:03:54.300 say a photograph, it simply asks, given the pixels 00:03:54.300 --> 00:03:57.300 in this image, what is the probability that the 00:03:57.300 --> 00:04:00.780 label for this image is cat? Ah, I see. It just 00:04:00.780 --> 00:04:02.879 tries to draw a mathematical boundary between 00:04:02.879 --> 00:04:05.620 the cats and the dogs. Mathematically, that's 00:04:05.620 --> 00:04:08.900 P of Y given X. So it's just drawing a line in 00:04:08.900 --> 00:04:10.400 the sound. Exactly. It doesn't care about the 00:04:10.400 --> 00:04:13.129 anatomy of a cat. It doesn't care about... the 00:04:13.129 --> 00:04:14.949 lighting in the photograph or the concept of 00:04:14.949 --> 00:04:17.529 fur. It's just looking for shortcuts. Yes. It 00:04:17.529 --> 00:04:19.509 just looks for the bare minimum mathematical 00:04:19.509 --> 00:04:21.790 signals required to safely categorize the data 00:04:21.790 --> 00:04:24.470 and move on. OK, so a discriminative model is 00:04:24.470 --> 00:04:26.569 like an art critic. Ooh, I like that. Yeah, it 00:04:26.569 --> 00:04:28.449 walks into a gallery, looks at a painting, and 00:04:28.449 --> 00:04:31.029 says, yes, this is a Picasso, or no, this is 00:04:31.029 --> 00:04:33.129 a Monet. It is just categorizing what is already 00:04:33.129 --> 00:04:35.430 in front of it. Right. But a generative model 00:04:35.430 --> 00:04:40.209 is the actual artist. It has studied the underlying 00:04:40.209 --> 00:04:42.949 process of the art so incredibly well. It has 00:04:42.949 --> 00:04:45.870 mapped out the very DNA of the style that it 00:04:45.870 --> 00:04:48.410 can paint you a brand new, completely original 00:04:48.410 --> 00:04:51.769 Picasso. What's fascinating here is how perfectly 00:04:51.769 --> 00:04:54.889 that analogy aligns with the actual statistical 00:04:54.889 --> 00:04:58.230 architecture from the source. Really? Yes. The 00:04:58.230 --> 00:05:00.529 discriminative algorithm, your art critic does 00:05:00.529 --> 00:05:02.230 not care at all how the data was generated. It 00:05:02.230 --> 00:05:04.290 just takes a signal and categorizes it. Makes 00:05:04.290 --> 00:05:07.360 sense. But a generative algorithm... actively 00:05:07.360 --> 00:05:10.459 models the entire data generating process. Instead 00:05:10.459 --> 00:05:13.019 of just drawing a line between categories, it 00:05:13.019 --> 00:05:15.579 models what we call the joint probability distribution. 00:05:16.060 --> 00:05:19.060 Okay, wait. Joint probability distribution sounds 00:05:19.060 --> 00:05:21.480 like exactly the kind of textbook jargon we promised 00:05:21.480 --> 00:05:25.220 to avoid. Fair point, fair point. What does mapping 00:05:25.220 --> 00:05:28.199 the joint probability actually mean in practice? 00:05:28.839 --> 00:05:31.180 Think of it this way. Instead of just asking 00:05:31.180 --> 00:05:35.240 is this a cat or a dog, a generative model asks 00:05:35.240 --> 00:05:38.019 a much deeper, more fundamental question. Which 00:05:38.019 --> 00:05:41.019 is? It asks, based on all the data I've ever 00:05:41.019 --> 00:05:43.660 seen, what are all the possible combinations 00:05:43.660 --> 00:05:46.459 of pixels that make up a cat, and how likely 00:05:46.459 --> 00:05:49.000 is each combination to occur in the real world? 00:05:49.079 --> 00:05:51.879 Oh, wow. It's learning the underlying rules of 00:05:51.879 --> 00:05:54.439 reality. mathematically speaking, rather than 00:05:54.439 --> 00:05:56.600 just learning how to sort the outputs of reality. 00:05:56.639 --> 00:05:59.420 That's p of x, y, the joint probability. It's 00:05:59.420 --> 00:06:01.379 a completely different philosophical approach. 00:06:01.480 --> 00:06:03.480 The critic discriminates, the artist generates. 00:06:03.660 --> 00:06:05.360 Exactly. But, you know, being an artist sounds 00:06:05.360 --> 00:06:07.800 like magic. And yet the source material insists 00:06:07.800 --> 00:06:10.420 that this profound ability to create all boils 00:06:10.420 --> 00:06:13.220 down to probability tables. It really does. So 00:06:13.220 --> 00:06:15.420 how does knowing this joint probability actually 00:06:15.420 --> 00:06:18.480 let a machine generate a synthetic image or like 00:06:18.480 --> 00:06:21.079 synthetic text? To really grasp the mechanics 00:06:21.079 --> 00:06:23.370 of that, we have to walk through a wonderfully 00:06:23.370 --> 00:06:25.990 simple mathematical example provided right in 00:06:25.990 --> 00:06:28.329 the text. OK, let's hear it. It strips away all 00:06:28.329 --> 00:06:30.470 the billions of parameters, all the neural networks, 00:06:30.649 --> 00:06:33.470 and just leaves the raw underlying logic. I really 00:06:33.470 --> 00:06:36.439 love this example. So learner. Imagine we are 00:06:36.439 --> 00:06:39.439 looking at a tiny microscopic universe of data. 00:06:39.800 --> 00:06:41.860 There are only four data points in existence. 00:06:42.040 --> 00:06:44.620 That's it. That's four points. Our input, which 00:06:44.620 --> 00:06:47.019 we'll call the observation, can only be the number 00:06:47.019 --> 00:06:49.980 one or the number two, and our label, the category, 00:06:50.040 --> 00:06:52.639 can only be a zero or a one. Keep it super simple. 00:06:52.800 --> 00:06:55.860 So our entire universe consists of four exact 00:06:55.860 --> 00:06:59.100 combinations. The input one with label zero, 00:06:59.420 --> 00:07:01.720 input one with label one, input two with label 00:07:01.720 --> 00:07:04.620 zero, and input two with label one. Perfectly 00:07:04.620 --> 00:07:07.100 simple. Now, let's look at how our two different 00:07:07.100 --> 00:07:10.040 models, our critic and our artist, perceive this 00:07:10.040 --> 00:07:12.639 tiny four -point universe. OK. The discriminative 00:07:12.639 --> 00:07:14.920 model is just looking for a quick answer. If 00:07:14.920 --> 00:07:17.459 we hand it an input of 1, it looks at the universe 00:07:17.459 --> 00:07:19.459 and sees that half the time the label is 0 and 00:07:19.459 --> 00:07:21.399 half the time the label is 1. Right. It just 00:07:21.399 --> 00:07:24.459 sees a 50 -50 coin toss. Exactly. And if we give 00:07:24.459 --> 00:07:26.800 it an input of 2, it's the exact same thing. 00:07:27.040 --> 00:07:29.639 Another 50 -50 coin toss. So it's not really 00:07:29.639 --> 00:07:32.360 learning much. No. The discriminative model essentially 00:07:32.360 --> 00:07:36.120 shrugs. It just sees a flat unhelpful probability 00:07:36.120 --> 00:07:38.920 for every single input. It's a very narrow view. 00:07:39.540 --> 00:07:42.259 But the generative model looks at the exact same 00:07:42.259 --> 00:07:45.199 four points and doesn't just see individual coin 00:07:45.199 --> 00:07:48.019 tosses. It sees the architecture. Yes, because 00:07:48.019 --> 00:07:50.720 it maps that joint probability. It maps the entire 00:07:50.720 --> 00:07:52.959 empirical measure. The whole landscape. Right. 00:07:53.139 --> 00:07:54.920 So instead of just looking at the coin talk after 00:07:54.920 --> 00:07:57.699 the fact, it sees that each of the four possible 00:07:57.699 --> 00:08:01.040 scenarios makes up exactly 25 percent or one 00:08:01.040 --> 00:08:03.500 fourth of the total reality. It understands the 00:08:03.500 --> 00:08:05.879 holistic shape of the data. And because it maps 00:08:05.879 --> 00:08:08.660 that full landscape, it understands the distribution 00:08:08.660 --> 00:08:10.439 well enough to just reach into that landscape 00:08:10.439 --> 00:08:13.420 and pull out entirely new samples. Examples that 00:08:13.420 --> 00:08:15.899 resemble the observed data, which the source 00:08:15.899 --> 00:08:18.519 calls synthetic data generation. And if you ever 00:08:18.519 --> 00:08:20.399 need to translate between the artist's perspective 00:08:20.399 --> 00:08:22.579 and the critic's perspective, there's a very 00:08:22.579 --> 00:08:24.779 famous mathematical bridge called Bayes' Rule. 00:08:25.019 --> 00:08:27.240 Oh, I've heard of Bayes' rule, but it usually 00:08:27.240 --> 00:08:30.160 comes with a massive terrifying equation attached 00:08:30.160 --> 00:08:32.360 to it. We can skip the equation entirely today. 00:08:32.460 --> 00:08:34.559 Thank goodness. Just think of Bayes' rule as 00:08:34.559 --> 00:08:37.240 a mathematical time machine. It lets you reverse 00:08:37.240 --> 00:08:40.320 engineer a probability. How so? If you know the 00:08:40.320 --> 00:08:43.340 generative landscape, the blueprint of how everything 00:08:43.340 --> 00:08:46.399 in the universe is made, Bayes' rule lets you 00:08:46.399 --> 00:08:49.899 easily flip that information to answer a specific 00:08:49.899 --> 00:08:52.970 discriminative question. You can use the artist's 00:08:52.970 --> 00:08:55.190 deep knowledge to do the critic's job perfectly. 00:08:55.470 --> 00:08:58.210 Okay, let's unpack this. Wait. I'm stuck on something 00:08:58.210 --> 00:09:00.230 here. What's that? If the generative model has 00:09:00.230 --> 00:09:02.889 to map out every single possible combination 00:09:02.889 --> 00:09:06.059 of inputs... and labels the entire landscape, 00:09:06.120 --> 00:09:08.179 like you said, isn't that incredibly inefficient? 00:09:08.919 --> 00:09:11.379 I mean, in our microscopic four -point universe, 00:09:11.600 --> 00:09:13.860 sure, it's easy. But what if I just want a simple 00:09:13.860 --> 00:09:15.960 answer to a simple question? Give me an example. 00:09:16.220 --> 00:09:19.080 Well, what if I just want an AI to look at a 00:09:19.080 --> 00:09:21.860 10 -megapixel photo and tell me if it's a picture 00:09:21.860 --> 00:09:24.539 of a stop sign? Why would I want it to mathematically 00:09:24.539 --> 00:09:27.179 map out the jort probability of every single 00:09:27.179 --> 00:09:29.899 stop sign in the known universe? If we connect 00:09:29.899 --> 00:09:32.990 this to the bigger picture, You have just hit 00:09:32.990 --> 00:09:36.529 on the exact reason why generative models aren't 00:09:36.529 --> 00:09:39.090 always the best tool for every job. Oh, really? 00:09:39.490 --> 00:09:41.730 Yeah, the source explicitly highlights this. 00:09:42.809 --> 00:09:46.690 If your only goal is classification, just spotting 00:09:46.690 --> 00:09:50.289 the stop sign, discriminative models often perform 00:09:50.289 --> 00:09:52.730 vastly better, and they are much, much cheaper 00:09:52.730 --> 00:09:54.649 to run. Because they don't overcomplicate it. 00:09:54.789 --> 00:09:56.590 They aren't trying to understand what a stop 00:09:56.590 --> 00:09:58.470 sign is, just what it looks like mathematically. 00:09:58.850 --> 00:10:01.049 Right. Generative models make a massive amount 00:10:01.049 --> 00:10:03.149 of assumptions about the true distribution of 00:10:03.149 --> 00:10:05.779 the data. They suffer from what statisticians 00:10:05.779 --> 00:10:08.559 call the curse of dimensionality. The curse of 00:10:08.559 --> 00:10:11.059 dimensionality? That sounds dramatic. It is. 00:10:11.440 --> 00:10:14.600 As you add more pixels or more words, the number 00:10:14.600 --> 00:10:17.399 of possible combinations explodes exponentially. 00:10:17.419 --> 00:10:19.700 Right. If you are just trying to categorize a 00:10:19.700 --> 00:10:22.960 signal, mapping the entire data generating process 00:10:22.960 --> 00:10:25.519 is monumental overkill. It's like building an 00:10:25.519 --> 00:10:28.299 entire hyper -realistic global weather simulation 00:10:28.299 --> 00:10:30.460 just to decide if you need to wear a rain jacket 00:10:30.460 --> 00:10:32.559 to the grocery store. A quick look out the window 00:10:32.559 --> 00:10:35.759 does the job just fine. Exactly. But if your 00:10:35.759 --> 00:10:38.899 goal isn't just to spot a rain cloud, if your 00:10:38.899 --> 00:10:41.059 goal is to create a synthetic weather pattern 00:10:41.059 --> 00:10:43.580 that has never existed before, then you have 00:10:43.580 --> 00:10:45.779 no choice. You need the simulation. You need 00:10:45.779 --> 00:10:48.860 that massive, overly complex simulation. Which 00:10:48.860 --> 00:10:51.120 brings us to the most mind -bending part of this 00:10:51.120 --> 00:10:54.360 whole deep dive. the bridge between the theory 00:10:54.360 --> 00:10:57.899 and the reality. Yes. If mapping the full distribution 00:10:57.899 --> 00:11:00.879 is that incredibly resource -intensive, if the 00:11:00.879 --> 00:11:03.279 combinations explode exponentially with just 00:11:03.279 --> 00:11:06.240 a few variables, how in the world did we go from 00:11:06.240 --> 00:11:09.259 analyzing a tiny four -point probability table 00:11:09.259 --> 00:11:12.799 to having AI that can instantly write a cohesive 00:11:12.799 --> 00:11:16.690 college -level essay? Your answer is scale. mind 00:11:16.690 --> 00:11:19.210 -boggling, unprecedented computational scale. 00:11:19.269 --> 00:11:21.789 Okay. And to understand how that scale was achieved, 00:11:21.950 --> 00:11:24.210 we actually have to travel back to 1948. Wait, 00:11:24.789 --> 00:11:26.690 1948, back to the era of room -sized computers 00:11:26.690 --> 00:11:28.909 that could barely do arithmetic. Exactly. The 00:11:28.909 --> 00:11:31.129 source features a brilliant historical example 00:11:31.129 --> 00:11:33.230 from Claude Shannon. The father of information 00:11:33.230 --> 00:11:36.090 theory. That's the one. Long before modern neural 00:11:36.090 --> 00:11:38.450 networks existed, Shannon wanted to see if a 00:11:38.450 --> 00:11:40.409 mathematical model could generate English text. 00:11:40.600 --> 00:11:42.919 Wait, how on earth was he generating text in 00:11:42.919 --> 00:11:46.440 1948? He was acting as a manual analog generative 00:11:46.440 --> 00:11:51.139 model. Like, by hand? By hand. He used probability 00:11:51.139 --> 00:11:54.360 tables. He essentially took a book, picked a 00:11:54.360 --> 00:11:57.269 word, and then manually calculated the frequency 00:11:57.269 --> 00:11:59.549 of what word was most likely to come next. That 00:11:59.549 --> 00:12:02.529 sounds exhausting. Oh, it was. He was mapping 00:12:02.529 --> 00:12:05.509 the joint probability of word pairs in the English 00:12:05.509 --> 00:12:07.490 language. So he was mapping out the landscape 00:12:07.490 --> 00:12:10.450 of English just two words at a time. Yes. And 00:12:10.450 --> 00:12:12.970 using that incredibly basic table of probabilities, 00:12:13.490 --> 00:12:17.049 he mathematically generated a brand new synthetic 00:12:17.049 --> 00:12:19.549 sentence. And the result was, well, I teased 00:12:19.549 --> 00:12:22.730 it earlier. The result was the phrase, representing 00:12:22.730 --> 00:12:26.299 and speedily, isn't good. Which is obviously 00:12:26.299 --> 00:12:28.580 absolute gibberish. Complete nonsense. Yeah. 00:12:28.740 --> 00:12:30.440 But here's where it gets really interesting though. 00:12:30.759 --> 00:12:33.240 It is gibberish, but Shannon wasn't discouraged 00:12:33.240 --> 00:12:35.559 at all. Not even a little. He noted something 00:12:35.559 --> 00:12:38.679 profound about that nonsense phrase. He recognized 00:12:38.679 --> 00:12:40.779 that this gibberish would get closer and closer 00:12:40.779 --> 00:12:43.700 to proper coherent English if you just expand 00:12:43.700 --> 00:12:46.559 at the table. Like, if you moved from calculating 00:12:46.559 --> 00:12:49.220 the probability of word pairs to word triplets 00:12:49.220 --> 00:12:52.220 to groups of four to entire sentences... The 00:12:52.220 --> 00:12:54.820 underlying math was perfectly sound. The theory 00:12:54.820 --> 00:12:57.340 was proven. He just needed more power. Right. 00:12:57.840 --> 00:13:00.139 The only thing Shannon lacked was a computational 00:13:00.139 --> 00:13:03.019 canvas big enough to hold the probability distribution 00:13:03.019 --> 00:13:06.500 of the entire English language. Doing that by 00:13:06.500 --> 00:13:09.309 hand would take millions of lifetimes. Because 00:13:09.309 --> 00:13:11.350 of that curse of dimensionality, as soon as you 00:13:11.350 --> 00:13:13.629 move from word pairs to word triplets, the number 00:13:13.629 --> 00:13:16.169 of possible combinations skyrocket. It goes off 00:13:16.169 --> 00:13:18.990 the charts. Yeah. And that is exactly what modern 00:13:18.990 --> 00:13:21.409 technology solved. OK. The source introduces 00:13:21.409 --> 00:13:24.090 a critical term here, deep generative models, 00:13:24.409 --> 00:13:27.850 or DGMs. So what makes a generative model deep? 00:13:28.330 --> 00:13:30.789 DGMs are what happen when you take Claude Shannon's 00:13:30.789 --> 00:13:34.350 basic 1948 word pair idea, but you fuse it with 00:13:34.350 --> 00:13:36.470 deep neural networks. Oh, wow. And then you. 00:13:36.590 --> 00:13:39.149 exponentially scale up the training data using 00:13:39.149 --> 00:13:42.110 supercomputers. The neural networks act as an 00:13:42.110 --> 00:13:44.049 incredibly efficient way to compress and store 00:13:44.049 --> 00:13:47.190 that massive landscape of probabilities. The 00:13:47.190 --> 00:13:50.269 leap in scale is just staggering. I mean, the 00:13:50.269 --> 00:13:52.490 article lists some of the major models dominating 00:13:52.490 --> 00:13:54.710 the field today. Yeah, they're huge. You have 00:13:54.710 --> 00:13:57.710 autoregressive neural language models like GPT 00:13:57.710 --> 00:14:01.190 -3, which boast billions of parameters. Billions. 00:14:01.330 --> 00:14:04.149 Billions of mathematical dials tracking the complex 00:14:04.149 --> 00:14:06.429 dependencies of words over massive stretches 00:14:06.429 --> 00:14:08.870 of text. And it goes far beyond text, too. Right. 00:14:08.889 --> 00:14:11.450 We have image generators, architectures like 00:14:11.450 --> 00:14:15.250 Bigin and VQVAE. Right, the image ones. These. 00:14:15.659 --> 00:14:18.159 have hundreds of millions of parameters mapping 00:14:18.159 --> 00:14:20.759 the massive joint probabilities of individual 00:14:20.759 --> 00:14:23.779 pixels and color gradients to generate synthetic 00:14:23.779 --> 00:14:26.100 photographs that look entirely real. They even 00:14:26.100 --> 00:14:28.779 mention musical audio generators. Yeah, like 00:14:28.779 --> 00:14:31.639 jukebox. Yeah, jukebox. The source notes it contains 00:14:31.639 --> 00:14:34.120 billions of parameters to model the distribution 00:14:34.120 --> 00:14:37.059 of raw sound waves. It's incredible. Think about 00:14:37.059 --> 00:14:39.179 that timeline for a second. Shannon's sitting 00:14:39.179 --> 00:14:41.580 with a book, manually counting word pairs on 00:14:41.580 --> 00:14:44.620 a piece of paper in 1948. Doing the math. That 00:14:44.620 --> 00:14:47.929 is mathematically the direct ancestor to GPT 00:14:47.929 --> 00:14:51.549 -3 and jukebox. The core concept mapping the 00:14:51.549 --> 00:14:54.750 distribution to draw synthetic samples, it did 00:14:54.750 --> 00:14:57.200 not change. Not fundamentally, no. We just gave 00:14:57.200 --> 00:15:00.559 our artist a supercomputer canvas with billions 00:15:00.559 --> 00:15:02.799 of dimensions instead of just two. It really 00:15:02.799 --> 00:15:06.519 highlights that the magic of modern AI is, at 00:15:06.519 --> 00:15:10.039 its core, just the brute force application of 00:15:10.039 --> 00:15:13.519 immense statistical probability. Right. But as 00:15:13.519 --> 00:15:15.840 comprehensive as these traditional models are, 00:15:16.220 --> 00:15:18.139 the source actually introduces a fascinating 00:15:18.139 --> 00:15:20.559 rebel in the generative family. Oh, yeah, the 00:15:20.559 --> 00:15:22.659 rule breakers. A completely different architecture 00:15:22.659 --> 00:15:25.440 that doesn't play by these traditional rules 00:15:25.440 --> 00:15:27.740 or problems. Yes, because we've spent this whole 00:15:27.740 --> 00:15:29.720 time establishing that generative models are 00:15:29.720 --> 00:15:32.980 all about meticulously mapping probability distributions 00:15:32.980 --> 00:15:35.740 to create synthetic data. Right, the whole landscape. 00:15:35.940 --> 00:15:38.279 But the text explicitly points out that the term 00:15:38.279 --> 00:15:41.139 generative model has evolved. It has. It's now 00:15:41.139 --> 00:15:43.700 also used for models that generate outputs without 00:15:43.700 --> 00:15:46.779 a clear, strict relationship to probability distributions 00:15:46.779 --> 00:15:49.840 over potential inputs. Which is a huge structural 00:15:49.840 --> 00:15:51.279 departure from everything we've talked about. 00:15:51.299 --> 00:15:53.899 It really is. The prime example of this new rebel 00:15:53.899 --> 00:15:56.799 class is the Jan. the generative adversarial 00:15:56.799 --> 00:15:59.509 network. Okay, Jans are fascinating. Because 00:15:59.509 --> 00:16:02.309 unlike traditional statistical models, unlike 00:16:02.309 --> 00:16:04.750 naive Bayes or the autoregressive models we just 00:16:04.750 --> 00:16:08.590 talked about, JANs are not classifiers mapping 00:16:08.590 --> 00:16:11.090 a landscape. No, not at all. They are judged 00:16:11.090 --> 00:16:13.389 by a completely different metric. Which is? They 00:16:13.389 --> 00:16:15.850 are judged primarily by the similarity of their 00:16:15.850 --> 00:16:18.509 outputs to potential inputs. So what does this 00:16:18.509 --> 00:16:21.809 all mean for our analogy? Let's see. If traditional 00:16:21.809 --> 00:16:24.029 models, the ones calculating joint probabilities 00:16:24.029 --> 00:16:27.980 and using Bayes' rule, are like incredibly diligent 00:16:27.980 --> 00:16:30.899 math students carefully plotting probabilities 00:16:30.899 --> 00:16:34.000 on a giant graph to paint a mathematically perfect 00:16:34.000 --> 00:16:38.080 picture. A Jan is a counterfeiter. Oh, I like 00:16:38.080 --> 00:16:40.299 that. Break down the counterfeiter process. How 00:16:40.299 --> 00:16:42.419 does it actually work? Well, the counterfeiter 00:16:42.419 --> 00:16:45.659 isn't doing complex calculus to map the joint 00:16:45.659 --> 00:16:47.700 probability of the entire art world. Right. They 00:16:47.700 --> 00:16:50.169 don't care about the theory. Exactly. The counterfeiter 00:16:50.169 --> 00:16:52.590 just wants to make a quick buck. So they paint 00:16:52.590 --> 00:16:54.629 a fake painting and they hand it to an art critic. 00:16:55.029 --> 00:16:57.269 The art critic looks at it and says, this is 00:16:57.269 --> 00:16:59.490 obviously fake. The brushstrokes are all wrong. 00:17:00.070 --> 00:17:01.789 So the critic is acting as the discriminator. 00:17:02.250 --> 00:17:04.190 Exactly. So the counterfeiter goes back to the 00:17:04.190 --> 00:17:06.849 studio, tweaks the brushstrokes, and tries again. 00:17:07.450 --> 00:17:10.680 Still fake. says the critic, the colors are off. 00:17:10.980 --> 00:17:12.980 Right, there's a loop. This happens over and 00:17:12.980 --> 00:17:15.859 over thousands of times. The counterfeiter isn't 00:17:15.859 --> 00:17:18.220 learning the grand mathematical truth of the 00:17:18.220 --> 00:17:21.019 universe. They are simply learning exactly what 00:17:21.019 --> 00:17:24.660 it takes to fool that specific critic. It's an 00:17:24.660 --> 00:17:28.119 adversarial process. Create a fake, get caught, 00:17:28.519 --> 00:17:31.339 improve the fake, try again. This raises an important 00:17:31.339 --> 00:17:34.160 question, really, about how we even define success 00:17:34.160 --> 00:17:37.180 in machine learning now. How so? Well, the source 00:17:37.180 --> 00:17:39.519 material is showing us a massive paradigm shift 00:17:39.519 --> 00:17:42.200 in the field. For decades, from Claude Shannon 00:17:42.200 --> 00:17:43.920 all the way up through early neural networks, 00:17:44.440 --> 00:17:47.099 we judged models purely on mathematical likelihood. 00:17:47.359 --> 00:17:49.359 Did they get the math right? Did the generated 00:17:49.359 --> 00:17:51.759 data correctly align with the calculated probability 00:17:51.759 --> 00:17:54.700 distribution? It was a strict math test. Exactly. 00:17:55.099 --> 00:17:58.819 But with models like JANs, we've moved to a fundamentally 00:17:58.819 --> 00:18:01.220 different standard. Right. We are judging them 00:18:01.220 --> 00:18:03.680 on the subjective fidelity of their generated 00:18:03.680 --> 00:18:07.279 synthetic reality. Subjective fidelity. Does 00:18:07.279 --> 00:18:10.460 this output look real to us? Does it successfully 00:18:10.460 --> 00:18:13.329 fool the discriminator? It's a massive shift 00:18:13.329 --> 00:18:16.369 from mathematical precision to perceptual deception. 00:18:16.710 --> 00:18:19.569 It is genuinely wild to think about. We've spent 00:18:19.569 --> 00:18:22.250 this time unraveling the math, going from a tiny 00:18:22.250 --> 00:18:24.210 four -point table where a machine learns the 00:18:24.210 --> 00:18:27.890 25 % probability of a number, all the way to 00:18:27.890 --> 00:18:30.970 billion -parameter deep generative models and 00:18:30.970 --> 00:18:34.009 advertorial counterfeiters. And yet the end result 00:18:34.009 --> 00:18:36.430 of all that cold, hard math is something that 00:18:36.430 --> 00:18:39.089 actively tricks our inherently human senses into 00:18:39.089 --> 00:18:41.589 believing it has a soul. It is the ultimate illusion. 00:18:41.799 --> 00:18:44.440 powered entirely by statistics. So let's bring 00:18:44.440 --> 00:18:46.299 this all together. Learner, next time you use 00:18:46.299 --> 00:18:48.660 a tool to generate a glossy, high -resolution 00:18:48.660 --> 00:18:51.900 image for a presentation, or you ask an AI to 00:18:51.900 --> 00:18:54.339 write a polite, professional email to your boss. 00:18:54.440 --> 00:18:56.359 Or you listen to a synthetic voice reading a 00:18:56.359 --> 00:18:59.359 text. Exactly. I want you to remember what is 00:18:59.359 --> 00:19:02.039 actually happening under the hood. You aren't 00:19:02.039 --> 00:19:05.000 talking to a sentient being. You aren't interacting 00:19:05.000 --> 00:19:07.660 with a ghost in the machine. No, you are interacting 00:19:07.660 --> 00:19:10.240 with a deep generative model. You are interacting 00:19:10.240 --> 00:19:13.480 with a staggeringly massive probability distribution. 00:19:13.880 --> 00:19:16.359 It is exactly like that simple four -point table 00:19:16.359 --> 00:19:19.019 we discussed, where the machine knows the exact 00:19:19.019 --> 00:19:21.700 mathematical weight of every combination. But 00:19:21.700 --> 00:19:24.599 with billions of parameters. Billions. It is 00:19:24.599 --> 00:19:27.859 an engine that is rapidly silently drawing random 00:19:27.859 --> 00:19:30.579 mathematical instances that statistically resemble 00:19:30.579 --> 00:19:33.019 the data it observed during its training. Which 00:19:33.019 --> 00:19:36.059 leaves us with a genuinely fascinating new problem 00:19:36.059 --> 00:19:39.200 to ponder as we wrap up. Okay, what's that? We've 00:19:39.200 --> 00:19:41.160 spent this entire deep dive talking about how 00:19:41.160 --> 00:19:44.000 these models meticulously map the probability 00:19:44.000 --> 00:19:47.359 of human output. human paintings, human text, 00:19:47.759 --> 00:19:49.940 human music. Right, they're trained on what we 00:19:49.940 --> 00:19:52.339 create. But if a mathematical model can perfectly 00:19:52.339 --> 00:19:54.880 replicate the joint probability of human creativity, 00:19:55.019 --> 00:19:57.799 without possessing any humanity itself, does 00:19:57.799 --> 00:20:00.279 that mean human creativity is just a highly complex 00:20:00.279 --> 00:20:04.799 probability distribution? biological generative 00:20:04.799 --> 00:20:07.259 models mapping the data we've seen in our lives. 00:20:07.539 --> 00:20:10.660 That is heavy. Is human inspiration just a massive 00:20:10.660 --> 00:20:13.099 statistical calculation under the hood? Something 00:20:13.099 --> 00:20:15.359 for you to chew on the next time you feel a spark 00:20:15.359 --> 00:20:18.180 of inspiration. That is brilliant. Well thank 00:20:18.180 --> 00:20:20.759 you for joining us on this custom deep dive learner. 00:20:21.259 --> 00:20:23.660 We've gone from the rigid world of probability 00:20:23.660 --> 00:20:26.440 to the fluid illusion of creativity all through 00:20:26.440 --> 00:20:28.839 the lens of the generative model. Until next 00:20:28.839 --> 00:20:30.960 time, keep questioning the numbers behind the 00:20:30.960 --> 00:20:31.259 magic.