WEBVTT 00:00:00.000 --> 00:00:01.980 Every single time you pick up your phone and 00:00:01.980 --> 00:00:06.040 it unlocks just by scanning your face, or when 00:00:06.040 --> 00:00:07.839 you're dictating a text message while navigating 00:00:07.839 --> 00:00:09.880 traffic and the words just seamlessly appear 00:00:09.880 --> 00:00:12.980 on the screen, there's an entirely hidden, just 00:00:12.980 --> 00:00:15.919 immensely complex mathematical architecture making 00:00:15.919 --> 00:00:18.379 it possible. Right, yeah. And you interact with 00:00:18.379 --> 00:00:22.120 it constantly, like every day. Yeah. But if someone 00:00:22.120 --> 00:00:24.239 asked you to physically write out the proof that 00:00:24.239 --> 00:00:27.559 guarantees your phone won't just suddenly forget 00:00:27.559 --> 00:00:29.460 what a human face looks like tomorrow, Could 00:00:29.460 --> 00:00:31.399 you do it? Yeah. I mean, it really is the silent 00:00:31.399 --> 00:00:33.320 engine of the modern world. And the fascinating 00:00:33.320 --> 00:00:35.359 part is that we often just treat it like magic, 00:00:35.460 --> 00:00:38.259 right? We just sort of vaguely gesture at the 00:00:38.259 --> 00:00:40.939 concept of algorithms. But it's not magic at 00:00:40.939 --> 00:00:44.320 all. It is strictly rigorously defined mathematics. 00:00:44.579 --> 00:00:47.380 Yeah. And so today, we're doing a deep dive into 00:00:47.380 --> 00:00:50.420 the absolute bedrock of that engine. We are looking 00:00:50.420 --> 00:00:53.420 at a really comprehensive Wikipedia breakdown 00:00:53.420 --> 00:00:56.619 of statistical learning theory. Which is a fantastic 00:00:56.619 --> 00:01:00.280 source for this. Totally. And our mission today 00:01:00.280 --> 00:01:03.899 is to move past the introductory stuff. Like, 00:01:03.920 --> 00:01:05.980 if you're listening to this, you probably already 00:01:05.980 --> 00:01:07.859 know the basics of machine learning. You know 00:01:07.859 --> 00:01:10.040 what a neural network is. You know what training 00:01:10.040 --> 00:01:12.879 data is. Right, the 101 level stuff. Yeah. So 00:01:12.879 --> 00:01:15.900 we aren't going to spend time talking about flashcards 00:01:15.900 --> 00:01:18.159 or teaching a computer to tell a cat from a dog. 00:01:18.739 --> 00:01:21.739 Instead, we are going to map out the exact framework, 00:01:21.959 --> 00:01:24.439 which is borrowed from statistics and functional 00:01:24.439 --> 00:01:27.900 analysis that physically proves a machine can 00:01:27.900 --> 00:01:30.599 understand data and predict the future reliably. 00:01:31.579 --> 00:01:33.379 You know, while we are definitely going to get 00:01:33.379 --> 00:01:36.480 into some heavy graduate level math today, I 00:01:36.480 --> 00:01:38.760 really want to assure you that this deep dive 00:01:38.760 --> 00:01:40.739 will give you the foundational blueprint for 00:01:40.739 --> 00:01:43.140 how artificial intelligence actually constructs 00:01:43.140 --> 00:01:46.400 its reality. We're looking at the literal mathematical 00:01:46.400 --> 00:01:49.120 mechanics of machine comprehension is how they 00:01:49.120 --> 00:01:51.640 think for lack of a better word. OK, let's unpack 00:01:51.640 --> 00:01:53.819 this because we have to start by defining what 00:01:53.819 --> 00:01:56.019 learning actually means in this really strict 00:01:56.019 --> 00:01:59.319 mathematical sense, right? Like the core goals 00:01:59.319 --> 00:02:02.400 are understanding and prediction. specifically 00:02:02.400 --> 00:02:05.620 within the realm of supervised learning, but 00:02:05.620 --> 00:02:07.840 we need to look at how that is formally mapped 00:02:07.840 --> 00:02:11.139 out. Right, so let's establish the mathematical 00:02:11.139 --> 00:02:13.840 landscape first. We start with vector spaces. 00:02:14.060 --> 00:02:16.879 So we have a vector space called X. And this 00:02:16.879 --> 00:02:20.139 represents all possible inputs. So if we are 00:02:20.139 --> 00:02:22.599 talking about facial recognition, X isn't just 00:02:22.599 --> 00:02:24.560 a collection of, like, pictures. Right, it's 00:02:24.560 --> 00:02:27.219 not a photo album. No, it's a massive multi -dimensional 00:02:27.219 --> 00:02:30.240 vector space where every single element represents 00:02:30.240 --> 00:02:33.080 an individual pixel's value. Wow. And then we 00:02:33.080 --> 00:02:35.419 have a second vector space called Y, which represents 00:02:35.419 --> 00:02:37.759 all the possible outputs. Right. And here is 00:02:37.759 --> 00:02:39.819 where we get to the core assumption of the entire 00:02:39.819 --> 00:02:42.120 theory, which I found super interesting. in the 00:02:42.120 --> 00:02:45.439 source. There's a true underlying rule out there 00:02:45.439 --> 00:02:48.060 in the universe dictating how these inputs and 00:02:48.060 --> 00:02:50.400 outputs relate, but it is completely unknown 00:02:50.400 --> 00:02:53.280 to us. Yes, the ground truth. Right, and mathematically 00:02:53.280 --> 00:02:56.060 it's defined as an unknown probability distribution 00:02:56.060 --> 00:02:59.500 over the joint space of X and Y, and it's usually 00:02:59.500 --> 00:03:02.590 denoted as Z. where z is the Cartesian product 00:03:02.590 --> 00:03:06.210 of x and y. And that is a really vital distinction 00:03:06.210 --> 00:03:08.590 to make. We aren't just multiplying numbers together 00:03:08.590 --> 00:03:11.370 here. The Cartesian product means we are looking 00:03:11.370 --> 00:03:14.569 at every single possible input. paired with every 00:03:14.569 --> 00:03:17.610 single possible output. Oh, wow. Yeah. So there 00:03:17.610 --> 00:03:20.150 is a true distribution in that massive joint 00:03:20.150 --> 00:03:23.789 space. And the entire learning problem basically 00:03:23.789 --> 00:03:27.830 boils down to the machine trying to infer a functional 00:03:27.830 --> 00:03:30.710 relationship that approximates that unknown truth. 00:03:31.129 --> 00:03:32.729 Right. And the nature of that function actually 00:03:32.729 --> 00:03:35.689 depends on what the vector space Y looks like. 00:03:36.189 --> 00:03:39.270 Like if Y is a continuous range of values, we 00:03:39.270 --> 00:03:42.110 are doing regression. Like think about discovering 00:03:42.110 --> 00:03:44.819 Ohm's law from scratch, right? You have voltage 00:03:44.819 --> 00:03:47.199 as your input vector, current as your output 00:03:47.199 --> 00:03:49.180 vector, and the machine is just trying to find 00:03:49.180 --> 00:03:51.139 the continuous functional relationship, which 00:03:51.139 --> 00:03:54.800 ends up being V equals I times R. Exactly. But 00:03:54.800 --> 00:03:57.819 if Y is a discrete set of labels, like specific 00:03:57.819 --> 00:04:00.560 names of people for that facial recognition example, 00:04:01.080 --> 00:04:03.039 then we are doing classification. Precisely. 00:04:03.340 --> 00:04:05.460 What's fascinating here is that to the machine, 00:04:05.599 --> 00:04:08.080 it doesn't actually see a face. It doesn't know 00:04:08.080 --> 00:04:11.060 what a nose or an eye is. It just sees that massive 00:04:11.060 --> 00:04:13.340 multi -dimensional vector of pixels we talked 00:04:13.340 --> 00:04:15.979 about. The learning is entirely about finding 00:04:15.979 --> 00:04:18.560 the mathematical relationship between the front 00:04:18.560 --> 00:04:21.439 of the metaphorical flash card, the pixels, and 00:04:21.439 --> 00:04:24.459 the back, the name. So, okay, the machine knows 00:04:24.459 --> 00:04:26.360 its input, and it knows what kind of output it 00:04:26.360 --> 00:04:30.079 needs, but how does it actually, like, hunt for 00:04:30.079 --> 00:04:31.899 the rule that connects them? Well, we have to 00:04:31.899 --> 00:04:34.060 give the algorithm a sandbox to play in, basically. 00:04:34.360 --> 00:04:36.459 And in the math, this is called the hypothesis 00:04:36.459 --> 00:04:40.230 space. usually denoted by a script H. Okay, hypothesis 00:04:40.230 --> 00:04:42.970 space. Yeah, the hypothesis space is the specific 00:04:42.970 --> 00:04:45.470 restricted space of functions that the algorithm 00:04:45.470 --> 00:04:47.889 is actually allowed to search through. But, like, 00:04:47.889 --> 00:04:50.790 as it's searching through that space, it needs 00:04:50.790 --> 00:04:53.069 feedback, right? It needs a way to mathematically 00:04:53.069 --> 00:04:56.629 measure how spectacularly it's failing. It definitely 00:04:56.629 --> 00:04:58.550 does. And that is where the loss function comes 00:04:58.550 --> 00:05:02.490 in, written as v of f of x comma y. It calculates 00:05:02.490 --> 00:05:04.430 the exact difference between the predicted value 00:05:04.430 --> 00:05:07.230 and the true value. Right. And there is a strict 00:05:07.230 --> 00:05:09.649 non -negotiable rule here for these loss functions 00:05:09.649 --> 00:05:12.009 if we want the algorithm to actually work in 00:05:12.009 --> 00:05:15.029 practice. The loss function must be convex. Okay, 00:05:15.129 --> 00:05:17.629 here's where it gets really interesting. Why 00:05:17.629 --> 00:05:21.410 convexity? Because I know in geometry, a convex 00:05:21.410 --> 00:05:23.490 shape doesn't have any inward dents. But what 00:05:23.490 --> 00:05:26.129 does that actually mean for a machine searching 00:05:26.129 --> 00:05:29.720 for a function? Right. So think of the loss function 00:05:29.720 --> 00:05:32.319 as a landscape that the algorithm is trying to 00:05:32.319 --> 00:05:35.639 navigate to find the lowest possible point, the 00:05:35.639 --> 00:05:37.860 absolute minimum error. OK, I'm picturing it. 00:05:38.019 --> 00:05:40.819 If the function is convex, that landscape is 00:05:40.819 --> 00:05:43.579 shaped exactly like a perfectly smooth round 00:05:43.579 --> 00:05:46.980 bowl. If you drop a marble into that bowl from 00:05:46.980 --> 00:05:49.579 literally anywhere, gravity is just going to 00:05:49.579 --> 00:05:51.819 pull it down to the absolute bottom. Right. Because 00:05:51.819 --> 00:05:53.959 there's nowhere else for it to go. Exactly. There 00:05:53.959 --> 00:05:56.240 is only one bottom. That is the global minimum. 00:05:56.379 --> 00:05:58.819 Ah. If it wasn't convex, the landscape would 00:05:58.819 --> 00:06:00.800 be like, I don't know, a bumpy mountain range. 00:06:01.439 --> 00:06:03.620 The marble might roll down a hill and get stuck 00:06:03.620 --> 00:06:06.040 in some random crater halfway up the mountain. 00:06:06.740 --> 00:06:08.180 And it thinks it's at the bottom because every 00:06:08.180 --> 00:06:10.360 direction around it goes up. But it's actually 00:06:10.360 --> 00:06:13.100 just stuck in a local minimum, nowhere near the 00:06:13.100 --> 00:06:16.350 true best answer. That's exactly it. What's fascinating 00:06:16.350 --> 00:06:19.410 here is that convexity mathematically guarantees 00:06:19.410 --> 00:06:21.350 that the algorithm won't get trapped in those 00:06:21.350 --> 00:06:24.610 craters. It ensures that gradient descent, which 00:06:24.610 --> 00:06:27.089 is the method the machine uses to update its 00:06:27.089 --> 00:06:30.529 guesses, will reliably converge on the single 00:06:30.529 --> 00:06:33.819 best solution in that hypothesis space. OK, I 00:06:33.819 --> 00:06:36.519 want to jump in with an analogy here. The hypothesis 00:06:36.519 --> 00:06:38.259 space is basically like standing in front of 00:06:38.259 --> 00:06:41.019 a massive wardrobe trying to find the perfect 00:06:41.019 --> 00:06:42.819 outfit for the day. Right. I like where this 00:06:42.819 --> 00:06:45.480 is going. And the loss function is the mirror 00:06:45.480 --> 00:06:47.980 brutally telling you exactly how far off you 00:06:47.980 --> 00:06:51.019 are from looking good. Yes. And to build on that 00:06:51.019 --> 00:06:54.120 mirror analogy, different problems require entirely 00:06:54.120 --> 00:06:56.680 different types of mirrors. The shape of that 00:06:56.680 --> 00:06:59.480 bowl changes. Right. For regression, the standard 00:06:59.480 --> 00:07:02.220 is the L2 norm, or square loss. And I really 00:07:02.220 --> 00:07:04.480 love the mechanics of this, because it doesn't 00:07:04.480 --> 00:07:06.480 just measure the distance between the guess and 00:07:06.480 --> 00:07:08.980 the truth. It actually squares that distance. 00:07:09.800 --> 00:07:11.639 So instead of a mirror just telling you that 00:07:11.639 --> 00:07:14.459 you made a mistake, the L2 norm is like a rubber 00:07:14.459 --> 00:07:16.579 band attaching your prediction to the true answer. 00:07:16.680 --> 00:07:18.879 That is a highly accurate way to visualize it, 00:07:19.019 --> 00:07:21.579 yeah. Right, because if your prediction is just 00:07:21.579 --> 00:07:24.259 slightly off, the rubber band is gently taught. 00:07:24.379 --> 00:07:27.180 It's a small penalty. Right. but if your prediction 00:07:27.180 --> 00:07:30.259 is wildly wrong that rubber band is stretched 00:07:30.259 --> 00:07:33.689 to its absolute limit and the snapback is exponentially 00:07:33.689 --> 00:07:36.709 more brutal. Exactly. Squaring the error means 00:07:36.709 --> 00:07:39.069 the algorithm is mathematically forced to care 00:07:39.069 --> 00:07:41.990 immensely about massive outliers. It has to adjust 00:07:41.990 --> 00:07:44.269 its function to bring those extreme errors down 00:07:44.269 --> 00:07:46.730 fast. That makes total sense. Or, alternatively, 00:07:46.829 --> 00:07:49.110 you could use the L1 norm, which is the absolute 00:07:49.110 --> 00:07:51.250 value loss. That one doesn't square the distance, 00:07:51.350 --> 00:07:53.709 it just treats all errors proportionally. So 00:07:53.709 --> 00:07:55.689 the rubber band's tension increases at a constant 00:07:55.689 --> 00:07:59.649 rate. OK. But for classification, you can't really 00:07:59.649 --> 00:08:02.009 measure distance like that, right? Like you can't 00:08:02.009 --> 00:08:03.829 be three pounds away from being a picture of 00:08:03.829 --> 00:08:06.029 a cat. You either are a cat or you aren't. Right. 00:08:06.069 --> 00:08:08.949 It's discrete. So for classification, the math 00:08:08.949 --> 00:08:11.649 shifts to the zero one indicator function. It 00:08:11.649 --> 00:08:14.209 uses the heavy sidestep function for binary choices, 00:08:14.209 --> 00:08:16.790 and it's an incredibly harsh mirror. Like pass 00:08:16.790 --> 00:08:19.370 fail. Exactly. If the prediction matches the 00:08:19.370 --> 00:08:21.889 output, the loss is zero. If it doesn't, the 00:08:21.889 --> 00:08:24.350 loss is one. There is zero partial credit. OK, 00:08:24.430 --> 00:08:27.730 so. We have our hypothesis space, our wardrobe, 00:08:28.269 --> 00:08:30.649 and we have our loss function, our mirror acting 00:08:30.649 --> 00:08:33.610 as the judge. The logical next step seems pretty 00:08:33.610 --> 00:08:36.509 obvious. Just tell the machine to minimize that 00:08:36.509 --> 00:08:39.759 loss. Make the error zero. But pursuing that 00:08:39.759 --> 00:08:42.799 exact instinct leads to what the source describes 00:08:42.799 --> 00:08:45.899 as the most dangerous trap in all of statistical 00:08:45.899 --> 00:08:48.399 learning. It really does. And to understand the 00:08:48.399 --> 00:08:51.019 trap, we have to split the concept of risk into 00:08:51.019 --> 00:08:54.580 two distinct ideas. We have expected risk and 00:08:54.580 --> 00:08:57.320 empirical risk. Lay those out for us. So expected 00:08:57.320 --> 00:08:59.879 risk is the holy grail. It is the true measure 00:08:59.879 --> 00:09:03.559 of error over that entire massive unknown probability 00:09:03.559 --> 00:09:05.360 distribution Z that we talked about earlier. 00:09:05.559 --> 00:09:07.659 It's basically how the model will perform in 00:09:07.659 --> 00:09:10.269 the real world forever. But we cannot calculate 00:09:10.269 --> 00:09:12.409 expected risk. It's physically impossible because 00:09:12.409 --> 00:09:14.190 we don't have access to the entire universe of 00:09:14.190 --> 00:09:17.009 data. Exactly. We only have our tiny, finite 00:09:17.009 --> 00:09:20.029 slice of reality, our end samples in the training 00:09:20.029 --> 00:09:22.090 set. So we have to use a proxy measure instead, 00:09:22.250 --> 00:09:23.809 which is the empirical risk. And the empirical 00:09:23.809 --> 00:09:26.289 risk is just the average of the loss function 00:09:26.289 --> 00:09:29.169 calculated solely over those end training samples. 00:09:29.529 --> 00:09:31.889 Which naturally leads algorithms to a process 00:09:31.889 --> 00:09:36.019 called empirical risk minimization, or ERM. The 00:09:36.019 --> 00:09:39.039 algorithm simply scours the hypothesis space 00:09:39.039 --> 00:09:41.980 and chooses the function that brings the empirical 00:09:41.980 --> 00:09:45.100 risk the errors on the training data as close 00:09:45.100 --> 00:09:47.639 to zero as humanly possible. Okay, wait, hold 00:09:47.639 --> 00:09:49.720 on. This is where I'm going to push back a bit 00:09:49.720 --> 00:09:53.139 because intuitively this sounds exactly like 00:09:53.139 --> 00:09:55.580 what we want. It does sound like it. Right. If 00:09:55.580 --> 00:09:58.139 I am a teacher and I give my student a practice 00:09:58.139 --> 00:10:01.539 test, I want them to get a hundred percent. Why 00:10:01.539 --> 00:10:03.980 is achieving an empirical risk of zero considered 00:10:03.980 --> 00:10:07.240 a bad thing here? Because of how a vastly complex 00:10:07.240 --> 00:10:10.059 function actually achieves that zero, let's stick 00:10:10.059 --> 00:10:12.600 with your student analogy. If a student genuinely 00:10:12.600 --> 00:10:14.860 understands the underlying concepts, the actual 00:10:14.860 --> 00:10:17.220 ground truth, they will do well on the practice 00:10:17.220 --> 00:10:19.600 test. Right. But what if they just memorize the 00:10:19.600 --> 00:10:22.340 exact sequence of multiple choice answers? A, 00:10:22.379 --> 00:10:24.879 C, B, D, A. Oh, I see. They will get a 100 % 00:10:24.879 --> 00:10:26.740 on that practice test. Their empirical risk is 00:10:26.740 --> 00:10:29.240 essentially zero, but they didn't learn the subject, 00:10:29.279 --> 00:10:31.200 they just learned the test. And when they sit 00:10:31.200 --> 00:10:33.720 down for the final exam, which has totally different 00:10:33.720 --> 00:10:37.639 questions, completely bomb it. Yes. And in statistical 00:10:37.639 --> 00:10:40.139 learning theory, this is the crisis of overfitting. 00:10:40.720 --> 00:10:43.240 The algorithm has found a function so wildly 00:10:43.240 --> 00:10:46.000 convoluted that it contorts itself to perfectly 00:10:46.000 --> 00:10:48.779 hit every single data point in the training set. 00:10:49.240 --> 00:10:51.759 It has memorized the random noise and the weird 00:10:51.759 --> 00:10:54.440 anomalies of that specific data set, rather than 00:10:54.440 --> 00:10:57.759 extracting the true underlying signal. Wow. And 00:10:57.759 --> 00:11:00.259 mathematically, overfitting is described as an 00:11:00.259 --> 00:11:03.100 unstable solution. And instability means that 00:11:03.100 --> 00:11:05.740 if I went into the training data and change the 00:11:05.740 --> 00:11:08.220 value of just one single pixel in one image, 00:11:08.600 --> 00:11:10.840 the algorithm would spit out a completely radically 00:11:10.840 --> 00:11:13.940 different function. It is so hyper fixated on 00:11:13.940 --> 00:11:15.980 the exact coordinates of the data it was given 00:11:15.980 --> 00:11:19.240 that a microscopic shift causes massive variations 00:11:19.240 --> 00:11:21.519 in its logic. Which completely destroys the core 00:11:21.519 --> 00:11:23.940 mission. We define learning as prediction. If 00:11:23.940 --> 00:11:26.799 your model is unstable and overfit, its predictions 00:11:26.799 --> 00:11:29.480 for any new unseen data in the real world become 00:11:29.480 --> 00:11:31.679 effectively useless. It's just guess. at that 00:11:31.679 --> 00:11:34.539 point. Okay, so if empirical risk minimization 00:11:34.539 --> 00:11:36.799 is inherently flawed because it constantly tries 00:11:36.799 --> 00:11:40.860 to memorize the practice test, how do we physically 00:11:40.860 --> 00:11:44.419 stop it? How do we force the math to care about 00:11:44.419 --> 00:11:46.820 the general rule instead of the exact data points? 00:11:47.039 --> 00:11:49.799 The cure is a fundamental concept called regularization. 00:11:50.039 --> 00:11:52.720 And this is really where the heavy lifting of 00:11:52.720 --> 00:11:54.960 statistical learning theory happens. Remember 00:11:54.960 --> 00:11:57.539 the hypothesis space H? Yeah, the sandbox. Right. 00:11:57.980 --> 00:12:00.340 Regularization is the deliberate mathematical 00:12:00.340 --> 00:12:03.159 restriction of that search space. You are artificially 00:12:03.159 --> 00:12:05.100 shrinking the sandbox. Precisely. Because if 00:12:05.100 --> 00:12:07.299 you let the algorithm search through any possible 00:12:07.299 --> 00:12:11.019 function, it will invariably find, like, a thousand 00:12:11.019 --> 00:12:13.740 degree polynomial that just snakes its way through 00:12:13.740 --> 00:12:15.720 every single training point perfectly. Just to 00:12:15.720 --> 00:12:18.659 get that zero error. Exactly. But if you restrict 00:12:18.659 --> 00:12:21.679 the hypothesis space to say only linear functions 00:12:21.679 --> 00:12:24.820 or polynomials of a very low degree of p, you 00:12:24.820 --> 00:12:27.200 physically remove the algorithm's ability to 00:12:27.200 --> 00:12:29.399 overcomplicate things. You make it impossible 00:12:29.399 --> 00:12:31.919 for the empirical risk to ever reach zero because 00:12:31.919 --> 00:12:33.960 a straight line can never perfectly connect a 00:12:33.960 --> 00:12:36.759 bunch of randomly scattered dots. Oh, it's forced 00:12:36.759 --> 00:12:38.960 to find the trend line rather than connecting 00:12:38.960 --> 00:12:41.919 the dots. Exactly. So what does this all mean? 00:12:42.559 --> 00:12:45.519 Going back to the student analogy, regularization 00:12:45.519 --> 00:12:48.519 is basically the teacher taking away the student's 00:12:48.519 --> 00:12:51.539 scratch pad and forcing them to explain their 00:12:51.539 --> 00:12:55.000 answer in one simple sentence. I love that. By 00:12:55.000 --> 00:12:57.580 restricting their options, they can't overcomplicate 00:12:57.580 --> 00:12:59.440 it or memorize it. They have to actually understand 00:12:59.440 --> 00:13:02.139 it. And the specific mathematical mechanism for 00:13:02.139 --> 00:13:05.639 this is fascinating. Let's look at Tikhonov regularization, 00:13:05.720 --> 00:13:07.659 which was detailed in the source. Walk us through 00:13:07.659 --> 00:13:10.840 it. So, Tikhonov regularization takes the standard 00:13:10.840 --> 00:13:13.259 empirical risk equation, the one desperately 00:13:13.259 --> 00:13:16.120 trying to minimize errors, and it adds a penalty 00:13:16.120 --> 00:13:19.710 term to it. A fixed positive parameter represented 00:13:19.710 --> 00:13:22.570 by the Greek letter gamma. And this gamma acts 00:13:22.570 --> 00:13:25.549 like a complexity tax. A complexity tax. That's 00:13:25.549 --> 00:13:27.169 a great way to put it. Right. Think about it 00:13:27.169 --> 00:13:29.669 geometrically. The algorithm wants to bend and 00:13:29.669 --> 00:13:31.629 twist the function to hit a stray data point 00:13:31.629 --> 00:13:34.210 and reduce its loss. But every time the function 00:13:34.210 --> 00:13:37.250 bends, the gamma parameter charges a massive 00:13:37.250 --> 00:13:40.090 mathematical tax, which inflates the total equation. 00:13:40.389 --> 00:13:42.370 So the algorithm looks at the stray point and 00:13:42.370 --> 00:13:46.279 realizes, if I bend to hit that point, My empirical 00:13:46.279 --> 00:13:48.980 risk goes down a tiny bit, but my complexity 00:13:48.980 --> 00:13:51.899 tax shoots through the roof. So to minimize the 00:13:51.899 --> 00:13:54.299 total equation, the algorithm actively chooses 00:13:54.299 --> 00:13:56.899 to ignore the stray data point. It chooses a 00:13:56.899 --> 00:13:59.659 smoother, simpler, more stable line. It basically 00:13:59.659 --> 00:14:02.120 absorbs a slightly higher empirical risk on the 00:14:02.120 --> 00:14:04.460 training data in exchange for paying a much lower 00:14:04.460 --> 00:14:06.720 complexity tax. Which mathematically guarantees 00:14:06.720 --> 00:14:09.720 the existence, uniqueness, and stability of the 00:14:09.720 --> 00:14:13.149 solution. By adding that gamma parameter, the 00:14:13.149 --> 00:14:15.370 algorithm is barred from generating an unstable 00:14:15.370 --> 00:14:17.750 wildly contorted function. And if we connect 00:14:17.750 --> 00:14:20.049 this to the bigger picture, stability is the 00:14:20.049 --> 00:14:22.450 absolute linchpin of machine learning. If you 00:14:22.450 --> 00:14:24.570 can guarantee the mathematical stability of the 00:14:24.570 --> 00:14:27.289 solution, meaning a tiny change in input won't 00:14:27.289 --> 00:14:30.450 cause a massive swing in output, then generalization 00:14:30.450 --> 00:14:32.730 and consistency are mathematically guaranteed 00:14:32.730 --> 00:14:35.799 to follow. But... And this is crucial. Statistical 00:14:35.799 --> 00:14:37.679 learning theory doesn't just stop at, hey, it 00:14:37.679 --> 00:14:41.120 works. It demands rigorous proof that the proxy 00:14:41.120 --> 00:14:43.799 measure, our empirical risk, won't drastically 00:14:43.799 --> 00:14:46.279 betray us when we deploy the model in the real 00:14:46.279 --> 00:14:49.320 world. Oh, yeah. The math gets intense here. 00:14:49.519 --> 00:14:51.259 Yeah, this is where we get into the really advanced 00:14:51.259 --> 00:14:54.960 territory. Bounding the risk. using Heftin's 00:14:54.960 --> 00:14:57.980 inequality. This is truly a beautiful piece of 00:14:57.980 --> 00:15:00.120 statistics because even with regularization, 00:15:00.200 --> 00:15:03.080 there's always a lingering fear, right? What 00:15:03.080 --> 00:15:05.539 if our training data was just incredibly unlucky? 00:15:06.179 --> 00:15:09.379 What if it totally misrepresents the true probability 00:15:09.379 --> 00:15:11.919 distribution Z? Like drawing 10 red cards in 00:15:11.919 --> 00:15:13.929 a row from a deck. in assuming the whole deck 00:15:13.929 --> 00:15:16.909 is red. Exactly. Hufting's inequality allows 00:15:16.909 --> 00:15:19.370 us to mathematically bound the probability of 00:15:19.370 --> 00:15:21.549 that worst -case scenario. It calculates the 00:15:21.549 --> 00:15:24.330 exact probability that the gap between our training 00:15:24.330 --> 00:15:26.669 score, the empirical risk, and our real -world 00:15:26.669 --> 00:15:29.250 score, the expected risk, will exceed a certain 00:15:29.250 --> 00:15:31.909 tolerance level. Right. And what Hufting proves 00:15:31.909 --> 00:15:34.370 is that this deviation follows a subgaussian 00:15:34.370 --> 00:15:37.500 distribution. And for those visualizing the math 00:15:37.500 --> 00:15:39.799 at home, a subgaussian distribution is crucial 00:15:39.799 --> 00:15:43.139 because its tails drop off incredibly fast. In 00:15:43.139 --> 00:15:45.639 a standard normal distribution, extreme events 00:15:45.639 --> 00:15:49.139 are rare, sure, but in a subgaussian bound, the 00:15:49.139 --> 00:15:51.940 probability of the empirical risk being catastrophically 00:15:51.940 --> 00:15:55.360 misleading decays exponentially as you add more 00:15:55.360 --> 00:15:58.179 data points. Wow. It puts an airtight mathematical 00:15:58.179 --> 00:16:01.019 ceiling on how horribly wrong we can be. But 00:16:01.019 --> 00:16:03.639 wait, there is a massive catch here that I initially 00:16:03.639 --> 00:16:06.750 struggled with when the source. Hufting's inequality 00:16:06.750 --> 00:16:10.950 is great if you are just testing one single specific 00:16:10.950 --> 00:16:13.669 function. But in machine learning, we aren't 00:16:13.669 --> 00:16:15.909 doing that, are we? No, we definitely aren't. 00:16:16.049 --> 00:16:18.350 We are doing empirical risk minimization. We 00:16:18.350 --> 00:16:21.149 are testing an entire hypothesis space of functions 00:16:21.149 --> 00:16:23.570 and picking the best one. Right, and that changes 00:16:23.570 --> 00:16:25.789 the math completely. Because if you test one 00:16:25.789 --> 00:16:28.450 million different functions, just by sheer dumb 00:16:28.450 --> 00:16:30.649 luck, one of those functions is going to happen 00:16:30.649 --> 00:16:33.000 to perfectly match your training data. purely 00:16:33.000 --> 00:16:35.580 by chance, even if it's completely useless in 00:16:35.580 --> 00:16:38.039 reality. Exactly. It's the multiple comparisons 00:16:38.039 --> 00:16:41.320 problem on steroids. So we can't just bound the 00:16:41.320 --> 00:16:43.860 risk of one single function. We have to bound 00:16:43.860 --> 00:16:46.419 the probability of the supremum, the absolute 00:16:46.419 --> 00:16:49.200 highest possible deviation across the entire 00:16:49.200 --> 00:16:51.860 infinite class of functions in our hypothesis 00:16:51.860 --> 00:16:55.019 space. And doing that introduces an extra mathematical 00:16:55.019 --> 00:16:57.399 cost. Yeah. We have to pay a penalty for searching 00:16:57.399 --> 00:17:00.220 such a large space. And that cost is quantified 00:17:00.220 --> 00:17:02.659 by something called the shattering number. denoted 00:17:02.659 --> 00:17:06.599 as s of script f comma n. And the shattering 00:17:06.599 --> 00:17:09.039 number is one of the most elegant concepts in 00:17:09.039 --> 00:17:11.299 functional analysis, hands down. I have to admit, 00:17:11.380 --> 00:17:13.769 when I first read shattering number, It sounded 00:17:13.769 --> 00:17:16.150 like a weapon from a sci -fi novel. Like, how 00:17:16.150 --> 00:17:18.329 does a function shatter data? Right. It's an 00:17:18.329 --> 00:17:20.410 aggressive term, but think of shattering as the 00:17:20.410 --> 00:17:22.609 ultimate test of a function class's complexity. 00:17:23.170 --> 00:17:25.130 Let's say you have a set of n data points on 00:17:25.130 --> 00:17:27.309 a graph, and some are labeled as cats, some as 00:17:27.309 --> 00:17:30.829 dogs. OK. If your hypothesis space is so incredibly 00:17:30.829 --> 00:17:32.890 flexible that no matter how you randomly scramble 00:17:32.890 --> 00:17:35.329 those labels, it can always draw a boundary that 00:17:35.329 --> 00:17:38.069 perfectly separates the cats from the dogs, then 00:17:38.069 --> 00:17:40.309 your hypothesis space has shattered that data 00:17:40.309 --> 00:17:43.460 set. So it's a measure of pure capacity. Like, 00:17:43.779 --> 00:17:46.279 if I have three data points on a 2D plane, a 00:17:46.279 --> 00:17:48.380 simple straight line can chatter them. No matter 00:17:48.380 --> 00:17:50.519 which ones are cats or dogs, I can draw a single 00:17:50.519 --> 00:17:52.579 straight line to separate them. Exactly. But 00:17:52.579 --> 00:17:55.359 what if you have four data points arranged in 00:17:55.359 --> 00:17:57.900 a square where the opposing corners are cats 00:17:57.900 --> 00:18:01.099 and the other corners are dogs, like an XOR configuration? 00:18:01.200 --> 00:18:03.359 Oh, a single straight line can't separate those? 00:18:03.500 --> 00:18:06.990 It's impossible. Precisely. A linear classifier 00:18:06.990 --> 00:18:09.769 cannot shatter four points in that configuration. 00:18:10.250 --> 00:18:12.569 Its shattering number is limited. And that limitation 00:18:12.569 --> 00:18:15.269 is a good thing, right? Yes. The shattering number 00:18:15.269 --> 00:18:18.170 mathematically quantifies the exact combinatorial 00:18:18.170 --> 00:18:21.369 capacity of your hypothesis space. When you plug 00:18:21.369 --> 00:18:23.309 that shattering number into the risk bounds, 00:18:23.450 --> 00:18:25.549 it rigorously proves that as long as the capacity 00:18:25.549 --> 00:18:27.549 of your function class is controlled, as long 00:18:27.549 --> 00:18:29.490 as it can't just shatter anything you throw at 00:18:29.490 --> 00:18:31.410 it and you have a sufficiently large n number 00:18:31.410 --> 00:18:33.930 of samples, your empirical risk will accurately 00:18:33.740 --> 00:18:37.000 reflect the true expected risk. Man, it's just 00:18:37.000 --> 00:18:38.900 incredible. We've traced the entire logical chain 00:18:38.900 --> 00:18:41.160 here. We really have. We started with inputs 00:18:41.160 --> 00:18:44.279 and outputs in massive vector spaces. We gave 00:18:44.279 --> 00:18:47.079 the machine a hypothesis space to search and 00:18:47.079 --> 00:18:50.480 a convex loss function, a rubber band, to brutally 00:18:50.480 --> 00:18:53.380 pull it toward the minimum error. We confronted 00:18:53.380 --> 00:18:56.500 the terrifying trap of overfitting, where memorizing 00:18:56.500 --> 00:18:59.559 the data leads to complete instability. And we 00:18:59.559 --> 00:19:02.319 saw how the math basically saves itself through 00:19:02.319 --> 00:19:04.660 regularization and complexity tax as we force 00:19:04.660 --> 00:19:07.259 the machine to seek stability. Right. And finally, 00:19:07.960 --> 00:19:10.500 using Hoeffding's bounds and shattering numbers, 00:19:10.940 --> 00:19:13.440 we don't just hope it works. We mathematically 00:19:13.440 --> 00:19:16.000 prove that the machine has extracted the true 00:19:16.000 --> 00:19:18.720 signal from the noise. So you now possess the 00:19:18.720 --> 00:19:20.980 underlying theorems of how AI models actually 00:19:20.980 --> 00:19:23.309 learn. The next time you see a machine doing 00:19:23.309 --> 00:19:25.589 something remarkably human, you know it is a 00:19:25.589 --> 00:19:28.430 magic. It's Cartesian products, convex optimization, 00:19:28.950 --> 00:19:31.049 regularization parameters, and risk bounding, 00:19:31.269 --> 00:19:33.529 all executing in fractions of a second. It is 00:19:33.529 --> 00:19:36.630 a beautifully rigorous framework, but this entire 00:19:36.630 --> 00:19:38.650 architecture raises a really profound question, 00:19:38.730 --> 00:19:40.730 and it's something I want to leave you to ponder. 00:19:40.930 --> 00:19:43.099 Oh, let's hear it. We established right at the 00:19:43.099 --> 00:19:45.900 beginning that statistical learning theory relies 00:19:45.900 --> 00:19:49.599 entirely on the existence of Z, an unknown but 00:19:49.599 --> 00:19:52.539 fixed probability distribution in the real world. 00:19:52.980 --> 00:19:55.299 Right. The math guarantees that if we sample 00:19:55.299 --> 00:19:57.779 enough data from that fixed reality, we can map 00:19:57.779 --> 00:20:00.400 it accurately. But think about how algorithms 00:20:00.400 --> 00:20:03.099 are actually deployed today. They are predicting 00:20:03.099 --> 00:20:06.119 financial markets, viral trends, human behavior. 00:20:06.160 --> 00:20:08.559 Which are definitely not fixed. Exactly. Human 00:20:08.559 --> 00:20:11.240 behavior is a moving target. The ground truth 00:20:11.079 --> 00:20:14.579 of society changes constantly. So what happens 00:20:14.579 --> 00:20:16.799 to the mathematical guarantees of machine learning 00:20:16.799 --> 00:20:19.200 when the underlying probability distribution 00:20:19.200 --> 00:20:21.640 of the world shifts faster than we can collect 00:20:21.640 --> 00:20:24.799 new data to train it? That is a staggering thought 00:20:24.799 --> 00:20:26.440 to leave on. Thank you so much for joining us 00:20:26.440 --> 00:20:27.339 on this deep dive.