WEBVTT 00:00:00.000 --> 00:00:02.580 Imagine waking up tomorrow and you just have 00:00:02.580 --> 00:00:06.219 this brain that remembers every single detail 00:00:06.219 --> 00:00:09.400 of your life flawlessly. Oh, wow. Yeah. Like 00:00:09.400 --> 00:00:11.640 absolute perfect recall. Exactly. You remember 00:00:11.640 --> 00:00:14.119 every drop of rain you've ever felt, every gust 00:00:14.119 --> 00:00:16.820 of wind, the exact shade of every cloud you've 00:00:16.820 --> 00:00:19.019 ever seen. Sounds like a superpower, right? Sounds 00:00:19.019 --> 00:00:21.399 incredible until you actually have to use it. 00:00:21.399 --> 00:00:23.519 Right. Because then you step outside your front 00:00:23.519 --> 00:00:26.780 door, you feel a drop of water, you notice a 00:00:26.780 --> 00:00:29.710 dog barking across the street. a red car drives 00:00:29.710 --> 00:00:32.469 by, and the wind shifts slightly. Just a bunch 00:00:32.469 --> 00:00:35.369 of random everyday noise. Yeah, and your perfectly 00:00:35.369 --> 00:00:38.670 detailed brain tries to calculate the exact mathematical 00:00:38.670 --> 00:00:41.689 relationship between all those random variables, 00:00:41.729 --> 00:00:45.310 just to make one single decision. Like, should 00:00:45.310 --> 00:00:47.590 you open your umbrella? You'd be stuck. Three 00:00:47.590 --> 00:00:49.570 hours later, you're still standing on your porch, 00:00:50.049 --> 00:00:52.450 completely paralyzed, trying to perfectly predict 00:00:52.450 --> 00:00:55.030 the weather. You would be completely frozen by 00:00:55.030 --> 00:00:57.810 the sheer volume of information. And you know, 00:00:57.929 --> 00:00:59.890 that paralysis isn't just some quirky thought 00:00:59.890 --> 00:01:02.689 experiment about human psychology. No, it is 00:01:02.689 --> 00:01:05.549 the absolute definition of a mathematical trap 00:01:05.549 --> 00:01:08.250 that governs literally any system trying to learn 00:01:08.250 --> 00:01:11.049 about the world. Well, welcome to today's custom 00:01:11.049 --> 00:01:14.129 deep dive. Today, we are opening up a single 00:01:14.129 --> 00:01:18.090 dense textbook level source, a Wikipedia article 00:01:18.090 --> 00:01:21.040 on the bias variance trade off. Which I know 00:01:21.040 --> 00:01:23.000 sounds intense. Yeah, I know that sounds like 00:01:23.000 --> 00:01:25.519 an intimidating statistical concept that only 00:01:25.519 --> 00:01:27.920 like machine learning engineers in a windowless 00:01:27.920 --> 00:01:30.239 basement care about. It definitely carries a 00:01:30.239 --> 00:01:32.359 heavy academic weight when you first glance at 00:01:32.359 --> 00:01:34.680 it. But our mission for you today is to show 00:01:34.680 --> 00:01:37.420 you why this trade -off is actually the fundamental 00:01:37.420 --> 00:01:40.480 law of learning itself. Yes, absolutely. Whether 00:01:40.480 --> 00:01:42.299 you're trying to understand how a massive neural 00:01:42.299 --> 00:01:45.359 network is being trained to drive a car or you 00:01:45.359 --> 00:01:47.819 just want to know how your own brain makes rapid 00:01:47.819 --> 00:01:51.140 -fire decisions on a Tuesday morning. understanding 00:01:51.140 --> 00:01:53.739 this trade -off is the ultimate shortcut. Because 00:01:53.739 --> 00:01:55.780 perfection in learning is just a mathematical 00:01:55.780 --> 00:01:58.260 illusion. Right. And by the end of this deep 00:01:58.260 --> 00:02:00.579 dive, you're gonna have a powerful new mental 00:02:00.579 --> 00:02:03.000 model for why trying to be absolutely perfect 00:02:03.000 --> 00:02:06.299 usually leads to terrible catastrophic predictions. 00:02:06.859 --> 00:02:09.939 Anytime you try to learn from raw data, you are 00:02:09.939 --> 00:02:13.259 immediately hunted by two opposing forces. We 00:02:13.259 --> 00:02:17.060 can think of them as the two monsters of machine 00:02:17.060 --> 00:02:19.580 learning. I love that. And they are constantly 00:02:19.580 --> 00:02:22.139 threatening to ruin our algorithms, bias, and 00:02:22.139 --> 00:02:24.919 variance. But before we can balance them, we 00:02:24.919 --> 00:02:28.419 need to understand how they attack. OK, let's 00:02:28.419 --> 00:02:30.759 unpack this. Let's start with bias. Because when 00:02:30.759 --> 00:02:32.919 we talk about bias in this statistical sense, 00:02:32.979 --> 00:02:36.039 we aren't talking about prejudice. Right, right. 00:02:36.159 --> 00:02:38.259 Not at all. Bias error comes from a learning 00:02:38.259 --> 00:02:40.780 algorithm making overly simplistic assumptions 00:02:40.780 --> 00:02:45.060 about the world. It's stubborn. Stubborn is the 00:02:45.060 --> 00:02:47.360 perfect word for it. A high bias model comes 00:02:47.360 --> 00:02:49.300 to the data with its mind already made up about 00:02:49.300 --> 00:02:51.680 what the answer should look like. And because 00:02:51.680 --> 00:02:54.280 it refuses to adapt to the nuance of the information 00:02:54.280 --> 00:02:56.919 you feed it, it completely misses the actual 00:02:56.919 --> 00:02:59.360 relevant relationships between the features and 00:02:59.360 --> 00:03:01.639 the target outcome. And in the data science world, 00:03:01.759 --> 00:03:04.139 we call this underfitting, right? Exactly. Underfitting. 00:03:04.259 --> 00:03:06.520 The model just under -represents the complexity 00:03:06.520 --> 00:03:09.379 of reality. And then waiting on the exact opposite 00:03:09.379 --> 00:03:11.759 end of the spectrum, we have variance. The second 00:03:11.759 --> 00:03:15.009 monster. Right. Variance error is this extreme 00:03:15.009 --> 00:03:18.590 hypersensitivity to tiny random fluctuations 00:03:18.590 --> 00:03:21.710 in the training data. A high variance model doesn't 00:03:21.710 --> 00:03:24.210 have rigid assumptions. It's actually too flexible. 00:03:24.530 --> 00:03:26.610 Way too flexible. It memorizes everything, including 00:03:26.610 --> 00:03:29.490 the random meaningless noise. Instead of finding 00:03:29.490 --> 00:03:32.469 the underlying pattern, it just maps the chaos. 00:03:33.090 --> 00:03:36.629 And that is what we call overfitting. If you're 00:03:36.629 --> 00:03:39.409 wondering why you should care about this, just 00:03:39.409 --> 00:03:41.569 think about how you judged the last person you 00:03:41.569 --> 00:03:43.379 met. Oh, that's a good way to look at it. Yeah. 00:03:43.460 --> 00:03:45.939 If you assume everyone from a certain city acts 00:03:45.939 --> 00:03:48.860 exactly the same way, you have high bias. You're 00:03:48.860 --> 00:03:51.080 stubborn, and you'll be completely wrong about 00:03:51.080 --> 00:03:53.699 their individual quirks. Right. But if you assume 00:03:53.699 --> 00:03:55.919 that because your new co -worker sneezed twice 00:03:55.919 --> 00:03:58.500 during a meeting, they must be allergic to your 00:03:58.500 --> 00:04:01.240 perfume and secretly hate you well, you have 00:04:01.240 --> 00:04:03.319 high variance. You are definitely overfitting 00:04:03.319 --> 00:04:06.580 to random noise. Exactly. Let me bring in a visual 00:04:06.580 --> 00:04:08.340 analogy from the source material to really nail 00:04:08.340 --> 00:04:10.900 this down, comparing this to accuracy and precision. 00:04:11.259 --> 00:04:14.159 because accuracy is a great way to quantify bias. 00:04:14.419 --> 00:04:17.639 Yes, it is. Imagine you have a scatter plot of 00:04:17.639 --> 00:04:20.279 data points on a graph, and they clearly form 00:04:20.279 --> 00:04:23.279 a U shape, like a big sweeping curve. Okay. If 00:04:23.279 --> 00:04:25.860 you try to fit a completely straight line through 00:04:25.860 --> 00:04:29.800 those points, that is a high bias model. You're 00:04:29.800 --> 00:04:32.339 selecting very local information. Sure, right 00:04:32.339 --> 00:04:34.319 at the two spots where the straight line accidentally 00:04:34.319 --> 00:04:36.759 crosses the U -curve, it looks perfectly accurate. 00:04:36.819 --> 00:04:39.759 But overall? Overall, it is completely underfitting 00:04:39.759 --> 00:04:42.480 the reality of that U shape. Precision, on the 00:04:42.480 --> 00:04:45.259 other hand, describes variance. Precision is 00:04:45.259 --> 00:04:47.899 about selecting from a broader space and trying 00:04:47.899 --> 00:04:50.519 to hit every single mark. Right. If we stick 00:04:50.519 --> 00:04:52.899 to your U -shaped data, a high -variance model 00:04:52.899 --> 00:04:56.220 absolutely refuses to draw a straight line. Instead, 00:04:56.660 --> 00:05:00.079 it draws a chaotic, wildly looping rollercoaster 00:05:00.079 --> 00:05:03.220 of a curve that perfectly connects every single 00:05:03.220 --> 00:05:05.759 tiny dot on the graph. Wow, OK. It captures the 00:05:05.759 --> 00:05:07.620 main data, but it also captures all the random 00:05:07.620 --> 00:05:10.540 measurement errors and outliers. So it's technically 00:05:10.540 --> 00:05:12.980 precise to the exact data it was trained on. 00:05:13.769 --> 00:05:16.250 you try to use that roller coaster curve to predict 00:05:16.250 --> 00:05:18.490 a new data point, it's going to be completely 00:05:18.490 --> 00:05:20.889 useless. Completely useless. It'll predict some 00:05:20.889 --> 00:05:23.009 wild spike just because there happened to be 00:05:23.009 --> 00:05:25.569 a typo in the original data. What's fascinating 00:05:25.569 --> 00:05:28.569 here is how the source visualizes this tug -of 00:05:28.569 --> 00:05:31.529 -war using a mathematical approximation tool. 00:05:32.310 --> 00:05:35.649 Radial basis functions or RBFs. Okay, wave it 00:05:35.649 --> 00:05:38.870 on me. Imagine a true underlying pattern that 00:05:38.870 --> 00:05:42.029 is shaped like a gentle curve, but it has a really 00:05:42.029 --> 00:05:44.769 deep dip right in the middle. The article shows 00:05:44.769 --> 00:05:46.990 what happens when we try to train a model to 00:05:46.990 --> 00:05:50.250 approximate that curve using noisy data over 00:05:50.250 --> 00:05:52.730 several different trials. So they control how 00:05:52.730 --> 00:05:54.949 flexible the model is allowed to be. Exactly. 00:05:55.389 --> 00:05:57.430 When they force the model to be rigid, giving 00:05:57.430 --> 00:06:00.350 it a wide spread, the bias is extremely high. 00:06:00.850 --> 00:06:03.029 The model completely misses that central dip 00:06:03.029 --> 00:06:05.709 in the curve. It just draws a lazy, shallow swoop 00:06:05.709 --> 00:06:07.970 over the whole thing. OK, so it underfits. Right. 00:06:08.290 --> 00:06:10.750 But here's the crucial part. If you run the trial 00:06:10.750 --> 00:06:13.029 10 different times, feeding it slightly different 00:06:13.029 --> 00:06:15.610 noisy data each time, that resulting shallow 00:06:15.610 --> 00:06:18.050 swoop looks almost identical every single time. 00:06:18.209 --> 00:06:21.050 Because high bias means low variance. Yes. It 00:06:21.050 --> 00:06:23.290 is consistently wrong in the exact same way. 00:06:23.449 --> 00:06:26.230 It's stable, but it's blind to the nuance. Then, 00:06:26.589 --> 00:06:28.649 they flip the script. They decrease the spread, 00:06:28.829 --> 00:06:31.670 making the model highly flexible. Suddenly, the 00:06:31.670 --> 00:06:34.189 approximation successfully hugs that deep dip 00:06:34.189 --> 00:06:36.970 in the curve. The bias drops completely. But 00:06:36.970 --> 00:06:39.829 I'm guessing there's a catch. A huge one. Because 00:06:39.829 --> 00:06:42.470 it's so sensitive now, the predictions for the 00:06:42.470 --> 00:06:45.470 exact same spot jump wildly depending on exactly 00:06:45.470 --> 00:06:47.350 where the noisy training data happened to fall 00:06:47.350 --> 00:06:49.689 in that specific trial. So the curves are just 00:06:49.689 --> 00:06:52.800 flying all over the place. Exactly. Low bias, 00:06:52.920 --> 00:06:55.779 but massive chaotic variance. Which naturally 00:06:55.779 --> 00:06:57.899 makes you wonder, why not just build a smarter 00:06:57.899 --> 00:07:01.120 model? Like, why can't we have an algorithm that 00:07:01.120 --> 00:07:03.980 hugs the deep dip perfectly, but just ignores 00:07:03.980 --> 00:07:06.480 the random noise? Why is it a trade -off? We 00:07:06.480 --> 00:07:08.720 can't just wish the problem away because we are 00:07:08.720 --> 00:07:11.779 trapped by a rigid mathematical reality. There 00:07:11.779 --> 00:07:14.220 is an actual equation called the bias -variance 00:07:14.220 --> 00:07:16.980 decomposition. It proves that whenever you measure 00:07:16.980 --> 00:07:20.139 your overall expected error on unseen data, what 00:07:20.139 --> 00:07:23.029 statisticians call the mean squared error, that 00:07:23.029 --> 00:07:25.350 total error is always broken down into exactly 00:07:25.350 --> 00:07:27.269 three parts. Right. The expected error is the 00:07:27.269 --> 00:07:30.069 bias squared plus the variance plus a third piece 00:07:30.069 --> 00:07:32.389 of the puzzle called the irreducible error. And 00:07:32.389 --> 00:07:36.269 that irreducible error forms a hard lower bound. 00:07:36.689 --> 00:07:38.649 You can never get your error to zero, no matter 00:07:38.649 --> 00:07:41.550 how smart your AI is, because the universe is 00:07:41.550 --> 00:07:44.089 just inherently noisy. Give me an example of 00:07:44.089 --> 00:07:46.029 that. Let's say you're building a model to predict 00:07:46.029 --> 00:07:50.019 house prices. You have data on the square footage, 00:07:50.259 --> 00:07:52.300 the neighborhood, the number of bedrooms. Sure, 00:07:52.639 --> 00:07:54.579 standard stuff. But maybe the buyer was just 00:07:54.579 --> 00:07:57.339 in a fantastic mood that day because their favorite 00:07:57.339 --> 00:08:00.740 sports team won, so they paid $10 ,000 over asking. 00:08:01.000 --> 00:08:04.800 Ah. That random human element is the irreducible 00:08:04.800 --> 00:08:07.800 error. You just cannot predict it. So if the 00:08:07.800 --> 00:08:10.459 irreducible error is locked in, you can only 00:08:10.459 --> 00:08:13.019 control the other two dials, bias and variance. 00:08:13.199 --> 00:08:15.439 Exactly. And the equation dictates that as a 00:08:15.439 --> 00:08:18.360 model's complexity increases, it captures more 00:08:18.360 --> 00:08:21.680 data points so the bias goes down. But to capture 00:08:21.680 --> 00:08:24.199 them, it has to twist and bend more, which forces 00:08:24.199 --> 00:08:26.480 the variance to go up. That is the trade -off 00:08:26.480 --> 00:08:29.319 in a nutshell. But wait, I have to push back 00:08:29.319 --> 00:08:31.699 on this idea of complexity. Isn't complexity 00:08:31.699 --> 00:08:34.159 just about the number of parameters a model has? 00:08:34.419 --> 00:08:37.059 That's a very common assumption, actually. Because 00:08:37.059 --> 00:08:39.039 you hear this all the time in tech news, like, 00:08:39.299 --> 00:08:41.620 this new AI has a trillion parameters. It's incredibly 00:08:41.620 --> 00:08:45.320 complex. If we keep adding tunable dials to a 00:08:45.320 --> 00:08:47.860 model, doesn't it automatically become a high 00:08:47.860 --> 00:08:50.340 -variance, overfitting mess? If we connect this 00:08:50.340 --> 00:08:52.750 to the bigger picture... That assumption is a 00:08:52.750 --> 00:08:56.210 classic textbook fallacy. The source specifically 00:08:56.210 --> 00:08:58.850 warns against this. Oh, really? Yeah, assuming 00:08:58.850 --> 00:09:01.409 that complexity just equals number of parameters 00:09:01.409 --> 00:09:04.350 and that a low parameter count means a simple 00:09:04.350 --> 00:09:08.830 high bias model is entirely wrong. How so? Like, 00:09:08.889 --> 00:09:11.309 if I only have two dials to turn, how much damage 00:09:11.309 --> 00:09:13.970 can I really do? Well, imagine a tailor who only 00:09:13.970 --> 00:09:16.779 knows how to sew a zigzag stitch. They only have 00:09:16.779 --> 00:09:20.039 two tools, two parameters. They can control how 00:09:20.039 --> 00:09:22.440 tall the zigzag is and how tight the spacing 00:09:22.440 --> 00:09:24.700 is between the zigzags. Okay, just two dials. 00:09:24.799 --> 00:09:27.059 Right. By the parameter count logic, this is 00:09:27.059 --> 00:09:29.200 the simplest, highest bias model in the world. 00:09:29.320 --> 00:09:31.600 Yeah, it sounds incredibly rigid. But if you 00:09:31.600 --> 00:09:33.779 give that tailor a piece of fabric with a hundred 00:09:33.779 --> 00:09:36.480 random dots scattered all over it, they can crank 00:09:36.480 --> 00:09:38.799 that tightness dial to an extreme. Oh, I see 00:09:38.799 --> 00:09:40.990 where this is going. They can make the zigzag 00:09:40.990 --> 00:09:44.230 oscillate at an incredibly high frequency, zipping 00:09:44.230 --> 00:09:46.610 up and down millions of times in a single inch, 00:09:47.250 --> 00:09:49.750 forcing the thread to eventually touch every 00:09:49.750 --> 00:09:53.090 single random dot perfectly. Wow. So they only 00:09:53.090 --> 00:09:55.830 have two dials, but they are acting like a completely 00:09:55.830 --> 00:09:59.080 unhinged high -variance machine. Exactly. It 00:09:59.080 --> 00:10:01.740 manages to have both high bias, because it's 00:10:01.740 --> 00:10:04.159 rigidly constrained to only sewing a zigzag pattern 00:10:04.159 --> 00:10:06.960 and high variance, because the exact path of 00:10:06.960 --> 00:10:10.399 that wild zigzag will change drastically if even 00:10:10.399 --> 00:10:13.360 one dot moves a millimeter. That is fascinating. 00:10:13.639 --> 00:10:15.960 Complexity is about how the model behaves and 00:10:15.960 --> 00:10:18.620 bends, its capacity to oscillate and fit the 00:10:18.620 --> 00:10:20.980 noise, not just how many moving parts it has 00:10:20.980 --> 00:10:23.740 inside. So if we are mathematically trapped by 00:10:23.740 --> 00:10:25.700 this equation, and we can't just judge a model 00:10:25.700 --> 00:10:28.259 by looking at its parameter count, how do machine 00:10:28.259 --> 00:10:30.519 learning engineers actually build AI that works 00:10:30.519 --> 00:10:33.399 in the real world? How do they hack this trade 00:10:33.399 --> 00:10:35.600 -off? They don't try to defeat it. They treat 00:10:35.600 --> 00:10:37.820 the trade -off as a series of levers they can 00:10:37.820 --> 00:10:40.000 intentionally tune based on what they need. They 00:10:40.000 --> 00:10:42.159 open up the engineer's toolkit. Let's look at 00:10:42.159 --> 00:10:44.279 some of those tools. First, the raw material 00:10:44.279 --> 00:10:47.279 itself, the data. Right. The data is key. Because 00:10:47.519 --> 00:10:50.620 Adding more training data generally decreases 00:10:50.620 --> 00:10:53.340 variance. It acts like a weighted blanket smoothing 00:10:53.340 --> 00:10:56.139 out the random noise because the true pattern 00:10:56.139 --> 00:10:59.080 becomes overwhelmingly louder than the outliers. 00:10:59.240 --> 00:11:02.080 That's a great analogy. But if you add more features, 00:11:02.360 --> 00:11:05.529 say, adding roof material and proximity to a 00:11:05.529 --> 00:11:08.409 coffee shop to our house pricing model, that 00:11:08.409 --> 00:11:10.669 tends to decrease bias because you're adding 00:11:10.669 --> 00:11:13.429 nuance. But it increases variance. Exactly, because 00:11:13.429 --> 00:11:16.210 you're giving the model more ways to accidentally 00:11:16.210 --> 00:11:18.629 overfit to random quirks. And then we look at 00:11:18.629 --> 00:11:21.350 the specific algorithms they deploy. Let's take 00:11:21.350 --> 00:11:25.870 K nearest neighbors or KNN. This is a model that 00:11:25.870 --> 00:11:28.149 predicts an outcome just by looking at the K 00:11:28.149 --> 00:11:30.990 closest data points. Let's ground this. If you're 00:11:30.990 --> 00:11:33.259 trying to price a house, and you set K to one, 00:11:33.600 --> 00:11:35.259 that's a one nearest neighbor approach. Meaning 00:11:35.259 --> 00:11:37.779 you just look at the exact price of the one single 00:11:37.779 --> 00:11:39.899 house right next door. Right. And if that house 00:11:39.899 --> 00:11:41.779 next door happened to be sold to a desperate 00:11:41.779 --> 00:11:44.720 buyer who overpaid, your prediction for your 00:11:44.720 --> 00:11:47.120 house is going to be way off. That is massive 00:11:47.120 --> 00:11:49.820 variance. You are hypersensitive to the local 00:11:49.820 --> 00:11:53.820 noise. But if you set K to 100, you are averaging 00:11:53.820 --> 00:11:56.980 the prices of the 100 nearest houses. You smooth 00:11:56.980 --> 00:12:00.000 out the crazy buyers. Setting K to a high number 00:12:00.000 --> 00:12:03.200 forces the model to have high bias and low variance. 00:12:03.799 --> 00:12:06.620 It's a safer bet, even if it misses the specific 00:12:06.620 --> 00:12:09.440 charm of your exact street. Makes sense. Interestingly, 00:12:09.639 --> 00:12:12.080 the source notes a fascinating mathematical quirk 00:12:12.080 --> 00:12:15.549 here. If you use that one year as neighbor estimator, 00:12:15.950 --> 00:12:18.070 but you let your training data approach infinity, 00:12:18.549 --> 00:12:20.309 meaning you have an infinite number of houses 00:12:20.309 --> 00:12:23.330 to compare the bias of that model, actually vanishes 00:12:23.330 --> 00:12:26.490 entirely. Wow, the math gets deep quickly. Yeah. 00:12:27.090 --> 00:12:29.029 But my absolute favorite part of the toolkit 00:12:29.029 --> 00:12:31.370 is ensemble learning. It feels like a magic trick 00:12:31.370 --> 00:12:33.590 where you just cheat the system. It really does. 00:12:33.809 --> 00:12:36.379 You have two main strategies here. boosting and 00:12:36.379 --> 00:12:39.179 bagging, two very different mechanisms for gaming 00:12:39.179 --> 00:12:41.659 the math. Let's look at boosting first. In boosting, 00:12:41.759 --> 00:12:44.139 you take a bunch of weak models, models that 00:12:44.139 --> 00:12:47.059 have terrible high bias. They're incredibly stubborn 00:12:47.059 --> 00:12:49.299 and make a lot of mistakes. Right. But you chain 00:12:49.299 --> 00:12:51.179 them together sequentially. Think of it like 00:12:51.179 --> 00:12:54.220 a relay race, where every runner is intensely 00:12:54.220 --> 00:12:56.759 focused on the specific patch of mud where the 00:12:56.759 --> 00:12:59.419 previous runner slipped. I like that. The first 00:12:59.419 --> 00:13:01.580 simple model makes a prediction and inevitably 00:13:01.580 --> 00:13:04.269 gets some things wrong. The second simple model 00:13:04.269 --> 00:13:06.730 looks only at the errors of the first model and 00:13:06.730 --> 00:13:08.769 tries to correct them. And the third focuses 00:13:08.769 --> 00:13:12.059 on the errors of the second. Exactly. By the 00:13:12.059 --> 00:13:14.879 end of the chain, this ensemble of highly biased 00:13:14.879 --> 00:13:18.600 weak models has collectively achieved a drastically 00:13:18.600 --> 00:13:21.679 lower bias than any of them could alone. And 00:13:21.679 --> 00:13:24.120 bagging works from the completely opposite direction. 00:13:24.659 --> 00:13:27.120 Is bagging essentially the jelly bean jar experiment? 00:13:27.240 --> 00:13:29.759 That is the perfect way to visualize it. If you 00:13:29.759 --> 00:13:32.320 ask one person to guess the number of jelly beans 00:13:32.320 --> 00:13:35.649 in a huge jar, Their guess might be wildly off. 00:13:35.750 --> 00:13:38.149 They might say 100. They might say 10 ,000. Any 00:13:38.149 --> 00:13:41.029 individual has high variance. Right. They are 00:13:41.029 --> 00:13:43.509 heavily overfitted to their own flawed perspective. 00:13:44.350 --> 00:13:46.970 But if you ask a thousand people to guess and 00:13:46.970 --> 00:13:49.629 you average their answers together, the random 00:13:49.629 --> 00:13:52.110 overestimates and the random underestimates cancel 00:13:52.110 --> 00:13:54.190 each other out. And the group's average is almost 00:13:54.190 --> 00:13:57.159 always shockingly close to the true number. It's 00:13:57.159 --> 00:13:59.200 spooky how well it works. That is exactly the 00:13:59.200 --> 00:14:02.100 mechanism of bagging. It combines strong learners, 00:14:02.500 --> 00:14:04.960 complex models that are heavily overfitted with 00:14:04.960 --> 00:14:07.299 high variance and averages their predictions. 00:14:07.399 --> 00:14:09.779 So it aggressively reduces the overall variance 00:14:09.779 --> 00:14:12.759 without sacrificing the low bias of the individual 00:14:12.759 --> 00:14:15.419 models. So what does this all mean in practice? 00:14:16.120 --> 00:14:19.240 Here is where I have to marvel at how counterintuitive 00:14:19.240 --> 00:14:22.500 the math gets. The source brings up regression 00:14:22.500 --> 00:14:26.299 methods, specifically lasso and ridge regression. 00:14:26.399 --> 00:14:29.720 Oh, yes. And this blew my mind. Engineers will 00:14:29.720 --> 00:14:32.179 take a standard regression model, which mathematically 00:14:32.179 --> 00:14:35.019 provides an unbiased estimate, and they will 00:14:35.019 --> 00:14:37.279 intentionally sabotage it. They deliberately 00:14:37.279 --> 00:14:39.850 inject bias into their own model. They make it 00:14:39.850 --> 00:14:42.370 purposely blind to certain details. Yes. Why 00:14:42.370 --> 00:14:44.090 would you intentionally make your AI dumber? 00:14:44.250 --> 00:14:46.309 Because variance measures sensitivity. Right. 00:14:46.590 --> 00:14:49.649 By introducing just a little bit of bias, anchoring 00:14:49.649 --> 00:14:52.529 the model so it can't twist as freely, the variance 00:14:52.529 --> 00:14:54.590 drops off a cliff. And since the total error 00:14:54.590 --> 00:14:57.629 equation is bias squared plus variance, the massive 00:14:57.629 --> 00:15:00.950 drop in variance completely overpowers the small 00:15:00.950 --> 00:15:04.889 increase in bias. The overall mean squared error 00:15:04.889 --> 00:15:08.730 on unseen data becomes vastly superior. They 00:15:08.730 --> 00:15:11.269 intentionally make the model slightly wrong about 00:15:11.269 --> 00:15:13.529 the training data so it can be drastically more 00:15:13.529 --> 00:15:16.049 right about the real world. It's genius. It is 00:15:16.049 --> 00:15:18.629 a profound realization about resource management 00:15:18.629 --> 00:15:22.190 and it extends far beyond simple regression algorithms. 00:15:23.009 --> 00:15:25.570 We see this accepted sabotage across multiple 00:15:25.570 --> 00:15:27.769 domains in computer science. Like where else? 00:15:28.000 --> 00:15:30.679 Take Monte Carlo methods, specifically Markov 00:15:30.679 --> 00:15:32.960 chain Monte Carlo, which is used for incredibly 00:15:32.960 --> 00:15:35.580 complex probability simulations. Like predicting 00:15:35.580 --> 00:15:38.460 weather patterns or stock market fluctuations. 00:15:38.720 --> 00:15:41.000 Traditional approaches to those simulations strive 00:15:41.000 --> 00:15:44.100 for absolute zero bias, but modern applications 00:15:44.100 --> 00:15:46.879 are often only asymptotically unbiased. Meaning 00:15:46.879 --> 00:15:49.360 they only hit zero bias if you run the simulation 00:15:49.360 --> 00:15:51.580 on a supercomputer until the end of time. Exactly. 00:15:51.740 --> 00:15:53.720 But computational budgets are limited, time is 00:15:53.720 --> 00:15:57.389 limited, so engineers face the trade -off. They 00:15:57.389 --> 00:15:59.690 will accept a controlled amount of bias into 00:15:59.690 --> 00:16:02.669 the simulation simply to dramatically reduce 00:16:02.669 --> 00:16:05.669 the variance. It yields a much tighter, more 00:16:05.669 --> 00:16:07.950 reliable estimation error when you only have 00:16:07.950 --> 00:16:10.909 a few hours of compute power to spare. Even in 00:16:10.909 --> 00:16:13.649 reinforcement allocation, learning like training 00:16:13.649 --> 00:16:16.870 an AI agent to play a complex video game or control 00:16:16.870 --> 00:16:20.149 a robotic arm in a factory, the algorithm's failure 00:16:20.149 --> 00:16:23.210 to be perfect is broken down into an asymptotic 00:16:23.210 --> 00:16:26.399 bias from the algorithm's design. plus an overfitting 00:16:26.399 --> 00:16:28.879 term caused by the fact that the agent only has 00:16:28.879 --> 00:16:31.480 limited data about its environment. The trade 00:16:31.480 --> 00:16:33.919 -off is inescapable everywhere in the digital 00:16:33.919 --> 00:16:36.960 world. Which brings us to a crucial pivot. We've 00:16:36.960 --> 00:16:39.539 talked extensively about math, code, and artificial 00:16:39.539 --> 00:16:42.259 intelligence. But there is a biological machine 00:16:42.259 --> 00:16:44.440 that has been solving the bias -variance trade 00:16:44.440 --> 00:16:47.120 -off every single second for millions of years. 00:16:47.299 --> 00:16:49.460 The human brain. The human brain. Here's where 00:16:49.460 --> 00:16:51.659 it gets really interesting. This entire concept 00:16:51.659 --> 00:16:53.820 leaps out of the computer science department. 00:16:53.950 --> 00:16:56.750 and completely explains human cognitive psychology. 00:16:56.830 --> 00:16:59.250 It really does. The source references the work 00:16:59.250 --> 00:17:02.009 of Gurd Jagiransar, a psychologist who studies 00:17:02.009 --> 00:17:05.009 how humans learn and make decisions. And his 00:17:05.009 --> 00:17:07.690 research argues that the human brain inherently 00:17:07.690 --> 00:17:10.589 resolves this mathematical dilemma by adopting 00:17:10.589 --> 00:17:14.289 high bias, low variance heuristics. And heuristics 00:17:14.289 --> 00:17:16.880 are just simple rules of thumb. Right. Think 00:17:16.880 --> 00:17:18.940 about the training data the human brain receives 00:17:18.940 --> 00:17:22.319 from daily experience. It is sparse, it is incredibly 00:17:22.319 --> 00:17:25.079 noisy, and it is poorly categorized. Very true. 00:17:25.140 --> 00:17:27.859 If our brains tried to adopt a zero bias approach, 00:17:28.059 --> 00:17:30.759 we would be presuming that we have precise, perfect 00:17:30.759 --> 00:17:32.720 knowledge of the true state of the world at all 00:17:32.720 --> 00:17:35.460 times. And we almost never do. This goes right 00:17:35.460 --> 00:17:37.640 back to the scenario we opened with. If you try 00:17:37.640 --> 00:17:40.240 to translate a zero bias AI into a human being, 00:17:40.599 --> 00:17:42.599 you get the person paralyzed on their front porch. 00:17:42.759 --> 00:17:44.859 Yeah, completely stuck. If you have absolutely 00:17:44.859 --> 00:17:47.380 no bias, you overfit to everything. You overthink 00:17:47.380 --> 00:17:49.539 every minor detail. You try to map the wind speed 00:17:49.539 --> 00:17:51.680 to the color of the passing cars just to decide 00:17:51.680 --> 00:17:53.859 if it's going to rain. You would drown in the 00:17:53.859 --> 00:17:55.640 variance. You would overfit to the immediate 00:17:55.640 --> 00:17:58.019 noise of your surroundings and completely fail 00:17:58.019 --> 00:18:00.589 to generalize to the next day. Judrenser points 00:18:00.589 --> 00:18:03.210 out that our resulting heuristics, what we often 00:18:03.210 --> 00:18:06.250 casually dismiss as our cognitive biases, are 00:18:06.250 --> 00:18:10.400 simple. but they produce far better, faster inferences 00:18:10.400 --> 00:18:12.960 in a wide variety of unpredictable situations. 00:18:13.599 --> 00:18:15.839 The source also mentions German and his colleagues 00:18:15.839 --> 00:18:19.079 who research generic object recognition, like 00:18:19.079 --> 00:18:21.519 how we actually learn to see and identify things. 00:18:21.779 --> 00:18:24.180 This raises an important question about how learning 00:18:24.180 --> 00:18:27.759 begins. German argues that the bias -variance 00:18:27.759 --> 00:18:30.480 dilemma mathematically proves you actually cannot 00:18:30.480 --> 00:18:33.539 learn object recognition entirely from scratch. 00:18:33.740 --> 00:18:36.400 Really? Yeah, a model -free approach, a brain 00:18:36.400 --> 00:18:39.039 born as a completely blank slate with absolutely 00:18:39.039 --> 00:18:41.980 no preconceptions, would require an impractically 00:18:41.980 --> 00:18:44.960 huge training dataset just to avoid catastrophic 00:18:44.960 --> 00:18:48.200 variance. Meaning? If a baby was born with zero 00:18:48.200 --> 00:18:50.759 biological bias, they would have to look at a 00:18:50.759 --> 00:18:52.480 million different chairs from a million different 00:18:52.480 --> 00:18:54.259 angles in a million different lighting conditions 00:18:54.259 --> 00:18:57.039 before they could reliably point to a new object 00:18:57.039 --> 00:18:59.740 and recognize the concept of a chair. Exactly. 00:19:00.000 --> 00:19:01.920 But babies don't do that. They figure it out 00:19:01.920 --> 00:19:03.960 after seeing just a few chairs. Right, because 00:19:03.960 --> 00:19:06.460 they aren't blank slates. Jemin argues there 00:19:06.460 --> 00:19:08.980 has to be a significant degree of hard wiring 00:19:08.980 --> 00:19:11.910 in the brain from birth. biological constraints. 00:19:12.490 --> 00:19:14.970 Yes. Just like a machine learning engineer has 00:19:14.970 --> 00:19:17.549 to program regularizations and constraints into 00:19:17.549 --> 00:19:19.869 an AI so it doesn't get lost, chasing the noise 00:19:19.869 --> 00:19:22.869 in its training data, human evolution has provided 00:19:22.869 --> 00:19:25.869 us with hardwired biological biases. That makes 00:19:25.869 --> 00:19:28.450 perfect sense. These innate biases constrain 00:19:28.450 --> 00:19:30.970 our learning space just enough so that we can 00:19:30.970 --> 00:19:33.289 quickly generalize from the incredibly limited 00:19:33.289 --> 00:19:36.170 messy data we experience as we grow up. Okay, 00:19:36.170 --> 00:19:38.970 we have covered some serious ground today. To 00:19:38.970 --> 00:19:41.190 wrap this all up, Whether you are an artificial 00:19:41.190 --> 00:19:43.529 neural network trying to predict global supply 00:19:43.529 --> 00:19:46.410 chains or a human being trying to predict if 00:19:46.410 --> 00:19:48.730 that shadow in the alley is a threat, learning 00:19:48.730 --> 00:19:51.170 is a tightrope walk. A very delicate tightrope. 00:19:51.410 --> 00:19:53.450 Lean too far into your stubborn assumptions, 00:19:53.990 --> 00:19:56.450 and you underfit reality. You become blind to 00:19:56.450 --> 00:19:58.769 the nuance. But if you react too much to every 00:19:58.769 --> 00:20:02.150 single new data point you see, you overfit. You 00:20:02.150 --> 00:20:04.390 start hallucinating patterns in the random noise. 00:20:04.599 --> 00:20:06.900 The foundational math of the universe dictates 00:20:06.900 --> 00:20:09.099 that you cannot perfectly solve both at the same 00:20:09.099 --> 00:20:11.680 time. You have to tune the dial. And that is 00:20:11.680 --> 00:20:13.680 the ultimate takeaway. And that mathematical 00:20:13.680 --> 00:20:17.119 reality leaves us with one final, perhaps provocative 00:20:17.119 --> 00:20:19.519 thought to walk away with. Oh, lay on me. In 00:20:19.519 --> 00:20:22.480 our modern culture, the word bias is almost universally 00:20:22.480 --> 00:20:25.160 treated as a defect. Yeah, totally. We use it 00:20:25.160 --> 00:20:28.420 as a synonym for a flaw in logic, a failure of 00:20:28.420 --> 00:20:30.660 objectivity that needs to be completely eradicated 00:20:30.660 --> 00:20:33.970 from our minds. But when we look at the cold, 00:20:34.410 --> 00:20:37.009 hard mathematics of the bias -variance trade 00:20:37.009 --> 00:20:39.769 -off, we have to ask a difficult question. I 00:20:39.769 --> 00:20:41.230 think I see where you're going with this. How 00:20:41.230 --> 00:20:44.430 often do we criticize human bias in decision 00:20:44.430 --> 00:20:46.809 -making without realizing that mathematically, 00:20:47.170 --> 00:20:50.690 a baseline level of bias might be the exact evolutionary 00:20:50.690 --> 00:20:53.470 feature keeping us functional? Wow. It is the 00:20:53.470 --> 00:20:55.690 biological constraint keeping us from drowning 00:20:55.690 --> 00:20:58.289 in the infinite paralyzing noise of our daily 00:20:58.289 --> 00:21:01.349 lives. It's an evolutionary feature, not a bug. 00:21:01.529 --> 00:21:04.890 The math demands a compromise. Well, thank you 00:21:04.890 --> 00:21:07.630 for joining us on this deep dive. As you go out 00:21:07.630 --> 00:21:09.190 into the world today, whether you are looking 00:21:09.190 --> 00:21:11.609 at the latest AI models or just observing your 00:21:11.609 --> 00:21:13.670 own thought processes, remember perfection is 00:21:13.670 --> 00:21:15.630 mathematically impossible. Don't look for the 00:21:15.630 --> 00:21:18.230 pristine crystal ball. Just look for the signal 00:21:18.230 --> 00:21:19.069 in the noise.