WEBVTT 00:00:00.000 --> 00:00:03.240 Imagine you have this incredible high -performance 00:00:03.240 --> 00:00:07.219 car engine. OK. It runs perfectly. It's revolutionizing 00:00:07.219 --> 00:00:09.699 the entire automotive industry, just breaking 00:00:09.699 --> 00:00:12.179 speed records left and right. But there is a 00:00:12.179 --> 00:00:15.500 major catch. There always is. Right. If you get 00:00:15.500 --> 00:00:17.539 all the world's top mechanics into one room and 00:00:17.539 --> 00:00:19.500 you ask them what the spark plugs are actually 00:00:19.500 --> 00:00:22.019 doing, they violently disagree. Like completely 00:00:22.019 --> 00:00:24.160 at odds with each other. Exactly. They're literally 00:00:24.160 --> 00:00:26.219 shouting at each other across the table. You 00:00:26.219 --> 00:00:28.339 know, it sounds totally absurd when you put it 00:00:28.339 --> 00:00:32.340 that way, but in the realm of hard science, this 00:00:32.340 --> 00:00:36.100 happens far more often than you'd think. We build 00:00:36.100 --> 00:00:39.689 things that work. long before we actually understand 00:00:39.689 --> 00:00:42.189 the mathematics of why they work. And that is 00:00:42.189 --> 00:00:44.509 the exact phenomenon we are looking at today. 00:00:44.829 --> 00:00:47.590 Welcome to a deep dive into the hidden architecture 00:00:47.590 --> 00:00:50.530 of modern artificial intelligence. You handed 00:00:50.530 --> 00:00:53.229 us a massive stack of dense research papers, 00:00:53.509 --> 00:00:55.909 articles spanning the last decade of AI development. 00:00:56.090 --> 00:00:58.299 That's a lot of material, yeah. It is. And our 00:00:58.299 --> 00:01:01.020 goal today is to look through your sources to 00:01:01.020 --> 00:01:03.960 explore this technique introduced back in 2015. 00:01:04.379 --> 00:01:07.579 It's called batch normalization, or batch norm 00:01:07.579 --> 00:01:09.640 for short. Which absolutely revolutionized how 00:01:09.640 --> 00:01:12.659 we build neural networks. Totally. But here we 00:01:12.659 --> 00:01:15.780 are, roughly a decade later, and the top AI experts 00:01:15.780 --> 00:01:18.480 in the world are still arguing over why it actually 00:01:18.480 --> 00:01:20.379 works. And understanding this debate, I mean, 00:01:20.459 --> 00:01:22.799 it isn't just some abstract academic trivia for 00:01:22.799 --> 00:01:25.280 you to listen to. It is a crucial piece of the 00:01:25.280 --> 00:01:28.120 AI puzzle. Batch norm is essentially the engine 00:01:28.120 --> 00:01:30.540 component that makes training artificial intelligence 00:01:30.540 --> 00:01:33.879 vastly faster and way more stable. Right. But 00:01:33.879 --> 00:01:36.459 beyond just the engineering, your sources provide 00:01:36.459 --> 00:01:38.760 this brilliant case study in the scientific method 00:01:38.760 --> 00:01:41.659 itself. It shows us why critical thinking and 00:01:41.659 --> 00:01:44.019 constantly questioning established truths is 00:01:44.019 --> 00:01:46.400 so essential. Even with math. Especially with 00:01:46.400 --> 00:01:49.040 math. Even in fields entirely governed by rigid 00:01:49.040 --> 00:01:52.000 equations, the human theory attempting to explain 00:01:52.000 --> 00:01:54.680 that math can be, well, completely and fundamentally 00:01:54.680 --> 00:01:57.989 wrong. Okay. Let's unpack this. Our mission for 00:01:57.989 --> 00:02:01.069 you today is to demystify the math inside your 00:02:01.069 --> 00:02:04.290 research stack and explore this ultimate scientific 00:02:04.290 --> 00:02:07.030 debate. We're going to trace the journey from 00:02:07.030 --> 00:02:10.469 that original 2015 theory to the massive plot 00:02:10.469 --> 00:02:12.909 twist that completely debunked it. That's my 00:02:12.909 --> 00:02:15.129 favorite part. Oh, it's so good. And finally, 00:02:15.409 --> 00:02:17.770 to the cutting edge math that is trying to explain 00:02:17.770 --> 00:02:20.969 it today. But, to really understand the current 00:02:20.969 --> 00:02:23.969 debate, we first have to understand what batch 00:02:23.969 --> 00:02:26.689 norm was originally designed to fix. Right. So 00:02:26.689 --> 00:02:29.849 let's step back to 2015. Researchers Sergei Of 00:02:29.849 --> 00:02:32.930 and Christian Segedy introduced batch normalization 00:02:32.930 --> 00:02:36.189 to the world. At its core, The technique does 00:02:36.189 --> 00:02:38.710 two primary things to the data inputs at each 00:02:38.710 --> 00:02:41.030 layer of a neural network. Which are? It re -centers 00:02:41.030 --> 00:02:43.349 them around and averages zero, and it re -scales 00:02:43.349 --> 00:02:45.150 them so they all have a standard variance of 00:02:45.150 --> 00:02:47.889 one. Okay, so it's basically organizing and standardizing 00:02:47.889 --> 00:02:50.370 the data mid -flight, but why? Like, why do we 00:02:50.370 --> 00:02:52.090 need to do that? What was the specific villain 00:02:52.090 --> 00:02:54.229 they were trying to fight back then? The villain 00:02:54.229 --> 00:02:56.770 was a phenomenon they called internal covariance 00:02:56.770 --> 00:03:00.400 shift, or ICS. ICS, got it. You see, a neural 00:03:00.400 --> 00:03:03.639 network is made of dozens or even hundreds of 00:03:03.639 --> 00:03:06.520 sequential layers. As the network trains, it 00:03:06.520 --> 00:03:08.639 tweaks the mathematical weights of the earlier 00:03:08.639 --> 00:03:11.560 layers to get better answers. Makes sense. But 00:03:11.560 --> 00:03:13.800 because those early layers are changing, the 00:03:13.800 --> 00:03:15.960 actual distribution of the numbers being passed 00:03:15.960 --> 00:03:18.099 into the deeper layers is constantly shifting 00:03:18.099 --> 00:03:21.340 around. The deeper layers have to, like, continually 00:03:21.340 --> 00:03:25.270 readjust to these new totally unpredictable ranges 00:03:25.270 --> 00:03:28.370 of data. So if I'm, say, layer 50 in a network, 00:03:28.629 --> 00:03:30.930 I might be expecting numbers between 1 and 10. 00:03:31.430 --> 00:03:33.750 But suddenly, layer 49 decides to start sending 00:03:33.750 --> 00:03:36.129 me numbers between negative 1 ,000 and positive 00:03:36.129 --> 00:03:39.169 1 ,000. Exactly. It's chaotic. And especially 00:03:39.169 --> 00:03:41.750 in deep networks, small shifts in the early layers 00:03:41.750 --> 00:03:43.669 get exponentially amplified by the time they 00:03:43.669 --> 00:03:47.349 reach the end. OK, I'm picturing. a bucket brigade 00:03:47.349 --> 00:03:49.789 putting out a fire. Oh, I like this. If every 00:03:49.789 --> 00:03:51.870 person randomly changes how they throw a bucket, 00:03:51.990 --> 00:03:53.689 like one person throws underhand, the next throws 00:03:53.689 --> 00:03:56.750 overhand, the next tucks it sideways, the next 00:03:56.750 --> 00:03:59.030 person in line is constantly fumbling trying 00:03:59.030 --> 00:04:00.750 to catch it. Right. They have to adjust their 00:04:00.750 --> 00:04:03.349 stance every single time. Exactly. So the whole 00:04:03.349 --> 00:04:05.509 system slows to a crawl because nobody knows 00:04:05.509 --> 00:04:09.210 what to expect. So batch norm essentially steps 00:04:09.210 --> 00:04:11.889 in between every single person and forces them 00:04:11.889 --> 00:04:14.590 to throw the bucket the exact same way. Is that 00:04:14.590 --> 00:04:16.610 the right way to think about it? That is a highly 00:04:16.610 --> 00:04:18.930 accurate way to visualize the original theory. 00:04:19.850 --> 00:04:22.329 By standardizing the inputs, forcing everyone 00:04:22.329 --> 00:04:25.329 to throw the bucket the same way, the 2015 paper 00:04:25.329 --> 00:04:28.170 claimed to solve this internal covariate shift. 00:04:28.790 --> 00:04:30.949 And the initial benefits they reported were, 00:04:30.949 --> 00:04:33.589 I mean, undeniably massive. What kind of benefits 00:04:33.589 --> 00:04:35.389 did they see when they flipped the switch on 00:04:35.389 --> 00:04:38.110 this? For one, it allowed developers to use much 00:04:38.110 --> 00:04:40.410 higher learning rates. Meaning the AI learns 00:04:40.410 --> 00:04:43.629 faster. Yes. But normally, if an AI tries to 00:04:43.629 --> 00:04:46.329 learn too quickly, you run into vanishing or 00:04:46.329 --> 00:04:48.550 exploding gradients. Which sounds bad. It is. 00:04:48.810 --> 00:04:50.910 It means the mathematical updates to the network 00:04:50.910 --> 00:04:53.589 either shrink away to zero, completely stopping 00:04:53.589 --> 00:04:56.829 the learning, or they compound and grow uncontrollably 00:04:56.829 --> 00:04:59.449 large, which literally breaks the model and spits 00:04:59.449 --> 00:05:02.209 out error codes. Ah. Batch norm prevented that. 00:05:02.870 --> 00:05:05.819 It also acted as a regularizer. meaning the network 00:05:05.819 --> 00:05:08.259 got better at generalizing to new unseen data. 00:05:08.540 --> 00:05:10.459 It even reduced the need for an older technique 00:05:10.459 --> 00:05:12.699 called dropout. What's dropout? It's where a 00:05:12.699 --> 00:05:15.980 network intentionally drops or forgets random 00:05:15.980 --> 00:05:18.420 pieces of data so it doesn't just blindly memorize 00:05:18.420 --> 00:05:21.060 the training set. BatchNOR meant they didn't 00:05:21.060 --> 00:05:23.459 have to rely on that as much. So on paper, it 00:05:23.459 --> 00:05:26.240 was this miracle cure. It made the network robust, 00:05:26.680 --> 00:05:29.139 adaptable, and far less sensitive to its starting 00:05:29.139 --> 00:05:31.839 settings. That brings us to how it actually does 00:05:31.839 --> 00:05:34.600 this mechanically. How does this specific spark 00:05:34.600 --> 00:05:37.360 plug work? Let's look at the transformation step 00:05:37.360 --> 00:05:39.420 outlined in your sources. When you train a neural 00:05:39.420 --> 00:05:41.740 network, you don't feed it the entire data set 00:05:41.740 --> 00:05:44.379 of millions of images all at once. Right. I'd 00:05:44.379 --> 00:05:46.660 imagine that's computationally impossible. Totally 00:05:46.660 --> 00:05:48.360 impossible. You use what's called a mini -batch, 00:05:48.660 --> 00:05:52.019 so maybe 32 or 64 images at a time. During training, 00:05:52.360 --> 00:05:54.860 the batch norm layer looks at this specific mini 00:05:54.860 --> 00:05:57.699 -batch and calculates its empirical mean and 00:05:57.699 --> 00:06:00.079 variance. Just for those 64 images. Exactly. 00:06:00.220 --> 00:06:02.879 It then normalizes each dimension of the input 00:06:02.879 --> 00:06:05.199 based on those specific mini -batch numbers. 00:06:05.389 --> 00:06:08.430 Re -centering to zero and rescaling to one. Right. 00:06:08.709 --> 00:06:11.290 And there is a crucial mathematical detail here. 00:06:11.870 --> 00:06:14.649 During that division, it adds an arbitrarily 00:06:14.649 --> 00:06:17.790 small positive constant called epsilon to the 00:06:17.790 --> 00:06:20.509 variance. Epsilon. Yeah, this is purely for numerical 00:06:20.509 --> 00:06:23.529 stability to ensure the computer never accidentally 00:06:23.529 --> 00:06:26.910 divides by zero. Because if it did, it would 00:06:26.910 --> 00:06:29.129 completely crash the program. Wait, let me stop 00:06:29.129 --> 00:06:31.509 you there and push back a little on this. If 00:06:31.509 --> 00:06:34.629 we force every single mini -batch to have a perfect 00:06:34.629 --> 00:06:38.389 zero mean and a variance of one, aren't we essentially 00:06:38.389 --> 00:06:41.350 ironing out all the distinct mathematical wrinkles 00:06:41.350 --> 00:06:43.670 of the data? That's a great observation. Because 00:06:43.670 --> 00:06:46.430 doesn't the AI need those specific wrinkles to 00:06:46.430 --> 00:06:49.350 actually learn the difference between, say, a 00:06:49.350 --> 00:06:52.430 picture of a cat and a picture of a dog? If everything 00:06:52.430 --> 00:06:54.930 looks perfectly standardized, it seems like we 00:06:54.930 --> 00:06:57.009 are destroying the very information we are trying 00:06:57.009 --> 00:06:59.490 to process. That is exactly the right question 00:06:59.490 --> 00:07:02.180 to ask. And honestly, It's a trap a lot of early 00:07:02.180 --> 00:07:04.199 developers fell into. You are absolutely right. 00:07:04.300 --> 00:07:07.000 You can't just wipe away the data's unique characteristics 00:07:07.000 --> 00:07:09.660 and expect the network to learn complex patterns. 00:07:09.779 --> 00:07:12.819 Right. It would just be noise. So, to restore 00:07:12.819 --> 00:07:15.899 the representation power of the network, BatchNorm 00:07:15.899 --> 00:07:18.959 introduces two highly specific learned parameters 00:07:18.959 --> 00:07:21.240 for each dimension. They're called gamma and 00:07:21.240 --> 00:07:24.500 beta. Gamma and beta. What do they do? Gamma 00:07:24.500 --> 00:07:26.819 scales the newly normalized value. It basically 00:07:26.819 --> 00:07:28.920 allows the network to stretch the variance back 00:07:28.920 --> 00:07:31.720 out if it needs to, and beta shifts the mean 00:07:31.720 --> 00:07:34.459 away from zero. So it puts the wrinkles back 00:07:34.459 --> 00:07:38.560 in. Yes, but crucially, these aren't fixed numbers. 00:07:39.060 --> 00:07:41.120 They are learned by the network itself during 00:07:41.120 --> 00:07:44.089 the optimization process. The network gets to 00:07:44.089 --> 00:07:46.529 look at the standardized data and decide exactly 00:07:46.529 --> 00:07:49.149 how much rescaling and reshifting is actually 00:07:49.149 --> 00:07:51.569 useful for the specific task it's trying to accomplish. 00:07:52.050 --> 00:07:54.550 OK, I see. So it normalizes everything to a completely 00:07:54.550 --> 00:07:56.709 blank slate, so the network isn't overwhelmed. 00:07:56.850 --> 00:07:59.410 But then it uses gamma and beta to paint some 00:07:59.410 --> 00:08:01.490 of that necessary flavor back onto the slate. 00:08:01.490 --> 00:08:04.430 Right. But in a highly controlled, predictable 00:08:04.430 --> 00:08:07.519 way. Exactly. And if we connect this to the bigger 00:08:07.519 --> 00:08:09.399 picture, we also have to distinguish between 00:08:09.399 --> 00:08:12.319 how the AI trains in the lab and how it operates 00:08:12.319 --> 00:08:14.319 in the real world. The real world being what 00:08:14.319 --> 00:08:17.519 you call the inference stage. Right. During training, 00:08:17.600 --> 00:08:20.160 as we discussed, it's learning on the fly using 00:08:20.160 --> 00:08:22.579 the statistics of those random mini -batches. 00:08:22.740 --> 00:08:25.089 But during inference, say, When you deploy the 00:08:25.089 --> 00:08:28.569 AI to a user's phone to recognize a photo, depending 00:08:28.569 --> 00:08:30.750 on the random statistics of whatever else happens 00:08:30.750 --> 00:08:33.230 to be in a mini batch isn't useful. Right, because 00:08:33.230 --> 00:08:35.509 you just want an answer for one photo, not a 00:08:35.509 --> 00:08:38.470 whole batch of random stuff. Exactly. So it switches 00:08:38.470 --> 00:08:41.289 gears, it stops calculating the mean and variance 00:08:41.289 --> 00:08:45.149 on the fly, and instead uses fixed global population 00:08:45.149 --> 00:08:48.029 statistics. Like a running average. Yes, a running 00:08:48.029 --> 00:08:50.330 average of the overall mean and variance that 00:08:50.330 --> 00:08:52.769 it carefully calculated during the entire training 00:08:52.769 --> 00:08:55.620 process. This makes the output deterministic. 00:08:56.279 --> 00:08:58.320 When you give it a specific input in the real 00:08:58.320 --> 00:09:01.240 world, you get a reliable, consistent output 00:09:01.240 --> 00:09:03.919 entirely independent of any other data. Wow. 00:09:04.100 --> 00:09:06.340 The mechanics really do seem elegant, like it 00:09:06.340 --> 00:09:09.100 makes logical sense. They found a specific problem, 00:09:09.480 --> 00:09:11.899 internal covariate shift, and they built a brilliant 00:09:11.899 --> 00:09:14.120 mathematical machine to fix it. It was brilliant. 00:09:14.379 --> 00:09:16.639 But here's where it gets really interesting in 00:09:16.639 --> 00:09:19.320 the source material. Because a few years after 00:09:19.320 --> 00:09:22.980 this 2015 paper, the story takes a sharp, highly 00:09:22.980 --> 00:09:26.860 unexpected turn. A massive turn. As more researchers 00:09:26.860 --> 00:09:28.879 started poking at the mathematical foundation 00:09:28.879 --> 00:09:31.580 of bash normalization, they discovered something 00:09:31.580 --> 00:09:34.889 shocking. The original 2015 paper was likely 00:09:34.889 --> 00:09:37.250 completely wrong about the reason for its own 00:09:37.250 --> 00:09:40.090 success. So the spark plugs are making the car 00:09:40.090 --> 00:09:42.509 go faster, but definitely not for the reason 00:09:42.509 --> 00:09:45.610 the mechanic claimed they were. Precisely. Let 00:09:45.610 --> 00:09:47.590 me walk you through the primary studies in your 00:09:47.590 --> 00:09:50.110 stack that debunked the internal covariate shift 00:09:50.110 --> 00:09:53.929 theory. Researchers from MIT took a very standard, 00:09:54.389 --> 00:09:56.929 widely used neural network architecture, known 00:09:56.929 --> 00:10:00.639 as VGG16, and ran an experiment with three different 00:10:00.639 --> 00:10:03.480 models. Okay, setting up a race. Exactly. Model 00:10:03.480 --> 00:10:06.340 one was standard, no batch norm at all. Model 00:10:06.340 --> 00:10:08.940 two had standard batch norm applied, and model 00:10:08.940 --> 00:10:11.600 three was the wild card. It had batch norm, but 00:10:11.600 --> 00:10:13.860 the researchers intentionally injected random 00:10:13.860 --> 00:10:16.340 noise into the data after the batch norm layer, 00:10:16.500 --> 00:10:19.059 but before the next active layer. Wait, if they 00:10:19.059 --> 00:10:22.320 injected random noise with crazy means and unpredictable 00:10:22.320 --> 00:10:24.240 variances, the deeper layer should have been 00:10:24.240 --> 00:10:26.320 completely blinded. Right. It's like shaking 00:10:26.320 --> 00:10:28.320 the fire brigade's ladders while they're trying 00:10:28.320 --> 00:10:29.980 to pass the buckets. The network should have 00:10:29.980 --> 00:10:32.740 completely collapsed. That's exactly what foundational 00:10:32.740 --> 00:10:35.799 theory suggested should happen. They explicitly, 00:10:36.200 --> 00:10:39.460 purposely, cause massive internal covariate shift. 00:10:39.309 --> 00:10:42.250 They did the exact thing BatchNorm was supposed 00:10:42.250 --> 00:10:45.190 to be preventing. And yet... Don't tell me it 00:10:45.190 --> 00:10:47.470 worked. The noisy model performed just as well 00:10:47.470 --> 00:10:49.690 as the standard BatchNorm model. You're kidding. 00:10:49.889 --> 00:10:53.830 And crucially, both of them vastly outperformed 00:10:53.830 --> 00:10:56.649 the network that had no BatchNorm at all. Okay, 00:10:56.690 --> 00:11:00.889 so if IFA and Segody built a tool to fix a specific 00:11:00.889 --> 00:11:03.659 leak in the plumbing... But researchers proved 00:11:03.659 --> 00:11:06.360 the tool still works perfectly even when you 00:11:06.360 --> 00:11:08.200 intentionally blast a hole in that exact same 00:11:08.200 --> 00:11:11.460 pipe. What is the tool actually doing? Because 00:11:11.460 --> 00:11:13.820 clearly fixing that leak isn't the reason the 00:11:13.820 --> 00:11:15.759 house isn't flooding. And the evidence against 00:11:15.759 --> 00:11:18.039 the original theory just kept piling up. Your 00:11:18.039 --> 00:11:19.860 sources highlight another pivotal study called 00:11:19.860 --> 00:11:23.240 the measure experiment. These researchers decided 00:11:23.240 --> 00:11:25.679 to mathematically define what internal covariate 00:11:25.679 --> 00:11:27.220 shift actually looks like. How do you measure 00:11:27.220 --> 00:11:29.539 that? They did this by measuring the correlation 00:11:29.539 --> 00:11:32.120 of gradients, basically calculating how much 00:11:32.120 --> 00:11:34.980 the learning direction for a layer shifts around 00:11:34.980 --> 00:11:37.100 when the previous layers update their weights. 00:11:37.820 --> 00:11:40.399 If the shift is very small, the correlation is 00:11:40.399 --> 00:11:44.139 close to one. If it's chaotic, it drops. They 00:11:44.139 --> 00:11:46.740 then compared standard networks to batch norm 00:11:46.740 --> 00:11:50.120 networks. Let me guess. The networks using batch 00:11:50.120 --> 00:11:52.639 norm didn't actually have a correlation closer 00:11:52.639 --> 00:11:55.059 to one. They weren't actually reducing the shift. 00:11:55.360 --> 00:11:57.879 It was even more contradictory than that. The 00:11:57.879 --> 00:11:59.879 standard networks actually had higher gradient 00:11:59.879 --> 00:12:01.799 correlations than the networks using batch norm. 00:12:02.360 --> 00:12:04.720 Batch norm was technically causing more internal 00:12:04.720 --> 00:12:07.379 shift by that specific metric. That is wild. 00:12:07.559 --> 00:12:10.200 And to add insult to injury, using batch norm 00:12:10.200 --> 00:12:13.279 causes a major statistical headache. Because 00:12:13.279 --> 00:12:15.419 you are normalizing a batch based on its own 00:12:15.419 --> 00:12:18.360 internal mean, the value of image A suddenly 00:12:18.360 --> 00:12:20.480 depends mathematically on the values of images 00:12:20.480 --> 00:12:23.539 B, C, and D in that same batch. Meaning they 00:12:23.539 --> 00:12:26.899 are no longer independent. Exactly. Mathematically, 00:12:27.519 --> 00:12:30.000 the items are no longer independent and identically 00:12:30.000 --> 00:12:33.460 distributed, or EAD, as we say. In the realm 00:12:33.460 --> 00:12:37.759 of statistics, violating the EAD assumption usually 00:12:37.759 --> 00:12:40.820 degrades the quality of your gradient estimation. 00:12:41.120 --> 00:12:43.120 Which should make it worse. Theoretically, it 00:12:43.120 --> 00:12:45.720 should make training significantly harder. Not 00:12:45.720 --> 00:12:48.559 easier. This is insane. The tool works absolute 00:12:48.559 --> 00:12:51.639 miracles for speed and stability, but the blueprint 00:12:51.639 --> 00:12:55.100 explaining why is just entirely backwards. It's 00:12:55.100 --> 00:12:57.720 causing more shift, it's tangling the data dependencies, 00:12:57.899 --> 00:13:00.740 and yet it still wins. This raises a profoundly 00:13:00.740 --> 00:13:02.279 important question, and it really highlights 00:13:02.279 --> 00:13:04.299 the beauty of the scientific method at work in 00:13:04.299 --> 00:13:07.059 these papers. Just because the empirical result 00:13:07.059 --> 00:13:09.840 of an AI model is stellar -like, just because 00:13:09.840 --> 00:13:12.509 it works flawlessly in the real world, doesn't 00:13:12.509 --> 00:13:14.490 mean the foundational human theory behind it 00:13:14.490 --> 00:13:16.649 was correct. We just made up a good story. We 00:13:16.649 --> 00:13:19.009 have to keep measuring, testing, and being willing 00:13:19.009 --> 00:13:21.330 to throw out our most cherished assumptions when 00:13:21.330 --> 00:13:24.909 the math doesn't align. OK, so the internal covariate 00:13:24.909 --> 00:13:27.450 shift theory is effectively dead in the water. 00:13:27.870 --> 00:13:30.470 But the network is still learning faster. Something 00:13:30.470 --> 00:13:34.049 real and physical is happening. If it's not ICS, 00:13:34.129 --> 00:13:36.250 what are the new theories replacing it in the 00:13:36.250 --> 00:13:38.750 literature? The leading alternative explanation 00:13:38.750 --> 00:13:42.870 introduces the concept of smoothness. Researchers 00:13:42.870 --> 00:13:45.870 mapped out the optimization landscape, the mathematical 00:13:45.870 --> 00:13:48.250 terrain the AI has to navigate to find the right 00:13:48.250 --> 00:13:51.490 answers, and found that batch normalization actually 00:13:51.490 --> 00:13:54.409 produces a much smoother parameter space and 00:13:54.409 --> 00:13:56.750 smoother gradients. What exactly do we mean by 00:13:56.750 --> 00:13:58.649 a smoother gradient? Like I'm picturing a rolling 00:13:58.649 --> 00:14:01.210 hill instead of a jagged mountain range. That's 00:14:01.210 --> 00:14:03.909 a perfect visualization. In calculus, you can 00:14:03.909 --> 00:14:06.610 measure how jagged or smooth a mathematical landscape 00:14:06.610 --> 00:14:08.610 is using something called the Lipschitz constant. 00:14:08.730 --> 00:14:10.649 The Lipschitz constant. Right. It essentially 00:14:10.649 --> 00:14:13.169 puts a strict mathematical speed limit on how 00:14:13.169 --> 00:14:15.970 fast the slope of the landscape can change. Batch 00:14:15.970 --> 00:14:18.509 norm creates a much smaller Lipschitz constant. 00:14:18.789 --> 00:14:21.629 Meaning less jagged. The gradient magnitude is 00:14:21.629 --> 00:14:24.590 strictly bounded. Because the landscape is less 00:14:24.590 --> 00:14:27.250 jagged, the mathematical guide the network follows 00:14:27.250 --> 00:14:30.330 to improve is less chaotic. The gradients become 00:14:30.330 --> 00:14:34.000 highly predictive. Furthermore, the loss function's 00:14:34.000 --> 00:14:36.720 Hessian, which is this complex matrix used to 00:14:36.720 --> 00:14:38.539 measure the overall curvature of the landscape, 00:14:39.080 --> 00:14:41.080 becomes incredibly resilient to the variance 00:14:41.080 --> 00:14:44.700 of those random mini -batches. Okay, so by standardizing 00:14:44.700 --> 00:14:47.639 the inputs, it accidentally irons out the mathematical 00:14:47.639 --> 00:14:50.720 wrinkles of the landscape itself, making it incredibly 00:14:50.720 --> 00:14:53.220 easy for the AI to just hike down the hill and 00:14:53.220 --> 00:14:55.440 find the optimal solution at the bottom. Exactly. 00:14:55.639 --> 00:14:58.179 But knowing how AI research goes, I feel like 00:14:58.179 --> 00:15:00.980 there has to be a catch. Why isn't smoothness 00:15:00.980 --> 00:15:03.679 just the end of the story? Because there is a 00:15:03.679 --> 00:15:05.879 massive contradiction we have to introduce here. 00:15:06.220 --> 00:15:08.480 It's known as the paradox of gradient explosions. 00:15:08.679 --> 00:15:10.559 I knew it couldn't be a perfectly paved trail. 00:15:10.720 --> 00:15:13.460 Far from it. While a single batch norm layer 00:15:13.460 --> 00:15:16.159 smooths things out locally, when you start stacking 00:15:16.159 --> 00:15:18.679 them in very deep networks, which is exactly 00:15:18.679 --> 00:15:21.940 what all modern AI uses, it actually causes severe 00:15:21.940 --> 00:15:24.100 gradient explosion at the moment of initialization. 00:15:24.419 --> 00:15:26.419 Meaning the math just blows up before it even 00:15:26.419 --> 00:15:29.000 starts learning. Why does stacking them do that? 00:15:29.210 --> 00:15:32.250 It comes down to how the layers compound. Most 00:15:32.250 --> 00:15:34.590 modern networks use an activation function called 00:15:34.590 --> 00:15:37.629 RELLU. Which stands for? Rectified Linear Unit. 00:15:38.190 --> 00:15:40.889 A RELU activation is simply a mathematical filter 00:15:40.889 --> 00:15:43.610 that says if a number is negative, make it zero. 00:15:43.850 --> 00:15:46.330 If it's positive, leave it alone. Okay, simple 00:15:46.330 --> 00:15:48.190 enough. But when you run the math on infinite 00:15:48.190 --> 00:15:50.629 batch sizes through random initial weights, using 00:15:50.629 --> 00:15:53.750 RELLU because it zeroes out half the data, the 00:15:53.750 --> 00:15:56.279 network's variance doesn't stay neutral. The 00:15:56.279 --> 00:15:58.840 math of that compensation settles at a constant 00:15:58.840 --> 00:16:03.679 lambda value of roughly 1 .467. And 1 .467 is 00:16:03.679 --> 00:16:06.580 greater than 1. Exactly. The gradient of the 00:16:06.580 --> 00:16:08.659 first layer weights grows mathematically at a 00:16:08.659 --> 00:16:11.379 rate greater than some constant times. That lambda 00:16:11.379 --> 00:16:15.080 value of 1 .467 raised to the power of L, where 00:16:15.080 --> 00:16:17.080 L is the number of layers in the network. Oh, 00:16:17.080 --> 00:16:20.120 boy. Because 1 .467 is greater than 1, when you 00:16:20.120 --> 00:16:22.159 raise it to the power of 100 or 1 ,000 layers, 00:16:22.480 --> 00:16:25.100 the numbers compound exponentially. Practically, 00:16:25.320 --> 00:16:27.980 this means deep batch norm networks are completely 00:16:27.980 --> 00:16:30.399 untrainable on their own. The math literally 00:16:30.399 --> 00:16:32.830 explodes on step one. So to go back to my hiking 00:16:32.830 --> 00:16:36.110 trail analogy, smoothness means the path itself 00:16:36.110 --> 00:16:39.210 is beautifully paved, but the gradient explosion 00:16:39.210 --> 00:16:42.330 means the very first step of the trail drops 00:16:42.330 --> 00:16:46.269 off a massive jagged cliff. If you take one step, 00:16:46.269 --> 00:16:49.220 you fall to your mathematical death. So how do 00:16:49.220 --> 00:16:51.899 we even use this? How do we survive the drop 00:16:51.899 --> 00:16:54.200 to get to the paved path? We have to build a 00:16:54.200 --> 00:16:56.179 bridge. We use a separate architectural technique 00:16:56.179 --> 00:16:59.039 called skip connection. Skip connection. Yes. 00:16:59.259 --> 00:17:01.539 Heavily utilized in things like residual networks 00:17:01.539 --> 00:17:04.519 or resnets. These skip connections literally 00:17:04.519 --> 00:17:07.599 bypass layers, passing data cleanly over the 00:17:07.599 --> 00:17:10.680 volatile sections. They act as a bridge to get 00:17:10.680 --> 00:17:12.940 onto that beautifully paved path safely. Oh, 00:17:12.940 --> 00:17:16.299 that's clever. It resolves the paradox. A single 00:17:16.299 --> 00:17:18.759 batch norm smooths the landscape, but stacking 00:17:18.759 --> 00:17:21.599 them creates an explosive cliff, so we use skip 00:17:21.599 --> 00:17:24.039 connections to survive the drop. Okay, that makes 00:17:24.039 --> 00:17:27.119 sense. Surviving the cliff gets us onto the smooth 00:17:27.119 --> 00:17:29.859 path, and the smooth path lets us finish the 00:17:29.859 --> 00:17:33.299 hike. But, if I'm reasoning this out, that just 00:17:33.299 --> 00:17:35.559 explains how we are able to finish the training 00:17:35.559 --> 00:17:38.519 without crashing. It doesn't really explain why 00:17:38.519 --> 00:17:41.339 batch norm makes the AI practically sprint to 00:17:41.339 --> 00:17:44.569 the finish line. Smoothness doesn't automatically 00:17:44.569 --> 00:17:47.470 equal speed. You're exactly right. How do we 00:17:47.470 --> 00:17:50.369 explain the sheer velocity of learning? You've 00:17:50.369 --> 00:17:52.509 hit on the exact limitation of the smoothness 00:17:52.509 --> 00:17:55.009 theory. To explain the sheer speed of convergence, 00:17:55.309 --> 00:17:57.769 we have to turn to the absolute newest mathematical 00:17:57.769 --> 00:18:00.549 theory in your stack. It's called length -direction 00:18:00.549 --> 00:18:03.410 decoupling. Decoupling. Separating two things 00:18:03.410 --> 00:18:05.509 that are usually tangled together. Precisely. 00:18:06.069 --> 00:18:08.450 You can interpret batch norm mathematically as 00:18:08.450 --> 00:18:10.529 a fundamental reparametrization of the weight 00:18:10.529 --> 00:18:13.200 space itself. It takes the weight vectors of 00:18:13.200 --> 00:18:15.279 the network, the mathematical arrows pointing 00:18:15.279 --> 00:18:17.880 toward the solution, and completely separates 00:18:17.880 --> 00:18:19.799 their length from their direction. So what does 00:18:19.799 --> 00:18:21.579 this all mean? Let's go back to the car engine. 00:18:21.900 --> 00:18:25.119 If I'm driving, is decoupling, completely separating 00:18:25.119 --> 00:18:27.680 the steering wheel, which handles the direction, 00:18:28.380 --> 00:18:30.920 from the gas pedal, which handles the length 00:18:30.920 --> 00:18:33.089 or the speed. That's a great way to look at it. 00:18:33.289 --> 00:18:36.630 So by separating them, the AI can learn to navigate 00:18:36.630 --> 00:18:39.710 tight corners without constantly stalling out 00:18:39.710 --> 00:18:42.829 or accelerating uncontrollably. That is an excellent 00:18:42.829 --> 00:18:45.529 analogy. In traditional gradient descent, the 00:18:45.529 --> 00:18:48.069 length and direction are completely tangled together. 00:18:48.750 --> 00:18:51.230 Changing one inherently affects the other. making 00:18:51.230 --> 00:18:53.670 the optimization problem incredibly complex. 00:18:53.809 --> 00:18:55.789 You try to skier, and the car accidentally speeds 00:18:55.789 --> 00:18:57.750 up. Which would be terrifying. By decoupling 00:18:57.750 --> 00:19:01.109 them, batch norm breaks a massively complex mathematical 00:19:01.109 --> 00:19:04.329 problem into two independent simpler pieces. 00:19:04.710 --> 00:19:06.730 And what's fascinating here is that the math 00:19:06.730 --> 00:19:10.009 proves this decoupling directly leads to linear 00:19:10.009 --> 00:19:12.349 convergence. Linear convergence. Break that down 00:19:12.349 --> 00:19:14.609 for me. As opposed to what? As opposed to sub 00:19:14.609 --> 00:19:17.210 -linear convergence. In standard gradient descent, 00:19:17.869 --> 00:19:19.950 convergence is often sub -linear, meaning the 00:19:19.950 --> 00:19:22.700 AI gets slower and slower and takes smaller and 00:19:22.700 --> 00:19:24.859 smaller steps as it gets closer to the target. 00:19:25.059 --> 00:19:27.220 It drags on. Like it's tiptoeing to the finish 00:19:27.220 --> 00:19:29.920 line. Right. Linear convergence means the AI 00:19:29.920 --> 00:19:32.819 reduces its error by a consistent proportional 00:19:32.819 --> 00:19:35.900 fraction every single step. It doesn't slow down. 00:19:36.079 --> 00:19:38.500 It is mathematically guaranteed to find the solution 00:19:38.500 --> 00:19:41.180 at a much, much faster rate. And the researchers 00:19:41.180 --> 00:19:43.059 have actually proven this mathematically. This 00:19:43.059 --> 00:19:45.299 isn't just an observation or a guess like the 00:19:45.299 --> 00:19:48.000 2015 paper. Yes, this has been rigorously proven, 00:19:48.200 --> 00:19:50.400 though currently for highly specific scenarios. 00:19:51.000 --> 00:19:53.319 For instance, researchers applied batch norm 00:19:53.319 --> 00:19:55.900 to the ordinary least square problem. What's 00:19:55.900 --> 00:19:58.559 that? It's a classic mathematical model for fitting 00:19:58.559 --> 00:20:01.759 a line to data points. By separating the weights, 00:20:02.180 --> 00:20:04.059 the objective function fundamentally changes 00:20:04.059 --> 00:20:06.359 its shape. It takes the form of what we call 00:20:06.359 --> 00:20:09.740 a generalized Rayleigh quotient. A generalized 00:20:09.740 --> 00:20:11.519 Rayleigh quotient? That sounds intimidating. 00:20:11.940 --> 00:20:13.779 What is that actually doing? Think of it as a 00:20:13.779 --> 00:20:16.299 mathematical speed limit. It's a specific type 00:20:16.299 --> 00:20:18.859 of ratio that compares the variance of the network's 00:20:18.859 --> 00:20:21.539 direction to its length. By framing the problem 00:20:21.539 --> 00:20:23.960 as this ratio, mathematicians could calculate 00:20:23.960 --> 00:20:27.420 the exact eigenvalues. Think of them as structural 00:20:27.420 --> 00:20:29.640 stress points of the matrix. Okay, stress points. 00:20:29.859 --> 00:20:32.400 And they proved that starting from almost any 00:20:32.400 --> 00:20:35.059 point on the map, the AI takes a perfectly direct 00:20:35.059 --> 00:20:38.569 path and converges linearly. They've also proven 00:20:38.569 --> 00:20:40.369 it for what's called the learning half spaces 00:20:40.369 --> 00:20:43.769 problem. Learning half spaces. Which means what 00:20:43.769 --> 00:20:46.369 in the context of an AI? It refers to training 00:20:46.369 --> 00:20:48.589 a perceptron, which is essentially the simplest, 00:20:48.589 --> 00:20:50.950 most fundamental form of a neural network. Just 00:20:50.950 --> 00:20:53.490 a single node making a binary decision. Oh, okay. 00:20:53.609 --> 00:20:56.750 Like a basic yes or no. Exactly. Assuming a Gaussian 00:20:56.750 --> 00:20:59.329 input, meaning the data roughly follows a standard 00:20:59.329 --> 00:21:02.049 bell curve, distribution researchers designed 00:21:02.049 --> 00:21:04.289 a variation of traditional gradient descent called 00:21:04.289 --> 00:21:07.809 GDNP. or gradient descent in normalized parameterization. 00:21:08.390 --> 00:21:10.670 And I'm assuming GDMP relies on this decoupling. 00:21:10.809 --> 00:21:13.309 It separates the updates. Exactly. It completely 00:21:13.309 --> 00:21:15.769 isolates the steps. Going back to your car metaphor, 00:21:16.609 --> 00:21:19.089 in GDMP, for every single step, the algorithm 00:21:19.089 --> 00:21:22.089 calculates the steering the direction first using 00:21:22.089 --> 00:21:24.369 standard gradients. It locks the steering wheel 00:21:24.369 --> 00:21:27.890 in place. Then it uses a separate classical mathematical 00:21:27.890 --> 00:21:30.349 tool called a bisection algorithm to figure out 00:21:30.349 --> 00:21:33.289 exactly how hard to press the gas, the length, 00:21:33.490 --> 00:21:37.759 steer, then gas. steer then gas. Wow. So by separating 00:21:37.759 --> 00:21:40.359 the mechanisms they guarantee progress. Yes. 00:21:40.819 --> 00:21:43.220 They formally proved that the partial derivative 00:21:43.220 --> 00:21:45.420 against the length component converges to zero 00:21:45.420 --> 00:21:48.279 at a linear rate and the norm of the gradient 00:21:48.279 --> 00:21:51.019 with respect to the direction also converges 00:21:51.019 --> 00:21:53.920 linearly. That's incredible. And here's the kicker. 00:21:54.359 --> 00:21:56.799 Even though the rigorous mathematical proof relied 00:21:56.799 --> 00:22:00.289 on that neat bell curve input assumption, empirical 00:22:00.289 --> 00:22:03.829 experiments showed GDNP wildly accelerates optimization 00:22:03.829 --> 00:22:08.210 even on messy real -world data without that constraint. 00:22:08.410 --> 00:22:11.190 So it works outside the lab too. They even extended 00:22:11.190 --> 00:22:13.549 this mathematical proof to a slightly more complex 00:22:13.549 --> 00:22:16.210 network with a hidden layer, showing that optimizing 00:22:16.210 --> 00:22:18.970 each piece alternately still yields that blazing 00:22:18.970 --> 00:22:22.400 fast linear convergence. So by Breaking the problem 00:22:22.400 --> 00:22:24.500 apart, steering wheel locked in one hand, gas 00:22:24.500 --> 00:22:26.880 pedal isolated in the other, the math fundamentally 00:22:26.880 --> 00:22:29.259 guarantees a faster arrival at the destination. 00:22:29.599 --> 00:22:32.279 We've really gone on a wild ride today. We started 00:22:32.279 --> 00:22:35.759 with a 2015 fix for an unproven problem, moved 00:22:35.759 --> 00:22:38.160 through the mechanics of tweaking gamma and beta, 00:22:38.700 --> 00:22:41.140 discovered that it accidentally smooths the landscape 00:22:41.140 --> 00:22:44.140 but creates explosive mathematical cliffs, to 00:22:44.140 --> 00:22:46.519 finally arriving at a profoundly elegant tool 00:22:46.519 --> 00:22:48.680 that completely decouples learning itself. It 00:22:48.680 --> 00:22:51.359 is quite the journey. and it serves as a powerful 00:22:51.359 --> 00:22:54.640 reminder for anyone working in technology. Even 00:22:54.640 --> 00:22:56.619 the brilliant architects of our most advanced 00:22:56.619 --> 00:22:59.619 AI tools don't always fully understand why their 00:22:59.619 --> 00:23:01.930 creations work. They just know that they do. 00:23:02.150 --> 00:23:04.250 Right. They build based on intuition, they observe 00:23:04.250 --> 00:23:06.269 the astonishing results, they measure the anomalies 00:23:06.269 --> 00:23:08.549 when things break, and they vigorously debate 00:23:08.549 --> 00:23:11.490 the underlying reality for years. To you listening 00:23:11.490 --> 00:23:13.730 right now, thank you for sending in these sources 00:23:13.730 --> 00:23:16.289 and coming along on this dense but incredibly 00:23:16.289 --> 00:23:18.630 rewarding journey through the AI plumbing. We 00:23:18.630 --> 00:23:21.009 look at these modern AI systems generating text, 00:23:21.150 --> 00:23:23.849 creating photorealistic art, folding complex 00:23:23.849 --> 00:23:25.910 proteins, and it feels like pure magic. It really 00:23:25.910 --> 00:23:28.210 does. But underneath that magic, it's a messy, 00:23:28.450 --> 00:23:31.109 evolving world of mathematics. mechanics constantly 00:23:31.109 --> 00:23:33.369 arguing over what the spark plugs are actually 00:23:33.369 --> 00:23:35.589 doing. I would encourage you to critically evaluate 00:23:35.589 --> 00:23:38.430 the black box tools and systems you rely on daily. 00:23:39.170 --> 00:23:42.029 Consider that the accepted why in any field, 00:23:42.289 --> 00:23:44.730 even the hardest of hard sciences, might just 00:23:44.730 --> 00:23:47.289 be a temporary placeholder. It might just be 00:23:47.289 --> 00:23:50.430 the best story we have until a better, more rigorous 00:23:50.430 --> 00:23:52.450 mathematical proof comes along to replace it. 00:23:52.589 --> 00:23:55.170 Which leaves us with a truly mind -bending question 00:23:55.170 --> 00:23:58.460 to ponder as we wrap up this deep dive. If human 00:23:58.460 --> 00:24:01.279 engineers can build highly effective, world -changing 00:24:01.279 --> 00:24:04.440 AI systems based on misunderstood or even entirely 00:24:04.440 --> 00:24:07.380 flawed foundational theories, what happens when 00:24:07.380 --> 00:24:09.839 AI itself starts designing its own algorithms? 00:24:10.039 --> 00:24:12.059 Will we even be capable of understanding the 00:24:12.059 --> 00:24:14.019 underlying math, or will we simply have to trust 00:24:14.019 --> 00:24:16.339 the results? Because, like batch normalization 00:24:16.339 --> 00:24:17.740 in 2015, they just work.