WEBVTT 00:00:00.000 --> 00:00:02.240 You know, an AI doesn't actually know what a 00:00:02.240 --> 00:00:04.519 cat is. No, not at all. And it definitely doesn't 00:00:04.519 --> 00:00:06.459 know what the English language is. I mean, to 00:00:06.459 --> 00:00:08.660 an algorithm, the entire universe is just this 00:00:08.660 --> 00:00:11.220 massive cold string of zeros and ones. Right, 00:00:11.359 --> 00:00:14.199 it has no inherent concept of our physical reality. 00:00:14.410 --> 00:00:17.949 So when an AI makes a prediction and, you know, 00:00:18.070 --> 00:00:21.550 gets it completely wrong, how does it physically 00:00:21.550 --> 00:00:23.870 feel pain? Like, how does it know it needs to 00:00:23.870 --> 00:00:25.890 improve when it doesn't even know what reality 00:00:25.890 --> 00:00:28.210 is in the first place? Well, the answer to that 00:00:28.210 --> 00:00:31.170 is this ruthless mathematical penalty box called 00:00:31.170 --> 00:00:34.409 cross entropy. Yes, exactly. And that brings 00:00:34.409 --> 00:00:37.409 us to the mission of today's deep dive. Welcome, 00:00:37.570 --> 00:00:39.619 everyone. Glad to be here for this one. So we 00:00:39.619 --> 00:00:41.539 have this source material in front of us, and 00:00:41.539 --> 00:00:45.020 it's a rather dense, highly mathematical Wikipedia 00:00:45.020 --> 00:00:48.840 article on cross entropy. And if you, our listener, 00:00:49.039 --> 00:00:51.039 were to just pull up the math on this source 00:00:51.039 --> 00:00:54.219 text, your eyes might immediately glaze over. 00:00:54.280 --> 00:00:56.799 Oh, absolutely. It's just a wall of Greek letters 00:00:56.799 --> 00:00:59.079 and summation symbols. Yeah, it's intimidating. 00:00:59.560 --> 00:01:02.299 But our mission today is to completely demystify 00:01:02.299 --> 00:01:05.159 this. We're going to strip away all that heavy 00:01:05.159 --> 00:01:08.400 notation and show you how this single mathematical 00:01:08.489 --> 00:01:12.590 concept secretly powers basically all the machine 00:01:12.590 --> 00:01:15.150 learning and AI tools you use every single day. 00:01:16.069 --> 00:01:18.290 We are essentially going to look under the hood 00:01:18.290 --> 00:01:21.549 at the unseen ruler. that measures the distance 00:01:21.549 --> 00:01:24.329 between a machine's hallucination and actual 00:01:24.329 --> 00:01:27.209 reality. OK, let's unpack this. Because to really 00:01:27.209 --> 00:01:29.250 understand cross entropy, we actually have to 00:01:29.250 --> 00:01:31.269 step away from modern artificial intelligence 00:01:31.269 --> 00:01:32.909 for a second. Right. We have to go back to the 00:01:32.909 --> 00:01:35.829 roots. Exactly. We need to step back into the 00:01:35.829 --> 00:01:38.849 realm of classical information theory. Because 00:01:38.849 --> 00:01:41.870 long before this equation was training massive 00:01:41.870 --> 00:01:44.269 neural networks, it was simply a way to measure 00:01:44.269 --> 00:01:47.260 the transmission of information. And that is 00:01:47.260 --> 00:01:49.640 the perfect starting point. Because in information 00:01:49.640 --> 00:01:51.540 theory, you're dealing with messages, events, 00:01:51.799 --> 00:01:53.819 and crucially, the probability of those events 00:01:53.819 --> 00:01:56.299 happening. So the formal definition from our 00:01:56.299 --> 00:01:58.340 source states that the cross entropy between 00:01:58.340 --> 00:02:01.340 two probability distributions, and let's just 00:02:01.340 --> 00:02:03.859 call them distribution P and distribution Q. 00:02:03.900 --> 00:02:07.079 OK, P and Q. Right. It measures the average number 00:02:07.079 --> 00:02:10.419 of bits you need to identify an event. But, and 00:02:10.419 --> 00:02:12.879 here's the crucial catch, it's the number of 00:02:12.879 --> 00:02:15.879 bits needed when your coding scheme is optimized 00:02:15.879 --> 00:02:18.599 for an estimated probability distribution, which 00:02:18.599 --> 00:02:21.840 we call Q. Oh, OK. Rather than the true distribution, 00:02:21.960 --> 00:02:24.759 which is P. Exactly. Let me make sure I'm really 00:02:24.759 --> 00:02:26.840 tracking this. So we have a true reality, which 00:02:26.840 --> 00:02:30.750 is P. And we have our estimated guess of that 00:02:30.750 --> 00:02:33.490 reality, which is Q. You've got it. And looking 00:02:33.490 --> 00:02:35.490 at the Wikipedia source, it mentions something 00:02:35.490 --> 00:02:38.370 called the Kraft -McMillan theorem here. It says 00:02:38.370 --> 00:02:41.310 that any efficient way of coding a message inherently 00:02:41.310 --> 00:02:44.689 implies a certain probability distribution. So 00:02:44.689 --> 00:02:46.169 shorter codes are for things that happen all 00:02:46.169 --> 00:02:48.729 the time, and longer codes are for rare things. 00:02:48.830 --> 00:02:50.849 That's the core idea, yes. I'm trying to visualize 00:02:50.849 --> 00:02:54.069 this. Is it kind of like Morse code? Actually, 00:02:54.129 --> 00:02:56.949 Morse code is a brilliant example of that exact 00:02:56.949 --> 00:02:59.699 theorem. Oh, really? Yeah. Think about it. In 00:02:59.699 --> 00:03:03.039 English, the letter E is incredibly common. So 00:03:03.039 --> 00:03:05.139 in Morse code, it's literally just a single dot. 00:03:05.560 --> 00:03:07.719 Super fast. Right. Makes sense. But the letter 00:03:07.719 --> 00:03:11.199 Q is quite rare. So its code is dash dash dot 00:03:11.199 --> 00:03:14.870 dash. Much longer. the code length perfectly 00:03:14.870 --> 00:03:17.650 reflects the underlying probability of that letter 00:03:17.650 --> 00:03:20.729 appearing in normal English. I see. So if I suddenly 00:03:20.729 --> 00:03:23.009 started writing this weird avant -garde novel 00:03:23.009 --> 00:03:26.069 where Q was the most common letter, but I was 00:03:26.069 --> 00:03:28.210 still using standard Morse code to send it over 00:03:28.210 --> 00:03:30.250 a telegraph. You'd be tapping out dash dash dot 00:03:30.250 --> 00:03:32.770 dash constantly. Yeah, I'd be wasting a massive 00:03:32.770 --> 00:03:35.250 amount of time and like telegraph tape. And that 00:03:35.250 --> 00:03:37.229 exact waste, that extra telegraph tape you're 00:03:37.229 --> 00:03:39.810 printing out is cross entropy. Oh. Yeah, when 00:03:39.810 --> 00:03:42.900 your coding scheme is optimized for a reality 00:03:42.900 --> 00:03:44.960 that doesn't actually match the messages you 00:03:44.960 --> 00:03:47.599 are sending you pay a penalty in efficiency. 00:03:48.340 --> 00:03:50.240 Okay let's bring this out of telegraphs and into 00:03:50.240 --> 00:03:52.800 a physical space so you the listener can really 00:03:52.800 --> 00:03:55.699 feel the weight of this penalty. Imagine you're 00:03:55.699 --> 00:03:58.340 packing for a trip. A very relatable problem. 00:03:58.500 --> 00:04:01.080 Right so if I pack my suitcase assuming it's 00:04:01.080 --> 00:04:04.250 gonna rain That assumption is my estimated distribution, 00:04:04.550 --> 00:04:07.550 my Q. Your best guess. Exactly. So I pack these 00:04:07.550 --> 00:04:10.330 heavy rain boots, a massive umbrella, a huge 00:04:10.330 --> 00:04:12.849 raincoat. What's fascinating here is where the 00:04:12.849 --> 00:04:15.729 math actually places its feet, so to speak. What 00:04:15.729 --> 00:04:18.110 do you mean? Well, the expectation, the mathematical 00:04:18.110 --> 00:04:20.649 average you're calculating in this scenario is 00:04:20.649 --> 00:04:22.810 taken over the true probability distribution 00:04:22.810 --> 00:04:26.209 P. Not my estimated one, Q. Exactly. Not Q. Because 00:04:26.209 --> 00:04:27.990 no matter what you think the weather will be, 00:04:28.170 --> 00:04:30.449 the physical environment you actually encounter 00:04:30.449 --> 00:04:33.589 follows the true distribution. You are forced 00:04:33.589 --> 00:04:37.009 to live in reality even if your internal map 00:04:37.009 --> 00:04:40.269 of that reality is completely flawed. Ah, I get 00:04:40.269 --> 00:04:43.889 it. So if the true reality, P... Turns out to 00:04:43.889 --> 00:04:47.129 be a perfectly sunny 90 degree day. I'm basically 00:04:47.129 --> 00:04:49.649 wandering around a sunny beach sweating in a 00:04:49.649 --> 00:04:51.970 giant raincoat. Exactly. I'm dragging around 00:04:51.970 --> 00:04:54.209 this heavy umbrella for absolutely no reason. 00:04:54.670 --> 00:04:57.470 So the physical effort of carrying that useless 00:04:57.470 --> 00:05:00.129 umbrella, you know, the wasted space in my suitcase 00:05:00.129 --> 00:05:02.829 that could have held my sunglasses, that physical 00:05:02.829 --> 00:05:05.290 burden is the expected message length penalty. 00:05:05.470 --> 00:05:07.569 Yes, that wasted effort is literally our cross 00:05:07.569 --> 00:05:10.189 entropy. It's the cost of optimizing your luggage 00:05:10.189 --> 00:05:12.910 for a reality that simply doesn't exist. Man. 00:05:13.079 --> 00:05:15.680 That physical burden really captures it perfectly. 00:05:16.040 --> 00:05:18.579 It does. And mathematically, the text points 00:05:18.579 --> 00:05:21.100 out a really beautiful relationship to break 00:05:21.100 --> 00:05:23.279 that burden down even further. It's take the 00:05:23.279 --> 00:05:25.699 cross entropy equals the entropy of the true 00:05:25.699 --> 00:05:28.939 distribution P. OK, let's pause. Entropy of P. 00:05:29.240 --> 00:05:32.060 I can think of that as just the unavoidable baseline 00:05:32.060 --> 00:05:35.160 chaos of the weather itself. Exactly. The inherent 00:05:35.160 --> 00:05:37.860 randomness you can't escape. Plus something called 00:05:37.860 --> 00:05:40.339 the Kullback -Liebler divergence of P from Q. 00:05:40.660 --> 00:05:44.470 Whoa. OK. The callback Leibler divergence. We 00:05:44.470 --> 00:05:46.829 definitely can't just drop a massive term like 00:05:46.829 --> 00:05:48.589 that without translating it for the listener. 00:05:48.709 --> 00:05:51.149 Sure enough. If baseline entropy is just the 00:05:51.149 --> 00:05:53.569 unavoidable chaos of the weather, what on earth 00:05:53.569 --> 00:05:56.189 is this divergence thing measuring? Think of 00:05:56.189 --> 00:05:58.569 it as the surprise factor. Surprise factor. Yeah. 00:05:58.569 --> 00:06:00.649 If you expect a sunny day and you step outside 00:06:00.649 --> 00:06:02.870 into a category five hurricane, your physical 00:06:02.870 --> 00:06:05.889 surprise is massive. The callback Leibler divergence 00:06:05.889 --> 00:06:08.610 mathematically measures that exact gap between 00:06:08.610 --> 00:06:12.000 your expectation and reality. Oh, okay. So the 00:06:12.000 --> 00:06:14.300 baseline chaos of the universe plus the specific 00:06:14.300 --> 00:06:17.819 penalty for your bad guess equals the total cross 00:06:17.819 --> 00:06:19.800 entropy. You nailed it. Here's where it gets 00:06:19.800 --> 00:06:22.439 really interesting, though. We just established 00:06:22.439 --> 00:06:25.040 that to calculate this cross entropy penalty, 00:06:25.680 --> 00:06:29.540 the math absolutely requires us to know e -pay, 00:06:29.959 --> 00:06:31.939 the true distribution. Right. It's built into 00:06:31.939 --> 00:06:34.240 the formula. I have to know the actual weather 00:06:34.240 --> 00:06:36.899 to know how mathematically wrong my packing was, 00:06:36.920 --> 00:06:41.120 but... If we transition out of theory and into 00:06:41.120 --> 00:06:42.959 the real world of machine learning, we don't 00:06:42.959 --> 00:06:45.060 actually know the truth. We don't. Right. If 00:06:45.060 --> 00:06:47.560 we knew the perfect truth of the universe, we 00:06:47.560 --> 00:06:50.079 wouldn't need to train an AI to guess it in the 00:06:50.079 --> 00:06:52.480 first place. And that is the ultimate paradox 00:06:52.480 --> 00:06:55.360 of the entire field. The source actually uses 00:06:55.360 --> 00:06:57.740 language modeling as the prime example here. 00:06:58.000 --> 00:07:00.139 OK, let's look at that. Think about the massive 00:07:00.139 --> 00:07:02.699 language models generating text today. What is 00:07:02.699 --> 00:07:05.600 the true distribution of all words in the English 00:07:05.600 --> 00:07:07.589 language? Well, it's impossible to know. I mean, 00:07:07.709 --> 00:07:10.269 it's infinite and it's constantly shifting. Slang 00:07:10.269 --> 00:07:13.569 gets invented literally overnight. Contexts change 00:07:13.569 --> 00:07:16.310 based on whatever is happening in the news. The 00:07:16.310 --> 00:07:18.790 true distribution of human language is just a 00:07:18.790 --> 00:07:22.509 total mystery. So if we don't have P, how is 00:07:22.509 --> 00:07:25.790 our math not just a complete blind guess? How 00:07:25.790 --> 00:07:28.170 on earth do we calculate cross entropy without 00:07:28.170 --> 00:07:31.129 the core variable the formula demands? This is 00:07:31.129 --> 00:07:33.350 where mathematicians play a very, very clever 00:07:33.350 --> 00:07:35.899 estimation game. We use a workaround called the 00:07:35.899 --> 00:07:38.459 Monte Carlo estimate. Monte Carlo? Wait, like 00:07:38.459 --> 00:07:40.600 the casino? Are we just rolling dice in the math? 00:07:41.000 --> 00:07:43.519 In a way, yeah, actually. It relies on random 00:07:43.519 --> 00:07:47.000 sampling to approximate incredibly complex phenomena. 00:07:47.079 --> 00:07:50.040 OK. Since we can't possibly know the true distribution 00:07:50.040 --> 00:07:52.420 of all language that has ever existed or will 00:07:52.420 --> 00:07:55.920 exist, we take a finite test set of data. Let's 00:07:55.920 --> 00:07:58.160 say a set of N words. I write N words. We treat 00:07:58.160 --> 00:08:00.660 that random test set as if it were an actual 00:08:00.660 --> 00:08:03.319 perfect representative sample drawn from the 00:08:03.319 --> 00:08:05.259 true distribution peak. Oh, I get it. It's like 00:08:05.259 --> 00:08:07.480 a political poll. Exactly like a poll. You don't 00:08:07.480 --> 00:08:09.959 ask 300 million people who they are voting for. 00:08:10.199 --> 00:08:12.399 You ask 1 ,000 randomly selected people, and 00:08:12.399 --> 00:08:14.139 you just trust the math to reflect the whole 00:08:14.139 --> 00:08:16.819 country. We let a tiny slice of reality stand 00:08:16.819 --> 00:08:19.220 in for the whole universe. Precisely. So you 00:08:19.220 --> 00:08:21.399 have your model make its probability guesses, 00:08:21.720 --> 00:08:24.759 its Q values, for the words in this specific 00:08:24.759 --> 00:08:27.980 test set. Okay. Then, instead of trying to measure 00:08:27.980 --> 00:08:31.060 against the infinite unknown, you just average 00:08:31.060 --> 00:08:33.580 the log probability over the n words of your 00:08:33.580 --> 00:08:36.720 test set. You basically look at how surprised 00:08:36.720 --> 00:08:38.980 the model was by the actual words that appeared 00:08:38.980 --> 00:08:41.440 in that sample. So if the model assigned a really, 00:08:41.440 --> 00:08:43.639 really low probability to the word that actually 00:08:43.639 --> 00:08:45.980 showed up. The cross entropy penalty spikes. 00:08:46.220 --> 00:08:48.879 Makes sense. So we have this theoretical penalty 00:08:48.879 --> 00:08:51.240 box and we have a clever way to estimate the 00:08:51.240 --> 00:08:54.080 score by tasting just a tiny spoonful of the 00:08:54.080 --> 00:08:57.360 reality soup. Great analogy. Thanks. But how 00:08:57.360 --> 00:09:00.000 do developers actually weaponize this to train 00:09:00.000 --> 00:09:02.539 a machine? I mean, how does an abstract measurement 00:09:02.539 --> 00:09:05.279 of surprise actually force an algorithm to get 00:09:05.279 --> 00:09:08.200 smarter? Well, it becomes what developers call 00:09:08.200 --> 00:09:11.440 a loss function. Specifically in machine learning 00:09:11.440 --> 00:09:13.440 classification problems, you'll hear it referred 00:09:13.440 --> 00:09:17.440 to as log loss or logistic loss. So what does 00:09:17.440 --> 00:09:19.279 this all mean in practice? Let's ground this 00:09:19.279 --> 00:09:21.620 for the listener. A loss function is essentially 00:09:21.620 --> 00:09:24.100 like a video game score, but you want the absolute 00:09:24.100 --> 00:09:26.519 lowest score possible. Exactly. You want it as 00:09:26.519 --> 00:09:29.059 close to zero as you can get. So when a model 00:09:29.059 --> 00:09:31.820 guesses wrong, the cross entropy score shoots 00:09:31.820 --> 00:09:34.899 up. And that high score sends a mathematical 00:09:34.899 --> 00:09:37.950 shockwave. back into the algorithm, basically 00:09:37.950 --> 00:09:40.289 forcing it to adjust its internal wiring to get 00:09:40.289 --> 00:09:42.490 the score back down. That's back propagation 00:09:42.490 --> 00:09:44.850 in a nutshell. But wait, you called it log loss? 00:09:45.070 --> 00:09:46.830 Why is a logarithm involved here? I thought we 00:09:46.830 --> 00:09:48.789 were just calculating percentages and probabilities. 00:09:49.269 --> 00:09:52.419 Ah, that comes down to how computers... and probability 00:09:52.419 --> 00:09:55.059 itself actually function under the hood. When 00:09:55.059 --> 00:09:58.220 you calculate the probability of multiple independent 00:09:58.220 --> 00:10:00.679 events happening in a row, you have to multiply 00:10:00.679 --> 00:10:02.840 those probabilities together. Right, like coin 00:10:02.840 --> 00:10:05.320 flips, half times half times half. Exactly. But 00:10:05.320 --> 00:10:08.700 in AI, you are multiplying massive strings of 00:10:08.700 --> 00:10:11.279 tiny, tiny fractions. And that leads to numbers 00:10:11.279 --> 00:10:14.480 so infinitesimally small that computer processors 00:10:14.480 --> 00:10:16.960 literally cannot handle them. It's an error called 00:10:16.960 --> 00:10:19.340 underflow. So the computer essentially just runs 00:10:19.340 --> 00:10:22.080 out of decimal places to track it all. It does. 00:10:22.399 --> 00:10:25.279 But logarithms possess a magical mathematical 00:10:25.279 --> 00:10:27.899 property. They translate multiplication into 00:10:27.899 --> 00:10:30.820 addition. Yeah. By taking the logarithm of the 00:10:30.820 --> 00:10:32.759 probabilities, the algorithm can just add up 00:10:32.759 --> 00:10:34.960 the penalties instead of multiplying them. It 00:10:34.960 --> 00:10:36.720 saves the computer from breaking down. That's 00:10:36.720 --> 00:10:39.720 incredibly elegant. It is. But it also does something 00:10:39.720 --> 00:10:42.539 else very specific to the penalty itself, especially 00:10:42.539 --> 00:10:45.720 when we look at binary logistic regression. OK, 00:10:45.740 --> 00:10:48.090 let's define that real quick. The binary logistic 00:10:48.090 --> 00:10:50.509 regression just means the model is trying to 00:10:50.509 --> 00:10:52.950 classify observations into one of two buckets, 00:10:53.629 --> 00:10:57.490 like a zero or one, spam or not spam, dog or 00:10:57.490 --> 00:10:59.990 cat. Correct. And the model doesn't just confidently 00:10:59.990 --> 00:11:03.320 declare it's a dog, it Outputs a probability 00:11:03.320 --> 00:11:06.100 curve. It says, you know, I am 85 % sure this 00:11:06.100 --> 00:11:08.120 is a dog. And this is where the logarithm makes 00:11:08.120 --> 00:11:10.620 the loss function so incredibly ruthless. Yeah. 00:11:10.879 --> 00:11:13.299 Let's say the true label is 1. It is a dog. OK. 00:11:13.460 --> 00:11:15.960 True reality is dog. If the model predicted 0 00:11:15.960 --> 00:11:18.960 .99, meaning 99 % sure, the logarithm of that 00:11:18.960 --> 00:11:21.379 is basically 0, the model receives almost no 00:11:21.379 --> 00:11:23.919 penalty. It's rewarded for being confident and 00:11:23.919 --> 00:11:26.059 right. OK. That makes perfect sense. But what 00:11:26.059 --> 00:11:28.519 if the true label is 1 and the model predicted 00:11:28.519 --> 00:11:32.940 0 .01? meaning it was wildly aggressively confident 00:11:32.940 --> 00:11:36.559 that it was not a dog. Oh, boy. Let me guess. 00:11:36.919 --> 00:11:39.860 The logarithm just drops off a cliff. It drops 00:11:39.860 --> 00:11:43.080 straight down. The loss explodes into a massive 00:11:43.080 --> 00:11:46.259 number. It doesn't just dock a few points. It 00:11:46.259 --> 00:11:49.120 catastrophically penalizes the model for being 00:11:49.120 --> 00:11:51.500 confidently wrong. It's an arrogance penalty. 00:11:51.500 --> 00:11:54.019 Yes. If the AI doesn't really know the answer... 00:11:54.090 --> 00:11:56.990 and it offers a mild 50 -50 guess, it gets a 00:11:56.990 --> 00:11:59.649 mild penalty. But if it pounds the table and 00:11:59.649 --> 00:12:02.370 confidently asserts a totally wrong answer, the 00:12:02.370 --> 00:12:04.710 logarithmic curve absolutely destroys its score. 00:12:04.889 --> 00:12:08.330 Which is exactly the behavior you want to discourage 00:12:08.330 --> 00:12:10.629 when training a system. You want the machine 00:12:10.629 --> 00:12:13.350 to learn to calibrate its certainty. And the 00:12:13.350 --> 00:12:15.809 source note is something really cool here. Optimizing 00:12:15.809 --> 00:12:18.460 this log loss. bringing the average cross entropy 00:12:18.460 --> 00:12:21.080 of the sample as close to zero as possible is 00:12:21.080 --> 00:12:23.559 mathematically identical to maximizing the likelihood 00:12:23.559 --> 00:12:26.279 of the model. So likelihood maximization essentially 00:12:26.279 --> 00:12:28.500 amounts to the minimization of cross entropy. 00:12:28.740 --> 00:12:30.879 You've got it. That makes perfect sense. But 00:12:30.879 --> 00:12:32.539 I have to admit, as I'm looking at the Wikipedia 00:12:32.539 --> 00:12:35.279 article, right after this section, it dives into 00:12:35.279 --> 00:12:38.379 some really heavy calculus. It does get a bit 00:12:38.379 --> 00:12:41.620 deep there. It starts comparing the math of categorical 00:12:41.620 --> 00:12:45.039 guesses like dog versus cat 2 linear regression, 00:12:45.620 --> 00:12:47.580 which is just drawing a straight line through 00:12:47.580 --> 00:12:50.740 data points. Why are we comparing apples to oranges? 00:12:51.120 --> 00:12:52.960 Well, if we connect this to the bigger picture, 00:12:53.679 --> 00:12:56.620 there is a beautiful mathematical symmetry hidden 00:12:56.620 --> 00:13:00.000 in that calculus. When you train a linear regression 00:13:00.000 --> 00:13:02.139 model, where you are predicting a continuous 00:13:02.139 --> 00:13:05.000 number, like the future price of a house, you 00:13:05.000 --> 00:13:07.340 use a totally different loss function called 00:13:07.340 --> 00:13:09.320 squared error loss. Right, because you aren't 00:13:09.320 --> 00:13:11.500 guessing a category. You just measure the distance 00:13:11.500 --> 00:13:14.320 from your guess to the actual house price, square 00:13:14.320 --> 00:13:16.379 the number to make sure it's positive, and try 00:13:16.379 --> 00:13:18.840 to minimize that distance. It's a very straightforward 00:13:18.840 --> 00:13:22.179 geometric concept. Exactly. But cross entropy, 00:13:22.440 --> 00:13:24.799 with all its logarithms and infinite probability 00:13:24.799 --> 00:13:27.259 curves, seems like it exists in a totally different 00:13:27.259 --> 00:13:29.299 universe. from drawing a straight line, right? 00:13:29.299 --> 00:13:32.580 Totally. Yet the source provides the proof that 00:13:32.580 --> 00:13:35.080 the gradient of these two functions is essentially 00:13:35.080 --> 00:13:37.820 identical. OK, hold on. We need a quick ELI -5 00:13:37.820 --> 00:13:40.159 for gradient before we compare them. What actually 00:13:40.159 --> 00:13:43.080 is a mathematical gradient in this context? Think 00:13:43.080 --> 00:13:45.100 of a gradient like a compass that points directly 00:13:45.100 --> 00:13:47.919 down the steepest part of a mountain. OK, a compass. 00:13:48.340 --> 00:13:50.700 When an AI is trying to lower its error score, 00:13:51.100 --> 00:13:53.580 it's essentially trying to hike down to the bottom 00:13:53.580 --> 00:13:57.029 of a valley while blindfolded. The gradient is 00:13:57.029 --> 00:13:59.450 the mathematical compass that tells the algorithm 00:13:59.450 --> 00:14:02.769 exactly which direction to step and how big of 00:14:02.769 --> 00:14:05.330 a step to take to adjust its weights and get 00:14:05.330 --> 00:14:07.429 closer to the bottom. Okay, so the compass points 00:14:07.429 --> 00:14:09.190 down the mountain. But you're telling me that 00:14:09.190 --> 00:14:12.570 navigating the jagged logarithmic cliffs of cross 00:14:12.570 --> 00:14:15.629 entropy uses the exact same compass reading as 00:14:15.629 --> 00:14:18.730 the smooth geometric ball of linear regression? 00:14:18.889 --> 00:14:21.370 Yes, up to a constant factor. And you actually 00:14:21.370 --> 00:14:23.590 take the derivative when you calculate the compass 00:14:23.590 --> 00:14:26.440 reading to update the model's weight. All the 00:14:26.440 --> 00:14:29.500 messy logarithmic terms of cross entropy magically 00:14:29.500 --> 00:14:32.340 cancel out. Wait, really? The messy logs just 00:14:32.340 --> 00:14:34.779 vanish? They completely vanish. You are left 00:14:34.779 --> 00:14:37.500 with an incredibly simple formula. The input 00:14:37.500 --> 00:14:39.259 feature is multiplied by the difference between 00:14:39.259 --> 00:14:41.740 the predicted probability and the true label. 00:14:42.299 --> 00:14:44.919 The gradient for cross entropy loss is perfectly 00:14:44.919 --> 00:14:46.940 equal to the gradient of squared error loss. 00:14:47.179 --> 00:14:49.620 That is wild. So whether the algorithm is trying 00:14:49.620 --> 00:14:52.779 to guess a house price using straight lines or 00:14:52.779 --> 00:14:55.360 categorize a photo using complex probability 00:14:55.360 --> 00:14:58.840 curves, the fundamental mathematical compass 00:14:58.840 --> 00:15:01.899 guiding it toward the truth collapses down into 00:15:01.899 --> 00:15:05.220 the exact same elegant equation. It really is. 00:15:05.340 --> 00:15:08.299 It's almost as if nature is revealing a universal 00:15:08.299 --> 00:15:11.440 underlying mechanism for how learning itself 00:15:11.629 --> 00:15:13.929 functions. I absolutely love that. But, you know, 00:15:13.970 --> 00:15:16.250 it wouldn't be a deep dive into an academic concept 00:15:16.250 --> 00:15:18.330 if all the experts agreed on everything perfectly, 00:15:18.490 --> 00:15:21.309 right? True. The text brings up a section on 00:15:21.309 --> 00:15:23.570 something called the principle of minimum cross 00:15:23.570 --> 00:15:26.909 entropy, or minksent. Ah, yeah, the academic 00:15:26.909 --> 00:15:28.590 ambiguity section. This is where the literature 00:15:28.590 --> 00:15:31.620 gets a bit tangled. So we know the goal is to 00:15:31.620 --> 00:15:33.700 minimize cross entropy. We want the lowest score 00:15:33.700 --> 00:15:36.179 possible. And the source mentions Gibbs's inequality, 00:15:36.360 --> 00:15:38.940 which proves that cross entropy reaches its absolute 00:15:38.940 --> 00:15:41.980 minimal possible value when p equals q. When 00:15:41.980 --> 00:15:44.200 your estimated distribution perfectly matches 00:15:44.200 --> 00:15:46.379 reality. Which is logically sound. I mean, if 00:15:46.379 --> 00:15:48.559 your map perfectly matches the territory, your 00:15:48.559 --> 00:15:51.220 penalty is just the inherent entropy of the territory 00:15:51.220 --> 00:15:53.740 itself. There's no wasted effort. Exactly. But 00:15:53.740 --> 00:15:56.159 then the text points out this fascinating theoretical 00:15:56.159 --> 00:15:58.860 conflict in the research papers. It says that 00:15:58.860 --> 00:16:01.480 sometimes the The estimated distribution, Q, 00:16:01.700 --> 00:16:04.639 is treated as a fixed prior reference, and the 00:16:04.639 --> 00:16:07.899 true distribution, P, is the thing being optimized 00:16:07.899 --> 00:16:10.899 to be as close to Q as possible. It swaps them 00:16:10.899 --> 00:16:13.779 completely. Right. Authors like Cover, Thomas, 00:16:13.940 --> 00:16:15.620 and Goode even point out that the literature 00:16:15.620 --> 00:16:18.500 gets super confusing, with researchers constantly 00:16:18.500 --> 00:16:21.159 swapping cross -entropy with relative entropy, 00:16:21.480 --> 00:16:23.960 leading to totally misleading formulas. I have 00:16:23.960 --> 00:16:26.179 to laugh a little bit. Even the experts writing 00:16:26.179 --> 00:16:28.299 the textbooks get the variables mixed up. It 00:16:28.299 --> 00:16:30.519 is slightly amusing, yeah. But it's actually 00:16:30.519 --> 00:16:33.159 a profound issue, and it's vital to understand 00:16:33.159 --> 00:16:35.620 why this distinction matters so deeply. OK, laid 00:16:35.620 --> 00:16:37.840 it on me. When you minimize cross -entropy in 00:16:37.840 --> 00:16:40.100 standard machine learning, you are holding the 00:16:40.100 --> 00:16:43.460 truth. The real world data, P fixed, it is the 00:16:43.460 --> 00:16:45.960 anchor. You're twisting and bending your model 00:16:45.960 --> 00:16:48.379 Q until it looks like the truth. Right, bending 00:16:48.379 --> 00:16:50.820 my expectations to match the reality of the universe. 00:16:50.980 --> 00:16:53.820 Exactly. But in certain specialized optimization 00:16:53.820 --> 00:16:56.259 fields, like rare event probability estimations, 00:16:56.639 --> 00:16:59.120 they flip the anchor. They hold Q fixed. Okay, 00:16:59.240 --> 00:17:02.399 but why? Well, perhaps Q represents a known law 00:17:02.399 --> 00:17:05.519 of physics or a baseline prior belief that simply 00:17:05.519 --> 00:17:09.160 cannot be violated. Then, they optimize p, the 00:17:09.160 --> 00:17:11.799 true distribution they are trying to infer, subject 00:17:11.799 --> 00:17:14.299 to new constraints, so it stays as close to the 00:17:14.299 --> 00:17:16.980 baseline queue as possible. Oh, wow. Wait, so 00:17:16.980 --> 00:17:19.480 instead of making the model fit the data, you 00:17:19.480 --> 00:17:21.480 are trying to find the most reasonable version 00:17:21.480 --> 00:17:23.480 of the data that fits your prior assumptions? 00:17:23.940 --> 00:17:27.000 Precisely. In calculus, swapping which variable 00:17:27.000 --> 00:17:29.539 is the fixed anchor point changes the entire 00:17:29.539 --> 00:17:32.140 foundation of the optimization constraint. The 00:17:32.140 --> 00:17:34.299 two minimizations are completely different mathematical 00:17:34.299 --> 00:17:37.240 operations. Wow. Yeah, and if you confuse them, 00:17:37.359 --> 00:17:39.900 as the academic literature sometimes does, your 00:17:39.900 --> 00:17:42.160 algorithms will literally optimize for a reality 00:17:42.160 --> 00:17:44.579 you didn't intend to create. Bending reality 00:17:44.579 --> 00:17:46.559 to fit your model instead of bending your model 00:17:46.559 --> 00:17:48.680 to fit reality? I mean, that sounds like a recipe 00:17:48.680 --> 00:17:51.740 for a very stubborn, hallucinating AI. Or a fundamentally 00:17:51.740 --> 00:17:54.519 broken one. Okay, so we've built this ruthless 00:17:54.519 --> 00:17:57.500 penalty box for one AI model. We know how it 00:17:57.500 --> 00:17:59.500 learns, how it steps down the mountain, and how 00:17:59.500 --> 00:18:02.960 we have to anchor the truth. But modern AI isn't 00:18:02.960 --> 00:18:05.730 just one brain anymore, is it? No, not at all. 00:18:05.809 --> 00:18:08.369 It's usually a hive mind. Exactly. We use ensembles. 00:18:08.670 --> 00:18:11.069 Does this penalty system break when you have 00:18:11.069 --> 00:18:12.930 five algorithms trying to learn at the exact 00:18:12.930 --> 00:18:15.650 same time? That is the exact problem the final 00:18:15.650 --> 00:18:18.390 section of our source text addresses. Because 00:18:18.390 --> 00:18:20.769 if you train an ensemble of multiple distinct 00:18:20.769 --> 00:18:24.009 neural networks, the theory is that where one 00:18:24.009 --> 00:18:27.009 model has a blind spot, another model might excel. 00:18:28.089 --> 00:18:30.670 By averaging their predictions together, the 00:18:30.670 --> 00:18:33.789 combined accuracy is augmented. But wait, if 00:18:33.789 --> 00:18:36.470 I just train five models on the exact same data 00:18:36.470 --> 00:18:38.690 using the exact same cross entropy loss function 00:18:38.690 --> 00:18:40.809 we've been talking about, won't they all just 00:18:40.809 --> 00:18:42.710 hike down the exact same mountain and learn the 00:18:42.710 --> 00:18:44.950 exact same blind spots? Yes, they absolutely 00:18:44.950 --> 00:18:47.170 would. Which is why the researchers introduced 00:18:47.170 --> 00:18:49.250 something called amended cross entropy. Amended 00:18:49.250 --> 00:18:51.589 cross entropy. Right. It alters the loss function 00:18:51.589 --> 00:18:54.130 for an ensemble of, say, k number of classifiers. 00:18:54.809 --> 00:18:57.009 It introduces a new mathematical parameter into 00:18:57.009 --> 00:18:59.549 the equation called lambda. Okay. This lambda 00:18:59.549 --> 00:19:01.990 ranges from a value of zero to a value of one. 00:19:02.640 --> 00:19:05.400 and tweaking it fundamentally changes what the 00:19:05.400 --> 00:19:07.880 models are rewarded and penalized for during 00:19:07.880 --> 00:19:10.099 training. Okay, let's use an analogy to really 00:19:10.099 --> 00:19:12.619 lock down how this lambda parameter works because 00:19:12.619 --> 00:19:14.240 this sounds a lot like building a pub trivia 00:19:14.240 --> 00:19:17.559 team. Let's say lambda is set to zero. If lambda 00:19:17.559 --> 00:19:21.319 is zero, the math formula shows that each classifier 00:19:21.319 --> 00:19:24.170 operates entirely independently. It just tries 00:19:24.170 --> 00:19:27.089 to minimize its own individual error against 00:19:27.089 --> 00:19:29.990 the true probability, completely ignoring the 00:19:29.990 --> 00:19:31.869 existence of the other models in the ensemble. 00:19:32.069 --> 00:19:34.309 Right. So if I'm building my trivia team and 00:19:34.309 --> 00:19:37.609 lambda is zero, I'm just looking at each player's 00:19:37.609 --> 00:19:40.890 individual high score. I hire the five smartest 00:19:40.890 --> 00:19:43.220 people I can find. But because I'm not looking 00:19:43.220 --> 00:19:45.619 at how they interact, I might accidentally hire 00:19:45.619 --> 00:19:48.740 five guys who only know about sports. A very 00:19:48.740 --> 00:19:51.019 common mistake. Right. And if a history question 00:19:51.019 --> 00:19:53.480 comes up, the whole team fails because they all 00:19:53.480 --> 00:19:55.920 show the exact same blind spot. So how does the 00:19:55.920 --> 00:19:59.000 math fix my team when we turn lambda up? If you 00:19:59.000 --> 00:20:01.680 turn lambda up to one, the cost function completely 00:20:01.680 --> 00:20:03.769 changes. It still wants the individual models 00:20:03.769 --> 00:20:06.269 to be accurate, but it actively deducts points 00:20:06.269 --> 00:20:08.950 from the team's total score every time two models 00:20:08.950 --> 00:20:11.150 give the exact same correct answer. Hold on, 00:20:11.289 --> 00:20:12.990 it punishes them for agreeing with each other. 00:20:13.289 --> 00:20:16.329 It penalizes the models if their output probabilities 00:20:16.329 --> 00:20:19.250 are too mathematically similar. It basically 00:20:19.250 --> 00:20:21.789 forces the algorithms to diverge, seeking out 00:20:21.789 --> 00:20:23.890 different features of the data to arrive at the 00:20:23.890 --> 00:20:27.750 truth. Oh man. So by turning lambda to 1, the 00:20:27.750 --> 00:20:30.829 math actively penalizes my trivia team if everyone 00:20:30.829 --> 00:20:33.970 tries to be the sports guy. It forces the algorithms 00:20:33.970 --> 00:20:36.170 to study different subjects, ensuring that one 00:20:36.170 --> 00:20:38.589 model focuses on being the history buff, another 00:20:38.589 --> 00:20:40.730 becomes the pop -culture expert, and another 00:20:40.730 --> 00:20:43.450 handles science. And then when you average their 00:20:43.450 --> 00:20:45.650 diverse answers together, the ensemble becomes 00:20:45.650 --> 00:20:48.640 practically bulletproof. is an important question, 00:20:48.680 --> 00:20:50.680 doesn't it? I want you to really think about 00:20:50.680 --> 00:20:53.119 the elegance of what is happening here. We aren't 00:20:53.119 --> 00:20:55.000 just using calculus to find the right answer 00:20:55.000 --> 00:20:58.019 anymore. With amended cross entropy, we are using 00:20:58.019 --> 00:21:01.460 a mathematical parameter to explicitly encode 00:21:01.460 --> 00:21:04.099 the value of diversity of thought into artificial 00:21:04.099 --> 00:21:06.319 intelligence. We are mathematically proving that 00:21:06.319 --> 00:21:08.599 a diverse team of slightly different perspectives 00:21:08.599 --> 00:21:11.859 vastly outperforms a homogeneous team of identical 00:21:11.859 --> 00:21:14.440 experts. We are literally weaving the necessity 00:21:14.440 --> 00:21:17.380 of diverse perspectives directly into the cost 00:21:17.380 --> 00:21:19.509 function of the machine. That is truly incredible. 00:21:19.630 --> 00:21:21.369 Let's do a quick recap for you, our listener, 00:21:21.529 --> 00:21:24.049 because we have covered a massive amount of ground 00:21:24.049 --> 00:21:27.009 today. We started with the abstract idea of message 00:21:27.009 --> 00:21:29.950 lengths, Morse code, and the physical penalty 00:21:29.950 --> 00:21:33.039 of packing an umbrella for a sunny day. We defined 00:21:33.039 --> 00:21:35.299 the expected cost of an incorrect assumption 00:21:35.299 --> 00:21:38.200 against a true reality. Right. Then we moved 00:21:38.200 --> 00:21:41.180 into how developers use a tiny random slice of 00:21:41.180 --> 00:21:43.940 reality that Monte Carlo test set to estimate 00:21:43.940 --> 00:21:46.519 that penalty when the absolute truth is infinite 00:21:46.519 --> 00:21:49.380 and unknowable. Very important step. Yeah. We 00:21:49.380 --> 00:21:51.960 saw how log loss acts as a ruthless video game 00:21:51.960 --> 00:21:54.880 score, utilizing logarithms to translate massive 00:21:54.880 --> 00:21:57.200 multiplication problems into simple addition 00:21:57.200 --> 00:22:00.180 while severely punishing models for being arrogantly 00:22:00.180 --> 00:22:02.740 wrong. We navigated the mathematical symmetry 00:22:02.740 --> 00:22:04.859 of gradients, pointing our compass down the mountain. 00:22:05.359 --> 00:22:07.579 And we explore the academic drama of Mink's scent 00:22:07.579 --> 00:22:09.779 and the profound dangers of swapping your anchor 00:22:09.779 --> 00:22:12.700 variables. And finally, we arrived at amended 00:22:12.700 --> 00:22:16.380 cross entropy, where we saw how a single parameter, 00:22:16.460 --> 00:22:20.000 lambda, literally programs the value of diverse 00:22:20.000 --> 00:22:23.099 perspectives into a HiveMine AI's learning engine. 00:22:23.480 --> 00:22:26.079 It is absolutely amazing how much consequence 00:22:26.079 --> 00:22:29.319 is packed into a single Wikipedia page. It truly 00:22:29.319 --> 00:22:31.400 is the invisible mathematical architecture of 00:22:31.400 --> 00:22:33.599 the digital world. Yeah. But, you know, I will 00:22:33.599 --> 00:22:35.460 leave you with one final thought to mull over. 00:22:35.579 --> 00:22:37.700 Let's hear it. Something that isn't explicitly 00:22:37.700 --> 00:22:40.880 detailed in the text, but is the terrifyingly 00:22:40.880 --> 00:22:43.000 logical next step of everything we've discussed 00:22:43.000 --> 00:22:45.119 today. Okay, I'm listening. We've firmly established 00:22:45.119 --> 00:22:48.480 that cross entropy relies on measuring an AI's 00:22:48.480 --> 00:22:51.599 estimated distribution P. Right, it measures 00:22:51.599 --> 00:22:54.119 the machine against our physical reality. But 00:22:54.119 --> 00:22:57.119 right now, AI models are generating text, images, 00:22:57.259 --> 00:23:00.259 and data at an absolutely unprecedented global 00:23:00.259 --> 00:23:02.920 scale. What happens in the very near future when 00:23:02.920 --> 00:23:06.019 we run out of fresh human data? What happens 00:23:06.019 --> 00:23:08.779 when new AI models are trained almost entirely 00:23:08.779 --> 00:23:11.400 on the data generated by other AI models? The 00:23:11.400 --> 00:23:14.339 anchor point completely shifts. Exactly. If the 00:23:14.339 --> 00:23:17.380 true distribution, BIA, becomes entirely synthetic, 00:23:17.779 --> 00:23:21.029 generated by the algorithms themselves. Will 00:23:21.029 --> 00:23:23.750 our cross -entropy metric still be measuring 00:23:23.750 --> 00:23:26.589 reality, or will it just be measuring an infinite 00:23:26.589 --> 00:23:29.150 hall of digital mirrors? An algorithm perfectly 00:23:29.150 --> 00:23:31.930 optimizing itself to match the hallucinations 00:23:31.930 --> 00:23:34.730 of another algorithm. Yes. Will the machine even 00:23:34.730 --> 00:23:36.769 know what reality is anymore, or will it just 00:23:36.769 --> 00:23:38.789 know how to perfectly predict its own echoes? 00:23:39.839 --> 00:23:42.420 That is a haunting thought to end on and definitely 00:23:42.420 --> 00:23:44.819 something to keep an eye on as these models continue 00:23:44.819 --> 00:23:46.880 to scale. Thank you so much for joining us on 00:23:46.880 --> 00:23:48.920 this deep dive. Next time you see a machine do 00:23:48.920 --> 00:23:50.839 something incredibly smart, you'll know exactly 00:23:50.839 --> 00:23:52.859 the mathematical penalty it had to pay to get 00:23:52.859 --> 00:23:54.019 there. Until next time.