WEBVTT 00:00:00.000 --> 00:00:03.560 You know, um, when you or I sit down to play 00:00:03.560 --> 00:00:05.940 a brand new video game for the very first time, 00:00:06.459 --> 00:00:08.859 we usually have like some kind of context built 00:00:08.859 --> 00:00:11.199 in. Absolutely. Yeah. Like we know what a controller 00:00:11.199 --> 00:00:15.119 is. We know that, uh, glowing red obstacles are 00:00:15.119 --> 00:00:17.820 probably bad and shiny gold coins are usually 00:00:17.820 --> 00:00:20.820 good. We bring a lifetime of intuition into that 00:00:20.820 --> 00:00:23.399 completely new environment. The human brain, 00:00:23.620 --> 00:00:26.019 I mean, It simply doesn't start from a blank 00:00:26.019 --> 00:00:29.000 slate. Exactly. But imagine having absolutely 00:00:29.000 --> 00:00:31.379 zero contacts. Like you don't know what a screen 00:00:31.379 --> 00:00:33.500 is, you don't know what a button does. You have 00:00:33.500 --> 00:00:36.320 no concept of winning or losing. Right. Or even 00:00:36.320 --> 00:00:38.799 existing in a virtual space at all. You are just 00:00:38.799 --> 00:00:41.679 blindly flailing around in the dark. So how do 00:00:41.679 --> 00:00:44.340 you go from that state of total ignorance to 00:00:44.340 --> 00:00:47.179 not just playing the game, but completely dominating 00:00:47.179 --> 00:00:49.780 it? It's a huge leap. It really is. And you do 00:00:49.780 --> 00:00:52.689 it with a concept called cue learning. Welcome 00:00:52.689 --> 00:00:55.289 to today's deep dive. Our mission today is to 00:00:55.289 --> 00:00:57.229 shortcut your journey to understanding one of 00:00:57.229 --> 00:01:00.130 the absolute core mechanisms behind how machines 00:01:00.130 --> 00:01:02.329 actually learn to make decisions. And it's a 00:01:02.329 --> 00:01:05.290 fascinating mechanism. We are looking at a comprehensive 00:01:05.290 --> 00:01:08.170 Wikipedia article on Q -learning, and by the 00:01:08.170 --> 00:01:10.189 end of our time together, you are going to understand 00:01:10.189 --> 00:01:14.189 exactly how AI pulls off this magic trick without 00:01:14.189 --> 00:01:17.650 needing a degree in computer science. The transition 00:01:17.650 --> 00:01:21.120 from blind guessing to strategic mastery really 00:01:21.120 --> 00:01:24.079 is elegant once the underlying machinery is exposed. 00:01:24.239 --> 00:01:27.260 I mean, it all boils down to how a system mathematically 00:01:27.260 --> 00:01:30.319 learns to assign long -term value to its immediate 00:01:30.319 --> 00:01:32.519 choices. Okay, let's unpack this because before 00:01:32.519 --> 00:01:34.180 we dive into the math, I want to ground this 00:01:34.180 --> 00:01:36.379 in the physical world to explain what reinforcement 00:01:36.379 --> 00:01:38.739 learning actually is since Q -learning is like 00:01:38.739 --> 00:01:41.540 a specific algorithm operating under that larger 00:01:41.540 --> 00:01:43.500 umbrella. Right, it's a subset of reinforcement 00:01:43.500 --> 00:01:45.719 learning. Yeah. And the source lays out four 00:01:45.719 --> 00:01:48.319 basic building blocks. We have an agent, which 00:01:48.319 --> 00:01:51.000 is our AI learner. We have a set of states, which 00:01:51.000 --> 00:01:54.579 represents the specific situation the agent is 00:01:54.579 --> 00:01:57.280 currently in. We have actions, which are the 00:01:57.280 --> 00:01:59.659 moves the agent can make. And finally, we have 00:01:59.659 --> 00:02:02.099 a reward, which is simply a numerical score the 00:02:02.099 --> 00:02:04.299 agent receives after taking a specific action. 00:02:04.700 --> 00:02:06.900 Those four pillars create the entire foundation. 00:02:07.819 --> 00:02:09.960 And the defining characteristic of Q -learning 00:02:09.960 --> 00:02:12.539 within that framework is that it is model -free. 00:02:12.680 --> 00:02:14.860 Model -free. What does that mean exactly? It 00:02:14.860 --> 00:02:17.360 means the agent does not require a pre -existing 00:02:17.360 --> 00:02:19.819 map or an overarching model of its environment. 00:02:20.439 --> 00:02:23.020 It figures out the rules of the universe entirely 00:02:23.020 --> 00:02:25.620 on the fly through trial and error. Oh, wow. 00:02:25.819 --> 00:02:28.819 So it's just guessing at first. Exactly. If a 00:02:28.819 --> 00:02:31.360 certain action in a certain state yields a high 00:02:31.360 --> 00:02:34.680 reward, the agent learns to favor it. And that 00:02:34.680 --> 00:02:37.319 brings us to the Q and Q learning, which simply 00:02:37.319 --> 00:02:39.860 stands for the quality of an action taken in 00:02:39.860 --> 00:02:42.919 a given state. The quality, okay. So it is constantly 00:02:42.919 --> 00:02:45.219 computing the expected reward of every move. 00:02:45.460 --> 00:02:47.740 Precisely. The source use is a really relatable 00:02:47.740 --> 00:02:50.300 example to explain this. The daily commute of 00:02:50.300 --> 00:02:52.560 boarding a train. Like, let's look at how an 00:02:52.560 --> 00:02:55.400 AI would approach getting on a subway car. Let's 00:02:55.400 --> 00:02:57.759 say the reward we want is measured by the total 00:02:57.759 --> 00:02:59.500 time you spend boarding, and you obviously want 00:02:59.500 --> 00:03:02.000 that number to be as low as possible. Your cost 00:03:02.000 --> 00:03:04.969 is your time. It is a scenario most people have 00:03:04.969 --> 00:03:07.189 faced. You know, you are standing on the platform, 00:03:07.330 --> 00:03:09.629 the train arrives, the door is open. What is 00:03:09.629 --> 00:03:12.889 the optimal move? Right. So day one, the agent 00:03:12.889 --> 00:03:15.729 tries the forceful strategy. The door is open 00:03:15.729 --> 00:03:18.250 and it immediately pushes forward. The initial 00:03:18.250 --> 00:03:21.110 wait time is zero seconds. A very aggressive 00:03:21.110 --> 00:03:24.569 agent. Very. But because the train is crowded 00:03:24.569 --> 00:03:26.889 with people trying to get off, the agent spends 00:03:26.889 --> 00:03:30.620 like... 15 seconds awkwardly fighting its way 00:03:30.620 --> 00:03:33.479 past departing passengers. So the total cost 00:03:33.479 --> 00:03:36.680 for that forceful action is 15 seconds. A highly 00:03:36.680 --> 00:03:39.900 inefficient strategy, though. One you see humans 00:03:39.900 --> 00:03:42.340 employ every single day in major cities. Definitely. 00:03:42.379 --> 00:03:44.159 I've been that person. But then on the next day, 00:03:44.300 --> 00:03:46.659 the algorithm forces a random choice, a phase 00:03:46.659 --> 00:03:49.439 the source calls exploration. The agent decides 00:03:49.439 --> 00:03:51.560 to try a different action. Has to explore to 00:03:51.560 --> 00:03:54.099 learn. Yeah. So it decides to stand back and 00:03:54.099 --> 00:03:55.939 wait for the parting passengers to clear out. 00:03:55.979 --> 00:03:58.139 This is the patient strategy. And in the very 00:03:58.139 --> 00:03:59.879 short term, this looks like a worse decision. 00:04:00.520 --> 00:04:02.599 I mean, the immediate wait time goes from zero 00:04:02.599 --> 00:04:04.460 to five seconds. Right, it's just standing there. 00:04:04.740 --> 00:04:06.939 But because the path is now entirely clear, the 00:04:06.939 --> 00:04:09.379 agent spends zero seconds fighting its way onto 00:04:09.379 --> 00:04:12.280 the train. The overall cost is just the five 00:04:12.280 --> 00:04:14.460 seconds of waiting. Which is a huge improvement. 00:04:14.800 --> 00:04:19.089 Huge. Through forced exploration, the agent experiences 00:04:19.089 --> 00:04:22.149 this revelation. Even though the initial action 00:04:22.149 --> 00:04:25.470 of waiting had a higher immediate cost, the overarching 00:04:25.470 --> 00:04:29.189 long -term route is vastly superior. What's fascinating 00:04:29.189 --> 00:04:32.389 here is how the underlying math encodes that 00:04:32.389 --> 00:04:35.620 realization by shifting its focus. The agent's 00:04:35.620 --> 00:04:38.800 goal is to maximize total reward, not just immediate 00:04:38.800 --> 00:04:41.620 rewards. So it's looking ahead. Exactly. It calculates 00:04:41.620 --> 00:04:44.180 the quality of a state action combination by 00:04:44.180 --> 00:04:46.740 adding the maximum attainable reward from future 00:04:46.740 --> 00:04:49.360 states to whatever reward it is getting right 00:04:49.360 --> 00:04:51.870 now. So it's not just seeking an instant dopamine 00:04:51.870 --> 00:04:55.370 hit. It's almost calculating the compound interest 00:04:55.370 --> 00:04:57.790 of its moves. It's asking, does walking through 00:04:57.790 --> 00:04:59.629 this train door put me in a physical position 00:04:59.629 --> 00:05:01.889 where my next five moves will be highly rewarded? 00:05:02.149 --> 00:05:03.990 That concept of compound interest is a great 00:05:03.990 --> 00:05:06.029 way to visualize it. It learns the potential 00:05:06.029 --> 00:05:08.410 of an action by weighing the expected values 00:05:08.410 --> 00:05:10.970 of all the future steps that will cascade from 00:05:10.970 --> 00:05:12.889 that single choice. But if it's just wandering 00:05:12.889 --> 00:05:15.189 around a train station without a map, How does 00:05:15.189 --> 00:05:17.449 it actually update that math in its head? Like, 00:05:17.449 --> 00:05:19.529 how does it weigh an old belief against a new 00:05:19.529 --> 00:05:22.449 discovery? That happens through the algorithm's 00:05:22.449 --> 00:05:25.649 engine, specifically the Bellman equation. We 00:05:25.649 --> 00:05:27.689 don't need to write out the mathematical formula, 00:05:28.449 --> 00:05:31.649 but its function is essentially a simple value 00:05:31.649 --> 00:05:34.639 iteration update. Whenever the agent takes a 00:05:34.639 --> 00:05:37.740 step and learns something new, the equation calculates 00:05:37.740 --> 00:05:40.300 a weighted average. So it mixes them together. 00:05:40.439 --> 00:05:42.819 Right. It takes the old value the agent believed 00:05:42.819 --> 00:05:45.899 to be true and blends it with the new information 00:05:45.899 --> 00:05:48.759 it just discovered. And the programmer has a 00:05:48.759 --> 00:05:51.000 few levers they can pull to control exactly how 00:05:51.000 --> 00:05:53.639 that blending happens, right? The first one the 00:05:53.639 --> 00:05:56.100 source mentions is the learning rate, represented 00:05:56.100 --> 00:05:58.819 by the Greek letter alpha. And this operates 00:05:58.819 --> 00:06:01.699 on a scale from zero to one. Correct. The learning 00:06:01.699 --> 00:06:04.759 rate dictates how aggressively the AI overwrites 00:06:04.759 --> 00:06:07.199 its old knowledge. Yeah. If a programmer sets 00:06:07.199 --> 00:06:10.120 the learning rate to zero, the agent learns absolutely 00:06:10.120 --> 00:06:12.759 nothing from new experiences. Oh, wow. Yeah, 00:06:12.779 --> 00:06:15.259 it relies exclusively on whatever prior knowledge 00:06:15.259 --> 00:06:17.699 it already has. It becomes a brick wall. Like, 00:06:17.740 --> 00:06:19.540 you could show it a faster way onto the train 00:06:19.540 --> 00:06:21.579 a hundred times, and it would just ignore you. 00:06:21.800 --> 00:06:23.860 On the other end of the spectrum, if the learning 00:06:23.860 --> 00:06:26.870 rate is set to one, The agent considers only 00:06:26.870 --> 00:06:29.910 the most recent information and completely discards 00:06:29.910 --> 00:06:31.930 anything it learned in the past. So it just has 00:06:31.930 --> 00:06:34.910 no memory at all. It becomes entirely reactive, 00:06:35.089 --> 00:06:37.410 jumping from one new possibility to the next 00:06:37.410 --> 00:06:39.850 without building a foundation of knowledge. That 00:06:39.850 --> 00:06:43.009 seems bad, too. It is. In practical applications, 00:06:43.430 --> 00:06:45.730 developers often use a constant lower learning 00:06:45.730 --> 00:06:48.819 rate. something around, say, 0 .1, to allow the 00:06:48.819 --> 00:06:52.579 AI to build a steady, reliable memory while still 00:06:52.579 --> 00:06:54.980 adapting to new tricks. That makes total sense. 00:06:55.660 --> 00:06:58.540 Now, the next lever the algorithm uses is discount 00:06:58.540 --> 00:07:01.629 factor, represented by gamma. The source says 00:07:01.629 --> 00:07:04.009 this determines the importance of future rewards. 00:07:04.370 --> 00:07:06.350 Yes, the discount factor. So wait, so if I set 00:07:06.350 --> 00:07:09.050 the discount factor to zero, the AI only cares 00:07:09.050 --> 00:07:11.310 about the reward it gets in the very next second. 00:07:11.370 --> 00:07:13.670 Like, it's basically short -sighted. Exactly. 00:07:13.870 --> 00:07:15.970 The source actually uses the term myopic for 00:07:15.970 --> 00:07:19.189 that exact condition. A discount factor of zero 00:07:19.189 --> 00:07:22.069 means the AI will only ever optimize for the 00:07:22.069 --> 00:07:24.589 immediate step. As you increase that factor closer 00:07:24.589 --> 00:07:28.269 to one, the agent begins to strive for long -term 00:07:28.269 --> 00:07:32.629 delayed gratification. However, there is a mathematical 00:07:32.629 --> 00:07:35.029 danger zone here. The danger zone. If the discount 00:07:35.029 --> 00:07:38.310 factor reaches or exceeds one, the formula essentially 00:07:38.310 --> 00:07:40.970 breaks. Why does it break? Because it's suddenly 00:07:40.970 --> 00:07:43.529 valuing a reward 10 years from now just as highly 00:07:43.529 --> 00:07:46.970 as a reward today. Worse than that, if the environment 00:07:46.970 --> 00:07:49.310 doesn't have a definitive final state -like, 00:07:49.370 --> 00:07:52.509 a guaranteed end to the game, the action values 00:07:52.509 --> 00:07:55.649 can diverge to infinity. Oh man. The AI gets 00:07:55.649 --> 00:07:57.949 stuck in an endless loop of adding up future 00:07:57.949 --> 00:07:59.810 potential rewards that it will never actually 00:07:59.810 --> 00:08:02.389 reach and the learning process becomes completely 00:08:02.389 --> 00:08:05.550 unstable. It just crashes. Pretty much. Developers 00:08:05.550 --> 00:08:07.810 often start with a lower discount factor and 00:08:07.810 --> 00:08:09.649 cautiously increase it to accelerate learning 00:08:09.649 --> 00:08:11.870 safely without triggering that infinite loop. 00:08:12.079 --> 00:08:14.240 There is one more crucial lever we need to talk 00:08:14.240 --> 00:08:16.100 about before the AI can start exploring, and 00:08:16.100 --> 00:08:19.819 that's the initial conditions, or Q0. Since the 00:08:19.819 --> 00:08:22.279 AI starts with a completely blank slate, the 00:08:22.279 --> 00:08:24.500 programmer has to assign some kind of arbitrary 00:08:24.500 --> 00:08:27.019 starting value to every possible move before 00:08:27.019 --> 00:08:29.879 the game begins. And those initial values are 00:08:29.879 --> 00:08:32.740 incredibly powerful, because they can be used 00:08:32.740 --> 00:08:35.639 to artificially manipulate the AI's behavior. 00:08:35.840 --> 00:08:38.860 How so? The source details a technique called 00:08:38.860 --> 00:08:42.070 optimistic initial conditions. where you assign 00:08:42.070 --> 00:08:44.909 starting values that are significantly higher 00:08:44.909 --> 00:08:47.090 than any reward the agent could realistically 00:08:47.090 --> 00:08:50.250 achieve. Oh, I see. So the AI takes its very 00:08:50.250 --> 00:08:53.350 first action, receives a normal reward, and thinks, 00:08:53.570 --> 00:08:55.690 wow, that was incredibly disappointing compared 00:08:55.690 --> 00:08:58.009 to what I expected. I better go try all these 00:08:58.009 --> 00:09:00.289 other unexplored options because they still have 00:09:00.289 --> 00:09:02.450 those massive starting scores attached to them. 00:09:02.450 --> 00:09:04.830 Exactly. It's a psychological trick to force 00:09:04.830 --> 00:09:07.509 the machine to explore. It really is. It acts 00:09:07.509 --> 00:09:09.870 as a built -in mechanism to prevent the agent 00:09:09.870 --> 00:09:13.039 from finding one mediocre strategy and stubbornly 00:09:13.039 --> 00:09:15.279 sticking to it forever. That's brilliant. And 00:09:15.279 --> 00:09:17.539 the article also notes a fascinating parallel 00:09:17.539 --> 00:09:20.759 to human psychology here. There's a model called 00:09:20.759 --> 00:09:24.379 Reset of Initial Conditions, or RIC. In this 00:09:24.379 --> 00:09:26.700 model, the first time an action is taken, the 00:09:26.700 --> 00:09:29.240 actual reward received is used to set the baseline 00:09:29.240 --> 00:09:31.559 going forward. Okay, so resetting expectations. 00:09:32.080 --> 00:09:34.690 Right. And the source points out that this specific 00:09:34.690 --> 00:09:37.649 mathematical model predicts human behavior in 00:09:37.649 --> 00:09:40.870 repeated binary choice experiments far better 00:09:40.870 --> 00:09:43.649 than assuming humans operate on arbitrary conditions. 00:09:44.830 --> 00:09:47.269 So we clearly share a bit of that underlying 00:09:47.269 --> 00:09:49.389 math in our own brains when we're exploring our 00:09:49.389 --> 00:09:53.190 options. But as I'm picturing this giant spreadsheet, 00:09:53.409 --> 00:09:56.769 this Q table with every possible state and every 00:09:56.769 --> 00:09:59.850 possible action, a massive flaw jumps out at 00:09:59.850 --> 00:10:02.330 me. What's that? Keeping a table of every state 00:10:02.330 --> 00:10:05.490 is perfectly fine for a simple grid maze or a 00:10:05.490 --> 00:10:08.190 subway platform with three doors. But the real 00:10:08.190 --> 00:10:10.769 world is infinitely complex. Like, if I move 00:10:10.769 --> 00:10:12.450 my arm one inch, that's a new state. If I move 00:10:12.450 --> 00:10:14.470 it half an inch, that's another state. It's continuous. 00:10:14.649 --> 00:10:17.590 Yeah. How does a computer's memory not just immediately 00:10:17.590 --> 00:10:19.970 melt down when faced with an infinite universe 00:10:19.970 --> 00:10:22.309 of choices? That phenomenon is known in computer 00:10:22.309 --> 00:10:24.960 science as the curse of dimensionality. The curse 00:10:24.960 --> 00:10:27.840 of dimensionality sounds in tis. It is a major 00:10:27.840 --> 00:10:30.559 hurdle. As the number of variables increases, 00:10:30.980 --> 00:10:33.840 the sheer volume of possible states grows exponentially. 00:10:34.720 --> 00:10:36.720 The likelihood of an agent randomly visiting 00:10:36.720 --> 00:10:39.240 a specific state and trying a specific action 00:10:39.240 --> 00:10:42.100 becomes practically zero. Standard Q -learning 00:10:42.100 --> 00:10:44.799 tables simply cannot handle the infinite nature 00:10:44.799 --> 00:10:48.440 of the physical world. So how do we fix it? The 00:10:48.440 --> 00:10:50.840 source details a couple of fascinating workarounds. 00:10:51.080 --> 00:10:54.360 The first is a concept called quantization. To 00:10:54.360 --> 00:10:56.559 explain this, let's use the source's example 00:10:56.559 --> 00:10:59.039 of trying to teach an AI to balance a stick on 00:10:59.039 --> 00:11:01.940 its finger. Ah, a classic continuous physics 00:11:01.940 --> 00:11:04.700 problem. To understand the state of that system 00:11:04.700 --> 00:11:07.559 at any microsecond, the AI needs a four -element 00:11:07.559 --> 00:11:09.620 vector. A four -element vector, right. It has 00:11:09.620 --> 00:11:11.500 to track the position of the cart or finger in 00:11:11.500 --> 00:11:14.100 space, its velocity, the angle of the stick, 00:11:14.200 --> 00:11:16.220 and the angular velocity of the stick. But the 00:11:16.220 --> 00:11:18.360 exact angle of that stick could be infinitely 00:11:18.360 --> 00:11:21.159 detailed. It could be leaning at 1 .5 degrees 00:11:21.159 --> 00:11:24.679 or 1 .5 degrees or 1 .55 degrees going on forever. 00:11:24.919 --> 00:11:27.220 A spreadsheet tracking every single decimal point 00:11:27.220 --> 00:11:30.019 would require infinite memory. Quantization solves 00:11:30.019 --> 00:11:32.299 this by grouping those infinite ranges of values 00:11:32.269 --> 00:11:34.870 into discrete buckets. OK, so instead of the 00:11:34.870 --> 00:11:37.730 AI trying to memorize the exact atomic angle 00:11:37.730 --> 00:11:40.669 of the stick, it just throws the data into a 00:11:40.669 --> 00:11:43.710 bucket labeled near or far or leaning left or 00:11:43.710 --> 00:11:46.330 leaning right. It shrinks an infinite universe 00:11:46.330 --> 00:11:48.929 down into a manageable set of categories so the 00:11:48.929 --> 00:11:51.429 brain doesn't melt. Exactly. It creates an artificial 00:11:51.429 --> 00:11:55.529 grid over a continuous world. But for truly massive 00:11:55.529 --> 00:11:59.090 complex problems like driving a car or analyzing 00:11:59.090 --> 00:12:02.240 a video feed, even quantization isn't enough. 00:12:02.440 --> 00:12:04.759 There are just too many buckets. Right. That 00:12:04.759 --> 00:12:06.840 is where function approximation comes into play. 00:12:07.279 --> 00:12:09.379 Instead of building a table of discrete buckets, 00:12:09.720 --> 00:12:12.279 you use a secondary system to generalize past 00:12:12.279 --> 00:12:15.379 experiences to unseen states. Meaning, if the 00:12:15.379 --> 00:12:17.679 AI encounters a situation on the road that is 00:12:17.679 --> 00:12:19.720 only slightly different from when it has seen 00:12:19.720 --> 00:12:21.799 before, it doesn't need to start from scratch. 00:12:21.919 --> 00:12:24.419 It uses the approximation to guess what the quality 00:12:24.419 --> 00:12:26.879 of its next move will be. That approximation 00:12:26.879 --> 00:12:29.500 is frequently handled by an artificial neural 00:12:29.500 --> 00:12:31.779 network. However, However, the article highlights 00:12:31.779 --> 00:12:34.179 an alternative method integrating fuzzy rule 00:12:34.179 --> 00:12:37.740 interpolation, or FRI. Now, fuzzy rules sound 00:12:37.740 --> 00:12:40.259 inherently unscientific to me, but they actually 00:12:40.259 --> 00:12:42.440 solve a massive problem with neural networks, 00:12:42.860 --> 00:12:44.860 right? Because neural networks often act as a 00:12:44.860 --> 00:12:47.820 black box. They do, yes. Like, the AI might make 00:12:47.820 --> 00:12:49.779 a brilliant decision, but we have absolutely 00:12:49.779 --> 00:12:52.340 no idea how it arrived at that conclusion. And 00:12:52.340 --> 00:12:55.080 a black box can be dangerous in critical applications. 00:12:55.679 --> 00:12:58.700 The advantage of using sparse, fuzzy rule bases 00:12:58.700 --> 00:13:01.799 for function approximation is that it keeps the 00:13:01.799 --> 00:13:04.919 AI's internal logic transparent and human readable. 00:13:05.000 --> 00:13:07.500 Oh, that's really important. Instead of millions 00:13:07.500 --> 00:13:11.139 of hidden weights in a neural network, FRI generates 00:13:11.139 --> 00:13:13.559 rules like, if the stick is leaning slightly 00:13:13.559 --> 00:13:16.559 left and moving moderately fast, then move the 00:13:16.559 --> 00:13:18.679 base slightly left. So we can actually audit 00:13:18.679 --> 00:13:21.440 the AI's thought process while still allowing 00:13:21.440 --> 00:13:23.940 it to handle massive continuous environments. 00:13:23.980 --> 00:13:26.350 It's the best of both worlds. It allows the system 00:13:26.350 --> 00:13:29.389 to remain understandable while navigating spaces 00:13:29.389 --> 00:13:31.269 that would otherwise succumb to the curse of 00:13:31.269 --> 00:13:33.879 dimensionality. OK, so we've explored the core 00:13:33.879 --> 00:13:36.419 mechanics. We know how the Bellman equation updates 00:13:36.419 --> 00:13:38.759 its beliefs, how the learning levers manipulate 00:13:38.759 --> 00:13:41.200 its behavior, and how function approximation 00:13:41.200 --> 00:13:45.340 allows it to tackle infinite spaces. But why 00:13:45.340 --> 00:13:48.259 didn't we see AI dominating the world decades 00:13:48.259 --> 00:13:51.000 ago? Like, where did this actually lead historically? 00:13:51.279 --> 00:13:52.960 Well, the formal introduction of Q -learning 00:13:52.960 --> 00:13:55.820 is widely credited to Chris Watkins, who presented 00:13:55.820 --> 00:13:59.340 it in his 1989 doctoral dissertation, Learning 00:13:59.340 --> 00:14:02.840 from Delayed Rewards. 1989. Yes. And a few years 00:14:02.840 --> 00:14:06.360 later, in 1992, Watkins and Peter Dian published 00:14:06.360 --> 00:14:09.000 a formal convergence proof. They mathematically 00:14:09.000 --> 00:14:11.220 proved that given enough time and exploration, 00:14:11.740 --> 00:14:14.059 the algorithm will absolutely find the optimal 00:14:14.059 --> 00:14:16.600 policy. But the timeline has a hidden gem here. 00:14:16.980 --> 00:14:19.200 The source highlights a fascinating precursor 00:14:19.200 --> 00:14:21.720 that happened a full eight years before Watkins' 00:14:21.899 --> 00:14:24.519 dissertation back in 1981. A system called the 00:14:24.519 --> 00:14:27.559 Crossbar Adaptive Array, or CAA, created by S. 00:14:27.659 --> 00:14:29.929 Bozanowski. At the time, the challenge was framed 00:14:29.929 --> 00:14:32.750 as delayed reinforcement learning, but the CAA 00:14:32.750 --> 00:14:35.470 was solving the exact same problem. The architecture 00:14:35.470 --> 00:14:37.370 is what really surprised me. It wasn't just a 00:14:37.370 --> 00:14:39.210 vague theoretical precursor. It was practically 00:14:39.210 --> 00:14:42.429 a blueprint. The similarities are striking. The 00:14:42.429 --> 00:14:45.210 CAA used a memory matrix that functioned identically 00:14:45.210 --> 00:14:48.289 to a Q table. It introduced the concept of computing 00:14:48.289 --> 00:14:51.190 state values vertically and actions horizontally. 00:14:51.470 --> 00:14:53.960 Wow. And perhaps most interestingly, it borrowed 00:14:53.960 --> 00:14:56.440 the term secondary reinforcement from animal 00:14:56.440 --> 00:14:59.340 learning theory to model how the value of a successful 00:14:59.340 --> 00:15:03.299 state is mathematically back propagated to previously 00:15:03.299 --> 00:15:06.820 encountered situations. It was a pioneering demonstration 00:15:06.820 --> 00:15:09.379 of what would eventually become Q learning. And 00:15:09.379 --> 00:15:11.399 when you take those foundational concepts from 00:15:11.399 --> 00:15:13.659 the 80s and 90s and combine them with modern 00:15:13.659 --> 00:15:17.659 computing power, you arrive at 2014. Google DeepMind 00:15:17.659 --> 00:15:20.000 patents an application called Deep Reinforcement 00:15:20.000 --> 00:15:22.639 Learning or Deep Q Learning. A massive breakthrough. 00:15:22.919 --> 00:15:25.500 They point this new architecture at classic Atari 00:15:25.500 --> 00:15:28.500 2600 games, and the AI doesn't just figure out 00:15:28.500 --> 00:15:31.039 how to play them, it plays them at expert human 00:15:31.039 --> 00:15:33.580 levels, operating entirely from the raw pixels 00:15:33.580 --> 00:15:36.120 on the screen. DeepMind achieved this milestone 00:15:36.120 --> 00:15:38.620 by using a deep convolutional neural network 00:15:38.620 --> 00:15:41.820 to represent the Q function. But pairing Q learning 00:15:41.820 --> 00:15:44.559 with powerful nonlinear neural networks introduces 00:15:44.559 --> 00:15:47.029 a critical instability. This goes back to how 00:15:47.029 --> 00:15:50.590 the AI receives its data, right? If an AI is 00:15:50.590 --> 00:15:52.990 playing a video game, the frames it sees are 00:15:52.990 --> 00:15:54.870 highly correlated. Like if it is moving left 00:15:54.870 --> 00:15:57.269 for three seconds, it's just seeing a sequence 00:15:57.269 --> 00:15:59.690 of moving left images. And if the neural network 00:15:59.690 --> 00:16:02.169 is only learning from that immediate sequential 00:16:02.169 --> 00:16:05.690 feed, it begins to overcorrect. A sequence of 00:16:05.690 --> 00:16:08.710 moving left might cause the network to drastically 00:16:08.710 --> 00:16:11.370 alter its internal weights, essentially forgetting 00:16:11.370 --> 00:16:14.950 how to move right. Oh. The learning collapses. 00:16:15.370 --> 00:16:17.409 So how did DeepMind fix the memory collapse? 00:16:17.490 --> 00:16:19.850 They implemented a biologically inspired mechanism 00:16:19.850 --> 00:16:22.750 called experience replay. Experience replay. 00:16:23.029 --> 00:16:24.889 Instead of only updating its beliefs based on 00:16:24.889 --> 00:16:27.950 the very last frame it saw, the system constantly 00:16:27.950 --> 00:16:30.909 stores its past actions in a memory bank. During 00:16:30.909 --> 00:16:33.710 the learning phase, it pulls a random sample 00:16:33.710 --> 00:16:36.309 of those past experiences, a mix of moving left, 00:16:36.549 --> 00:16:38.649 moving right, winning, and losing, and learns 00:16:38.649 --> 00:16:41.029 from that shuffle deck. Wait, really? So to keep 00:16:41.029 --> 00:16:43.110 the neural network from becoming hopelessly biased 00:16:43.110 --> 00:16:45.870 by its immediate surroundings, the AI essentially 00:16:45.870 --> 00:16:48.289 sits around reminiscing about a random assortment 00:16:48.289 --> 00:16:50.850 of its past lives. It breaks the sequence so 00:16:50.850 --> 00:16:53.289 it doesn't get tunnel vision. Shuffling the deck 00:16:53.289 --> 00:16:55.929 removes the correlations in the observation sequence. 00:16:56.690 --> 00:17:00.450 It smooths out the data distribution, which stabilizes 00:17:00.450 --> 00:17:02.509 the neural network enough to allow it to master 00:17:02.509 --> 00:17:05.230 the Atari games. But even with experience replay, 00:17:05.509 --> 00:17:09.269 the AI still had a tendency to trip over its 00:17:09.269 --> 00:17:12.029 own feet in noisy environments. The article points 00:17:12.029 --> 00:17:15.109 out a persistent flaw where standard Q learning 00:17:15.109 --> 00:17:18.190 overestimates action values, drastically slowing 00:17:18.190 --> 00:17:20.690 down the learning process. The root cause of 00:17:20.690 --> 00:17:23.410 that overestimation lies in a structural flaw 00:17:23.410 --> 00:17:26.329 of the original math. Standard Q -learning uses 00:17:26.329 --> 00:17:29.009 the exact same mathematical function to select 00:17:29.009 --> 00:17:31.390 an action and to evaluate how good that action 00:17:31.390 --> 00:17:33.910 was. Okay. It inherently assumes that the highest 00:17:33.910 --> 00:17:36.950 estimated value is the most accurate. In an environment 00:17:36.950 --> 00:17:39.789 with random noise, a lucky spike might cause 00:17:39.789 --> 00:17:41.849 the AI to think a terrible move is actually brilliant. 00:17:42.029 --> 00:17:44.089 You described this earlier off mic as letting 00:17:44.089 --> 00:17:46.690 a student grade their own test. Like they're 00:17:46.690 --> 00:17:48.670 inherently going to be biased toward their own 00:17:48.670 --> 00:17:51.309 answers. So what's the structural fix? The fix 00:17:51.309 --> 00:17:53.809 is an algorithm variant called double Q -learning. 00:17:54.059 --> 00:17:57.480 You essentially split the AI's brain. You create 00:17:57.480 --> 00:18:00.160 two separate independent value functions. Let's 00:18:00.160 --> 00:18:03.700 call them QA and QB. You train them symmetrically 00:18:03.700 --> 00:18:06.000 using entirely separate sets of experiences. 00:18:06.279 --> 00:18:08.420 So they are learning independently. How do they 00:18:08.420 --> 00:18:10.680 interact when it's time to make a decision? When 00:18:10.680 --> 00:18:13.200 the agent needs to evaluate a future state, it 00:18:13.200 --> 00:18:15.839 uses QA to select what it thinks is the best 00:18:15.839 --> 00:18:19.259 action. But it doesn't let QA grade itself. It 00:18:19.259 --> 00:18:22.079 uses QB to evaluate the actual value of that 00:18:22.079 --> 00:18:24.609 selected action. Oh, that's clever. It creates 00:18:24.609 --> 00:18:28.289 a system of checks and balances. QA says, I think 00:18:28.289 --> 00:18:30.849 jumping right is the best move. And QB checks 00:18:30.849 --> 00:18:32.609 its own independent notes and says, hold on, 00:18:32.750 --> 00:18:34.769 my data says jumping right is actually terrible. 00:18:35.170 --> 00:18:37.410 It completely separates the decision maker from 00:18:37.410 --> 00:18:40.410 the evaluator. It neutralizes the overestimation 00:18:40.410 --> 00:18:42.970 bias. If we connect this to the bigger picture. 00:18:43.210 --> 00:18:45.509 When double -Q learning was eventually combined 00:18:45.509 --> 00:18:48.509 with deep neural networks in 2015, the resulting 00:18:48.509 --> 00:18:51.930 architecture, known as double -DQN, vastly outperformed 00:18:51.930 --> 00:18:54.549 the original Atari playing algorithms. It proved 00:18:54.549 --> 00:18:56.750 that by separating the selection and evaluation 00:18:56.750 --> 00:19:00.670 processes, AI could consistently conquer incredibly 00:19:00.670 --> 00:19:03.410 complex, noisy environments. Which brings us 00:19:03.410 --> 00:19:06.569 back to you, the listener. We've gone from a 00:19:06.569 --> 00:19:10.220 theoretical grid maze to mastering Atari. But 00:19:10.220 --> 00:19:12.779 these mechanisms aren't just abstract math, they 00:19:12.779 --> 00:19:15.380 mirror how we process the world. They absolutely 00:19:15.380 --> 00:19:18.339 do. Next time you are trying to learn a completely 00:19:18.339 --> 00:19:22.240 new skill -like, a new language, a complex piece 00:19:22.240 --> 00:19:25.380 of software, or a new sport, ask yourself what 00:19:25.380 --> 00:19:28.220 your internal learning rate is. Are you operating 00:19:28.220 --> 00:19:30.680 with a rate of zero, stubbornly ignoring new 00:19:30.680 --> 00:19:32.819 feedback because you think you already know best? 00:19:32.920 --> 00:19:35.079 Or are you operating at a rate of one? Exactly. 00:19:35.180 --> 00:19:37.779 Are you frantically throwing out your foundational 00:19:37.779 --> 00:19:39.940 basics every time you see a new trend on social 00:19:39.940 --> 00:19:42.900 media, finding that careful mathematical balance 00:19:42.900 --> 00:19:45.220 between retaining old wisdom and absorbing new 00:19:45.220 --> 00:19:48.440 discoveries is quite literally the secret algorithm 00:19:48.440 --> 00:19:50.920 to mastering the unknown. The source text does 00:19:50.920 --> 00:19:53.539 leave us with one final fascinating complication 00:19:53.539 --> 00:19:56.279 to ponder, found in a brief section on multi 00:19:56.279 --> 00:19:58.759 -agent learning. Oh yeah, this part is wild. 00:19:59.319 --> 00:20:01.740 Every scenario we have discussed today assumes 00:20:01.740 --> 00:20:04.640 the agent is alone. It assumes the environment 00:20:04.640 --> 00:20:07.910 is passive, just waiting to be solved. But what 00:20:07.910 --> 00:20:10.410 happens when you introduce multiple queue learning 00:20:10.410 --> 00:20:12.690 agents into the exact same environment? It stops 00:20:12.690 --> 00:20:15.250 being a solo puzzle. The entire foundation shifts. 00:20:15.769 --> 00:20:18.210 If a single agent learns the optimal path by 00:20:18.210 --> 00:20:20.849 assuming the world around it is static, how does 00:20:20.849 --> 00:20:23.190 it cope when the environment itself is constantly 00:20:23.190 --> 00:20:25.849 reacting? The other agents are simultaneously 00:20:25.849 --> 00:20:28.470 exploring, updating their own queue tables, and 00:20:28.470 --> 00:20:30.470 changing their behaviors based on what you do. 00:20:30.599 --> 00:20:34.180 the fixed mathematical horizon vanishes. It transforms 00:20:34.180 --> 00:20:36.920 from a static maze into a chaotic ecosystem. 00:20:37.279 --> 00:20:39.480 You go from a blank slate flailing in the dark 00:20:39.480 --> 00:20:41.920 to not only learning the rules of physics but 00:20:41.920 --> 00:20:44.420 having to anticipate the psychology of every 00:20:44.420 --> 00:20:47.039 other machine learning right alongside you. It's 00:20:47.039 --> 00:20:50.059 a complex game of digital chess and definitely 00:20:50.059 --> 00:20:52.160 something to ponder long after this deep dive 00:20:52.160 --> 00:20:52.380 ends.