WEBVTT 00:00:00.000 --> 00:00:02.960 So welcome back to the show. We are really thrilled 00:00:02.960 --> 00:00:05.280 to have you, the learner, joining us today for 00:00:05.280 --> 00:00:08.099 this deep dive. Because usually, when you think 00:00:08.099 --> 00:00:11.119 about a computer program, there is this stripped 00:00:11.119 --> 00:00:14.640 expectation of explicit instruction. Right, yeah, 00:00:14.660 --> 00:00:16.620 like it needs to be told exactly what to do. 00:00:16.859 --> 00:00:18.739 Exactly. It's like handing someone a recipe. 00:00:19.100 --> 00:00:20.620 You give the computer a list of ingredients. 00:00:20.679 --> 00:00:23.199 You tell it the exact chronological steps to 00:00:23.199 --> 00:00:25.620 bake the cake. And it bakes the cake. It's rigid. 00:00:25.879 --> 00:00:28.379 It's entirely predictable. Yeah, it operates 00:00:28.379 --> 00:00:31.140 purely on that classic if -then logic. I mean, 00:00:31.440 --> 00:00:33.899 if this specific condition is met, execute this 00:00:33.899 --> 00:00:36.520 specific command. The boundaries of the software 00:00:36.520 --> 00:00:38.859 are just completely defined by the human engineer 00:00:38.859 --> 00:00:40.840 who wrote the code in the first place. Right. 00:00:40.939 --> 00:00:43.939 But then you step into the world of artificial 00:00:43.939 --> 00:00:47.890 intelligence trying to navigate messy. Unpredictable 00:00:47.890 --> 00:00:50.429 reality and suddenly that recipe book is completely 00:00:50.429 --> 00:00:52.969 useless. Oh totally useless You're looking at 00:00:52.969 --> 00:00:55.969 a landscape where the computer has to somehow 00:00:55.969 --> 00:00:58.649 figure out the recipe all by itself Literally 00:00:58.649 --> 00:01:00.689 just by tasting the ingredients and interacting 00:01:00.689 --> 00:01:02.969 with the kitchen I mean we are talking about 00:01:02.969 --> 00:01:05.829 going from a computer playing backgammon in the 00:01:05.829 --> 00:01:10.400 early 90s to AI systems that can autonomously 00:01:10.400 --> 00:01:13.840 navigate stratospheric balloons on unpredictable 00:01:13.840 --> 00:01:16.560 wind currents. Which is just wild to think about. 00:01:16.700 --> 00:01:20.400 It is. And, you know, actively cutting Google's 00:01:20.400 --> 00:01:23.379 massive data center cooling bills by like... 00:01:23.069 --> 00:01:26.510 40%. It's a massive paradigm shift. We basically 00:01:26.510 --> 00:01:28.769 go from programming the rules of the world to 00:01:28.769 --> 00:01:31.010 programming the capacity for the machine to learn 00:01:31.010 --> 00:01:33.870 the rules of the world entirely on its own. And 00:01:33.870 --> 00:01:36.250 that shift is exactly what we are exploring with 00:01:36.250 --> 00:01:38.510 you today. The mission of this deep dive into 00:01:38.510 --> 00:01:40.709 the source material on deep reinforcement learning 00:01:40.709 --> 00:01:44.590 is to really demystify this incredibly complex 00:01:44.590 --> 00:01:46.870 world. We're going to break it down so you can 00:01:46.870 --> 00:01:49.549 grasp both the underlying mechanics and the monumental 00:01:49.549 --> 00:01:51.989 real -world implications, all without getting 00:01:51.989 --> 00:01:54.030 bogged down. down in the dense mathematics. Yeah 00:01:54.030 --> 00:01:56.209 we'll keep the math to a minimum. Okay let's 00:01:56.209 --> 00:01:59.489 unpack this. Where do we even start with a term 00:01:59.489 --> 00:02:02.750 as loaded as deep reinforcement learning? Well 00:02:02.750 --> 00:02:05.010 the absolute best place to start is honestly 00:02:05.010 --> 00:02:07.810 by looking at the name itself because it is fundamentally 00:02:07.810 --> 00:02:12.430 a mashup of two major highly successful subfields 00:02:12.430 --> 00:02:14.689 of machine learning. Right the two halves. Yeah 00:02:14.689 --> 00:02:17.129 you have deep learning on one side and reinforcement 00:02:17.129 --> 00:02:19.610 learning on the other and for a long time these 00:02:19.610 --> 00:02:22.060 were mostly separate tracks of research. Deep 00:02:22.060 --> 00:02:24.800 reinforcement learning, or deep RL, is basically 00:02:24.800 --> 00:02:26.800 what happens when you smash them together to 00:02:26.800 --> 00:02:29.120 solve the shortcomings of each. Right. So before 00:02:29.120 --> 00:02:31.479 we can understand how this mashup beats world 00:02:31.479 --> 00:02:35.060 champions at complex games, or dynamically manages 00:02:35.060 --> 00:02:37.120 a financial portfolio, we have to understand 00:02:37.120 --> 00:02:39.659 how it actually processes the world. Exactly. 00:02:39.740 --> 00:02:41.580 Let's look at the two ingredients, starting with 00:02:41.580 --> 00:02:44.639 the doing part reinforcement learning. Now. If 00:02:44.639 --> 00:02:48.120 you follow AI, you know basic RL is modeled on 00:02:48.120 --> 00:02:51.080 a Markov decision process. It's basically an 00:02:51.080 --> 00:02:53.240 agent making decisions through pure trial and 00:02:53.240 --> 00:02:55.900 error. Yeah, precisely. So at every time step, 00:02:56.039 --> 00:02:58.439 the agent observes a state, takes an action, 00:02:58.759 --> 00:03:01.520 and receives a scalar reward. It's trying to 00:03:01.520 --> 00:03:04.240 figure out a policy like a mapping from states 00:03:04.240 --> 00:03:07.840 to actions that maximizes its long -term return. 00:03:08.180 --> 00:03:11.180 But historically, traditional reinforcement learning 00:03:11.180 --> 00:03:14.870 hit a massive wall, right? the curse of dimensionality. 00:03:14.909 --> 00:03:17.870 Oh, a huge wall. Because it only works well when 00:03:17.870 --> 00:03:20.650 the state of the world is just a few clean, discreet. 00:03:21.150 --> 00:03:23.449 variables. Like the x and y coordinates on a 00:03:23.449 --> 00:03:26.110 small grid. Exactly. I mean, if you are using 00:03:26.110 --> 00:03:28.389 tabular Q -learning, the algorithm literally 00:03:28.389 --> 00:03:30.689 builds a spreadsheet of every possible state 00:03:30.689 --> 00:03:33.250 and every possible action. Wow, a literal spreadsheet. 00:03:33.430 --> 00:03:35.349 Yeah, but what happens when the real world gets 00:03:35.349 --> 00:03:37.330 messy? What if the state of the environment is, 00:03:37.330 --> 00:03:39.310 say, a high -definition video feed from a self 00:03:39.310 --> 00:03:41.849 -driving car? Oh man. Or the raw sensor stream 00:03:41.849 --> 00:03:44.439 from a robotic arm? The number of possible states 00:03:44.439 --> 00:03:46.520 becomes larger than the number of atoms in the 00:03:46.520 --> 00:03:49.759 universe. Traditional RL algorithms just choke 00:03:49.759 --> 00:03:52.460 on that high dimensional unstructured data because 00:03:52.460 --> 00:03:54.699 a human engineer would have to manually identify 00:03:54.699 --> 00:03:57.039 and hand code all the important features. And 00:03:57.039 --> 00:03:59.530 that is where the seeing part comes in. This 00:03:59.530 --> 00:04:01.509 is where deep learning solves the bottleneck. 00:04:01.710 --> 00:04:04.090 It does. Deep learning uses artificial neural 00:04:04.090 --> 00:04:07.610 networks to process that raw, messy data directly. 00:04:07.789 --> 00:04:11.449 Right. It acts as this phenomenal feature extractor, 00:04:11.710 --> 00:04:13.830 basically transforming massive sets of inputs 00:04:13.830 --> 00:04:16.449 into meaningful representations without any human 00:04:16.449 --> 00:04:19.629 handholding. So think of traditional reinforcement 00:04:19.629 --> 00:04:22.269 learning, like training a dog with treats. It 00:04:22.269 --> 00:04:25.490 learns to sit to get the reward. The inputs are 00:04:25.490 --> 00:04:28.290 simple and discreet. A voice command, a hand 00:04:28.290 --> 00:04:30.509 signal. Right, very simple. But deep reinforcement 00:04:30.509 --> 00:04:33.230 learning is like giving that dog high -definition 00:04:33.230 --> 00:04:35.709 vision and a radically complex brain so it can 00:04:35.709 --> 00:04:38.410 navigate a busy city street, avoiding traffic 00:04:38.410 --> 00:04:41.250 and reading pedestrian signals, all to find the 00:04:41.250 --> 00:04:43.509 ultimate treat. That is a highly effective way 00:04:43.509 --> 00:04:45.970 to visualize it, yeah. The magic of this mashup 00:04:45.970 --> 00:04:48.370 allows for what we call end -to -end reinforcement 00:04:48.370 --> 00:04:51.079 learning. Right. Instead of having one system 00:04:51.079 --> 00:04:53.160 process the vision and another system decide 00:04:53.160 --> 00:04:56.959 what to do, a single deep neural network handles 00:04:56.959 --> 00:05:00.199 the entire pipeline. It takes massive inputs, 00:05:00.480 --> 00:05:02.720 like every single pixel rendered on a video game 00:05:02.720 --> 00:05:05.639 screen, and maps them directly to the physical 00:05:05.639 --> 00:05:08.600 actions needed to optimize the objective. Wow. 00:05:09.600 --> 00:05:12.259 Now that we know how this system processes information, 00:05:13.360 --> 00:05:15.199 emerging trial and error with high dimensional 00:05:15.199 --> 00:05:17.819 sight. Let's look at how it proved its power 00:05:17.819 --> 00:05:20.660 to the world. Yeah. Because you obviously can't 00:05:20.660 --> 00:05:23.319 just unleash an untrained trial and error AI 00:05:23.319 --> 00:05:26.420 agent into a real busy city street or a power 00:05:26.420 --> 00:05:28.540 grid. Oh, definitely not. The stakes are way 00:05:28.540 --> 00:05:30.699 too high for that. Yeah, you need a highly controlled 00:05:30.699 --> 00:05:33.040 safe environment where the agent can fail millions 00:05:33.040 --> 00:05:35.500 of times at a hyper accelerated speed without 00:05:35.500 --> 00:05:37.500 breaking anything physical. And the absolute 00:05:37.500 --> 00:05:39.279 best proving ground for that is video games. 00:05:39.360 --> 00:05:41.800 It really is. It's the perfect laboratory. Which 00:05:41.800 --> 00:05:43.920 brings us to the origins. The sources point out 00:05:43.920 --> 00:05:46.560 that this actually started way back in 1992 with 00:05:46.560 --> 00:05:49.699 a program called TD Gammon. Yes, TD Gammon is 00:05:49.699 --> 00:05:52.680 an absolute landmark in the field. It was this 00:05:52.680 --> 00:05:55.199 computer program developed to play backgammon 00:05:55.199 --> 00:05:57.680 using an early form of reinforcement learning 00:05:57.680 --> 00:06:00.379 combined with a basic neural network. And what 00:06:00.379 --> 00:06:02.459 made it so remarkable was its input mechanism, 00:06:02.519 --> 00:06:06.199 right? Exactly. It used just 198 input signals, 00:06:06.379 --> 00:06:08.800 which literally simply represented the number 00:06:08.800 --> 00:06:11.180 of pieces of a given color at a given location 00:06:11.180 --> 00:06:14.360 on the board. gave it zero built -in knowledge 00:06:14.360 --> 00:06:17.259 about backgammon strategy. Zero. It just played 00:06:17.259 --> 00:06:19.740 against itself over and over. And through that 00:06:19.740 --> 00:06:23.100 pure self -play, evaluating board positions to 00:06:23.100 --> 00:06:25.720 maximize its chance of winning, it learned to 00:06:25.720 --> 00:06:27.920 play at a strong intermediate level. It figured 00:06:27.920 --> 00:06:30.160 out the strategy just from the mathematical layout 00:06:30.160 --> 00:06:32.360 of the board. Yeah. That's incredible. Yeah. 00:06:32.839 --> 00:06:35.639 But fast forward to 2013, and we get the true 00:06:35.639 --> 00:06:37.850 pixel revolution. This is where DeepMind steps 00:06:37.850 --> 00:06:40.689 in and changes the entire landscape of AI with 00:06:40.689 --> 00:06:43.110 Atari games. Yeah, this was the moment DeepRL 00:06:43.110 --> 00:06:46.290 truly arrived on the global stage. DeepMind created 00:06:46.290 --> 00:06:49.850 the DeepQ network, or DQN. The massive leap here 00:06:49.850 --> 00:06:52.829 was that they didn't feed the AI neat pre -processed 00:06:52.829 --> 00:06:54.850 information about where the game sprites were. 00:06:55.589 --> 00:06:57.990 Yeah, they didn't tell it like the ball was at 00:06:57.990 --> 00:06:59.949 coordinate X and the paddle was at coordinate 00:06:59.949 --> 00:07:04.209 Y. They just fed the neural network raw RGB pixels. 00:07:04.430 --> 00:07:06.629 So it was literally just looking at the screen 00:07:06.629 --> 00:07:09.509 like a human would. Basically, yes. The sources 00:07:09.509 --> 00:07:12.310 note it was specifically four stacked frames 00:07:12.310 --> 00:07:16.629 of 84 by 84 pixels plus the game score. And the 00:07:16.629 --> 00:07:19.189 four frames part is crucial, right? Because a 00:07:19.189 --> 00:07:21.329 single static frame doesn't tell you which direction 00:07:21.329 --> 00:07:24.509 a ball is moving. By stacking four frames, the 00:07:24.509 --> 00:07:27.149 neural network could automatically infer velocity 00:07:27.149 --> 00:07:30.250 and trajectory. Exactly. It had to derive the 00:07:30.250 --> 00:07:32.750 physics of the game entirely from pixel changes 00:07:32.750 --> 00:07:35.589 over time. And the crazy part is, using the exact 00:07:35.589 --> 00:07:38.209 same network architecture without tweaking the 00:07:38.209 --> 00:07:40.930 code for different games, this AI learned to 00:07:40.930 --> 00:07:45.180 play 49 different Atari games. Wow. 49 games 00:07:45.180 --> 00:07:47.420 with one architecture. Yeah, it outperformed 00:07:47.420 --> 00:07:49.819 competing methods on almost all of them and performed 00:07:49.819 --> 00:07:52.639 at a level comparable or superior to a professional 00:07:52.639 --> 00:07:55.540 human game tester. I mean, in the game Breakout, 00:07:55.740 --> 00:07:57.920 it spontaneously discovered the strategy of tunneling 00:07:57.920 --> 00:07:59.500 through the side of the wall to bounce the ball 00:07:59.500 --> 00:08:01.439 behind the bricks for maximum points. Nobody 00:08:01.439 --> 00:08:04.439 programmed it to do that. No! It just learned 00:08:04.439 --> 00:08:07.139 that that specific sequence of actions maximized 00:08:07.139 --> 00:08:09.420 the expected future reward. Here's where it gets 00:08:09.420 --> 00:08:12.100 really interesting. Because Atari is a fully 00:08:12.100 --> 00:08:15.199 observable, relatively simple environment. But 00:08:15.199 --> 00:08:18.639 by 2015, we get AlphaGo. Oh, AlphaGo, yeah. The 00:08:18.639 --> 00:08:21.459 first computer to beat a human professional at 00:08:21.459 --> 00:08:24.959 Go. We're talking about a 19 by 19 board game 00:08:24.959 --> 00:08:28.220 so mathematically complex that the branching 00:08:28.220 --> 00:08:30.980 factor of possible moves exceeds the number of 00:08:30.980 --> 00:08:33.559 atoms in the observable universe. It's just staggering. 00:08:33.879 --> 00:08:35.779 You cannot brute force a search tree for go. 00:08:36.120 --> 00:08:38.000 The deep RL agent had to develop what almost 00:08:38.000 --> 00:08:40.460 looks like human intuition to evaluate board 00:08:40.460 --> 00:08:43.419 states, and it didn't stop there. AI quickly 00:08:43.419 --> 00:08:45.960 started dominating multiplayer in Perfect Information 00:08:45.960 --> 00:08:47.820 games. Right, games where you don't even know 00:08:47.820 --> 00:08:49.940 the full state of the board. Exactly. OpenAI 00:08:49.940 --> 00:08:52.919 5 -beat world champions at Dota 2, and a program 00:08:52.919 --> 00:08:54.980 called Pluribus Beat Professionals at No Limit 00:08:54.980 --> 00:08:57.559 Texas Hold 'em, which involves mastering bluffing 00:08:57.559 --> 00:08:59.820 and hidden cards. What's fascinating here is 00:08:59.820 --> 00:09:02.000 that the real breakthrough wasn't just that a 00:09:02.000 --> 00:09:04.539 machine was winning games. I mean, we've had 00:09:04.539 --> 00:09:07.120 chess bots like Deep Blue that could beat humans 00:09:07.120 --> 00:09:11.039 for decades using alpha beta pruning and sheer 00:09:11.039 --> 00:09:13.120 computational search. Right. Deep Blue is basically 00:09:13.120 --> 00:09:15.860 just doing a lot of fast math. Exactly. The breakthrough 00:09:15.860 --> 00:09:18.799 here was the generalization. The Deep RL agent 00:09:18.799 --> 00:09:21.860 wasn't explicitly programmed with the rules of 00:09:21.860 --> 00:09:25.759 Go or the hand rankings of poker or the spell 00:09:25.759 --> 00:09:29.470 cooldowns of Dota 2. It was programmed to learn. 00:09:29.570 --> 00:09:32.389 It learned how to learn. Yes, it means the exact 00:09:32.389 --> 00:09:35.149 same fundamental mathematical architecture that 00:09:35.149 --> 00:09:37.710 learns to play Space Invaders can be applied 00:09:37.710 --> 00:09:40.889 to completely different, infinitely more complex 00:09:40.889 --> 00:09:44.070 domains. Okay, so it learns, but we really need 00:09:44.070 --> 00:09:46.370 to look under the hood here. Games are great, 00:09:46.809 --> 00:09:49.330 but the real world is infinitely more complex 00:09:49.330 --> 00:09:51.629 than a go board. Oh, definitely. How does the 00:09:51.629 --> 00:09:54.409 AI actually learn these rules without a human 00:09:54.409 --> 00:09:57.080 explicitly coding them? This takes us to the 00:09:57.080 --> 00:09:59.419 two main algorithmic approaches these systems 00:09:59.419 --> 00:10:02.059 use to navigate their environments, model -based 00:10:02.059 --> 00:10:03.820 and model -free learning. Yeah, those are the 00:10:03.820 --> 00:10:06.519 big two. Let's start with model -based. In model 00:10:06.519 --> 00:10:09.720 -based DeepRL, the AI attempts to explicitly 00:10:09.720 --> 00:10:12.139 understand the rules of the world it operates 00:10:12.139 --> 00:10:15.139 in. It tries to estimate a forward model of the 00:10:15.139 --> 00:10:17.559 environment's dynamics. Like it builds its own 00:10:17.559 --> 00:10:20.539 internal physics engine. Essentially, yes. It 00:10:20.539 --> 00:10:22.899 uses supervised learning to predict what will 00:10:22.899 --> 00:10:26.840 happen next. The AI mathematically states, if 00:10:26.840 --> 00:10:29.399 I am in this current state and I execute this 00:10:29.399 --> 00:10:32.059 specific action, I predict the world will transition 00:10:32.059 --> 00:10:35.000 to this new state and yield this specific reward. 00:10:35.039 --> 00:10:37.980 OK. Once it builds this internal physics engine 00:10:37.980 --> 00:10:40.720 or model of how the world works, it can plan 00:10:40.720 --> 00:10:43.279 its actions virtually, looking multiple steps 00:10:43.279 --> 00:10:45.759 ahead to find the best path before actually making 00:10:45.759 --> 00:10:47.679 a move. Wait, I have to jump in and push back 00:10:47.679 --> 00:10:50.480 on this. Sure. If the AI is just guessing how 00:10:50.480 --> 00:10:52.580 the environment works based on a learned model, 00:10:52.740 --> 00:10:54.990 doesn't it completely fall apart if the real 00:10:54.990 --> 00:10:57.409 world throws a curveball that diverges from its 00:10:57.409 --> 00:10:59.850 prediction? Yeah, that's the big issue. I mean 00:10:59.850 --> 00:11:02.389 the real world isn't a clean, predictable physics 00:11:02.389 --> 00:11:05.129 engine like a video game. There is wind resistance, 00:11:05.470 --> 00:11:08.149 hardware friction, unexpected obstacles. If the 00:11:08.149 --> 00:11:11.009 internal model is even slightly wrong, wouldn't 00:11:11.009 --> 00:11:13.370 that error compound with every step? That is 00:11:13.370 --> 00:11:15.350 an excellent point, and you are hitting on the 00:11:15.350 --> 00:11:17.730 exact vulnerability of model -based systems. 00:11:18.409 --> 00:11:21.250 The true environment dynamics almost always diverge 00:11:21.250 --> 00:11:23.309 from the learned dynamics. So it just breaks? 00:11:23.830 --> 00:11:25.990 Well... Because of this compounding error, a 00:11:25.990 --> 00:11:28.870 model -based agent often has to constantly replan 00:11:28.870 --> 00:11:31.789 as it carries out actions. It's computationally 00:11:31.789 --> 00:11:34.970 exhausting, and it's prone to cascading failures 00:11:34.970 --> 00:11:37.730 if the initial model isn't highly accurate. And 00:11:37.730 --> 00:11:39.970 that is exactly why researchers developed the 00:11:39.970 --> 00:11:42.570 alternative model -free algorithms. So in a model 00:11:42.570 --> 00:11:45.440 -free approach, the AI just skips the physics 00:11:45.440 --> 00:11:48.399 engine entirely. Yes, entirely. In a model -free 00:11:48.399 --> 00:11:51.120 system, the AI does not even try to explicitly 00:11:51.120 --> 00:11:53.919 model the world's dynamics. It doesn't care about 00:11:53.919 --> 00:11:55.879 predicting the wind or calculating the friction. 00:11:55.980 --> 00:11:58.240 OK. It skips the middleman and just learns a 00:11:58.240 --> 00:12:00.919 direct policy, or it learns a Q function, which 00:12:00.919 --> 00:12:03.120 directly estimates future returns through brute 00:12:03.120 --> 00:12:06.019 force experience. It basically just maps states 00:12:06.019 --> 00:12:08.460 to actions by saying, when I see this exact pixel 00:12:08.460 --> 00:12:11.340 arrangement, I move the joystick left. Because 00:12:11.340 --> 00:12:13.980 historically, across a million games, That specific 00:12:13.980 --> 00:12:16.679 action in this specific state gets me a high 00:12:16.679 --> 00:12:19.899 score. So it's pure reactive pattern recognition. 00:12:20.460 --> 00:12:22.480 Essentially. But that sounds mathematically chaotic, 00:12:23.100 --> 00:12:25.919 like really unstable. Oh, it can be. Doing this 00:12:25.919 --> 00:12:28.940 by directly estimating the policy gradient, which 00:12:28.940 --> 00:12:32.889 is... The mathematical curve the AI follows to 00:12:32.889 --> 00:12:35.669 improve its policy often suffers from extremely 00:12:35.669 --> 00:12:37.649 high variance. What does that mean in practice? 00:12:37.850 --> 00:12:41.169 It means it can be highly unstable. If the AI 00:12:41.169 --> 00:12:43.690 randomly tries a bad move and gets a terrible 00:12:43.690 --> 00:12:46.889 score, a standard policy gradient might overreact 00:12:46.889 --> 00:12:49.389 and drastically alter the neural network weights. 00:12:49.950 --> 00:12:52.250 It could suddenly forget a perfectly good strategy 00:12:52.250 --> 00:12:54.610 it had already learned. But the sources note 00:12:54.610 --> 00:12:57.269 that newer, highly influential algorithms have 00:12:57.269 --> 00:12:59.840 been developed to stabilize this process. The 00:12:59.840 --> 00:13:02.620 big one that comes up is PPO, or proximal policy 00:13:02.620 --> 00:13:04.899 optimization. Yeah, PPO is huge right now. And 00:13:04.899 --> 00:13:07.639 the mechanism behind PPO is fascinating. It essentially 00:13:07.639 --> 00:13:10.200 mathematically clips the updates to the AI's 00:13:10.200 --> 00:13:12.039 brain. Right, it sets the speed limit on learning. 00:13:12.539 --> 00:13:15.840 Exactly. If the AI discovers a new action that 00:13:15.840 --> 00:13:19.600 seems amazing, PPO restricts how much the algorithm 00:13:19.600 --> 00:13:22.620 can change its policy at one time. It forces 00:13:22.620 --> 00:13:25.330 the AI to stay proximal. or close to his old 00:13:25.330 --> 00:13:28.789 policy, taking small, safe learning steps rather 00:13:28.789 --> 00:13:31.210 than taking a massive catastrophic leap based 00:13:31.210 --> 00:13:34.370 on one weird data point. And that clipping mechanism 00:13:34.370 --> 00:13:37.830 is exactly why PPO has become the default reinforcement 00:13:37.830 --> 00:13:40.429 learning algorithm for so many massive AI projects 00:13:40.429 --> 00:13:42.549 today, including training the large language 00:13:42.549 --> 00:13:44.570 models we use every day. It just beautifully 00:13:44.570 --> 00:13:47.110 balances ease of tuning with reliable, stable 00:13:47.110 --> 00:13:49.289 learning. But whether it's building an internal 00:13:49.289 --> 00:13:51.409 model of the world or going completely model 00:13:51.409 --> 00:13:54.110 free with PPO, there is a fundamental tension 00:13:54.110 --> 00:13:57.090 here. How does the AI know when to try something 00:13:57.090 --> 00:14:00.070 new versus sticking to a strategy that kind of 00:14:00.070 --> 00:14:02.690 works? This raises an important question. As 00:14:02.690 --> 00:14:05.049 the agent gets better, how do we systematically 00:14:05.049 --> 00:14:07.590 incentivize a machine to keep exploring without 00:14:07.590 --> 00:14:09.730 completely derailing its progress? Right. You 00:14:09.730 --> 00:14:11.929 were talking about the exploration versus exploitation 00:14:11.929 --> 00:14:14.049 trade -off, which is easily one of the most heavily 00:14:14.049 --> 00:14:16.590 researched dilemmas in all of DeepRO. Because 00:14:16.590 --> 00:14:19.149 if it only exploits its current knowledge, it 00:14:19.149 --> 00:14:21.450 might find a strategy that scores 10 points and 00:14:21.450 --> 00:14:23.389 just do that forever, right? Yeah. Yeah, safely 00:14:23.389 --> 00:14:25.970 optimizing a mediocre result. Never realizing 00:14:25.970 --> 00:14:27.669 there's a button right next to it that scores 00:14:27.669 --> 00:14:30.629 a thousand points. Yeah. But if it only explores, 00:14:30.950 --> 00:14:33.110 it just behaves randomly forever and never actually 00:14:33.110 --> 00:14:35.990 achieves the goal. Exactly. Agents typically 00:14:35.990 --> 00:14:39.009 start by acting entirely randomly, just exploring 00:14:39.009 --> 00:14:41.909 the state space. But as they learn, they narrow 00:14:41.909 --> 00:14:44.750 down. So, to solve the problem of getting stuck 00:14:44.750 --> 00:14:47.330 in local optimums, researchers have to literally 00:14:47.330 --> 00:14:50.490 force the AI to explore. And one of the most 00:14:50.490 --> 00:14:53.250 fascinating workarounds is programming curiosity. 00:14:53.629 --> 00:14:55.909 This blew my mind. They literally mathematically 00:14:55.909 --> 00:14:59.289 engineer curiosity. They do. In curiosity -driven 00:14:59.289 --> 00:15:01.730 exploration, researchers modify the underlying 00:15:01.730 --> 00:15:04.190 loss function. They give the AI an intrinsic 00:15:04.190 --> 00:15:06.590 reward for prediction errors. Meaning? Meaning, 00:15:06.789 --> 00:15:08.929 the AI continuously tries to predict what the 00:15:08.929 --> 00:15:10.830 next state will look like. If it encounters a 00:15:10.830 --> 00:15:12.990 state it cannot accurately predict, it means 00:15:12.990 --> 00:15:15.690 it has found something novel. The algorithm actually 00:15:15.690 --> 00:15:18.370 rewards the AI for this unpredictability. It 00:15:18.370 --> 00:15:21.210 gets a mathematical dopamine hit simply for seeking 00:15:21.210 --> 00:15:24.809 out unknown outcomes. Yes. Regardless of whether 00:15:24.809 --> 00:15:28.070 it immediately helps win the game, this intrinsic 00:15:28.070 --> 00:15:30.730 motivation pushes the agent out of its comfort 00:15:30.730 --> 00:15:34.210 zone to find better, hidden solutions in complex 00:15:34.210 --> 00:15:37.090 environments. It's giving the AI a mathematical 00:15:37.090 --> 00:15:39.980 sense of wonder. I love that. But they don't 00:15:39.980 --> 00:15:42.419 just learn from curiosity. They also learn by 00:15:42.419 --> 00:15:45.240 watching masters, which brings us to inverse 00:15:45.240 --> 00:15:47.580 reinforcement learning. Right. Because defining 00:15:47.580 --> 00:15:50.799 a reward function is incredibly hard. Like, if 00:15:50.799 --> 00:15:53.039 you tell an AI to drive to the store as fast 00:15:53.039 --> 00:15:55.379 as possible, it might drive on the sidewalk and 00:15:55.379 --> 00:15:57.580 run over mailboxes because you didn't explicitly 00:15:57.580 --> 00:16:00.000 penalize that in the math. Yeah, it takes your 00:16:00.000 --> 00:16:02.539 instructions very literally. Inverse RL solves 00:16:02.539 --> 00:16:05.159 this by treating the AI like an apprentice. Yes. 00:16:05.399 --> 00:16:08.059 Inverse RL flips the entire script. Instead of 00:16:08.059 --> 00:16:10.019 giving the agent a handcrafted reward function 00:16:10.019 --> 00:16:12.220 and saying, you know, optimize this, the agent 00:16:12.220 --> 00:16:15.200 watches a human demonstrator. OK. By observing 00:16:15.200 --> 00:16:18.139 the human's behavior, the AI tries to reverse 00:16:18.139 --> 00:16:20.940 engineer and infer what the underlying reward 00:16:20.940 --> 00:16:23.220 function actually is. It's like a master chef 00:16:23.220 --> 00:16:25.679 tasting a competitor's signature dish. Oh, I 00:16:25.679 --> 00:16:27.580 like that. They don't have the recipe, and they 00:16:27.580 --> 00:16:29.379 aren't told what makes it good. They have to 00:16:29.379 --> 00:16:32.139 reverse engineer the exact ingredients, ratios, 00:16:32.340 --> 00:16:34.879 and cooking temperatures just by observing the 00:16:34.879 --> 00:16:37.159 final flavor profile. That's a great analogy. 00:16:37.470 --> 00:16:40.549 It learns the subtle, unwritten rules of the 00:16:40.549 --> 00:16:43.490 goal by watching the expert achieve it, rather 00:16:43.490 --> 00:16:47.129 than relying on flawed, hand -coded reward signals. 00:16:47.490 --> 00:16:49.129 And then there's hindsight experience replay, 00:16:49.490 --> 00:16:52.450 or AR, which might actually be my favorite concept 00:16:52.450 --> 00:16:54.690 in this entire deep dive. It's a really clever 00:16:54.690 --> 00:16:57.649 approach. This is how an AI learns from completely 00:16:57.649 --> 00:17:01.070 failing. Let's say the AI is controlling a physical 00:17:01.070 --> 00:17:03.509 robotic arm and the goal is to pick up a red 00:17:03.509 --> 00:17:06.369 block. The AI swings the arm, completely misses 00:17:06.369 --> 00:17:08.630 the red block, but accidentally knocks over a 00:17:08.630 --> 00:17:11.250 blue cylinder. Right. In traditional RL, that's 00:17:11.250 --> 00:17:13.849 just a failure. Zero reward. The data is thrown 00:17:13.849 --> 00:17:16.549 away. But with hindsight experience replay, the 00:17:16.549 --> 00:17:20.589 system retroactively moves the goalpost. It relabels 00:17:20.589 --> 00:17:22.589 the attempt in hindsight. It changes what it 00:17:22.589 --> 00:17:25.769 was trying to do. Yeah. It says, OK, we failed 00:17:25.769 --> 00:17:28.930 to pick up the red block, but we just executed 00:17:28.930 --> 00:17:31.210 a mathematically perfect sequence of actions 00:17:31.210 --> 00:17:34.869 for knocking over a blue cylinder. and it stores 00:17:34.869 --> 00:17:37.269 that knowledge. It's a massive leap in sample 00:17:37.269 --> 00:17:39.970 efficiency. It turns every single mistake into 00:17:39.970 --> 00:17:42.450 a successful demonstration of a different goal. 00:17:42.789 --> 00:17:44.789 So if it ever needs to knock over a cylinder 00:17:44.789 --> 00:17:47.609 in the future, it already has the policy cash. 00:17:47.730 --> 00:17:50.349 These advanced techniques, the intrinsic curiosity, 00:17:50.869 --> 00:17:53.490 the apprenticeship of inverse RL, the profound 00:17:53.490 --> 00:17:56.130 efficiency of learning from hindsight. These 00:17:56.130 --> 00:17:58.289 are exactly the mathematical tools that allow 00:17:58.289 --> 00:18:01.910 deep RL to escape virtual game boards and tackle 00:18:01.910 --> 00:18:04.829 messy, high stakes, real world problems. Which 00:18:04.829 --> 00:18:07.569 is where the entire field is focusing its energy 00:18:07.569 --> 00:18:10.490 right now. We're moving from the screen to physical 00:18:10.490 --> 00:18:12.990 reality, crossing what researchers call the sim 00:18:12.990 --> 00:18:15.369 to real gap. Let's talk about some of those physical 00:18:15.369 --> 00:18:17.349 applications, because they are just staggering 00:18:17.349 --> 00:18:20.049 the robotics alone. Yeah, the robotics are incredible. 00:18:20.430 --> 00:18:23.029 OpenAI trained a robotic hand equipped with a 00:18:23.029 --> 00:18:26.109 deep RL brain to autonomously solve a physical 00:18:26.109 --> 00:18:28.930 Rubik's Cube, adapting to the friction and physical 00:18:28.930 --> 00:18:31.369 constraints of the real world. Just unbelievable. 00:18:31.670 --> 00:18:35.769 And a project called Loon. used deep RL to navigate 00:18:35.769 --> 00:18:39.309 high -altitude stratospheric balloons, continuously 00:18:39.309 --> 00:18:42.250 adjusting to incredibly complex, unpredictable 00:18:42.250 --> 00:18:44.950 atmospheric wind currents to provide internet 00:18:44.950 --> 00:18:48.109 access. But you noted one application from the 00:18:48.109 --> 00:18:50.369 source is that it has massive implications for 00:18:50.369 --> 00:18:53.119 global sustainability. Yes, the DeepMind data 00:18:53.119 --> 00:18:56.220 center project. So Google's data centers require 00:18:56.220 --> 00:18:59.119 immense amounts of energy, specifically for the 00:18:59.119 --> 00:19:01.500 massive industrial cooling systems required to 00:19:01.500 --> 00:19:04.480 keep the servers from melting. Right. It is a 00:19:04.480 --> 00:19:07.279 highly complex, nonlinear thermodynamic environment. 00:19:07.900 --> 00:19:11.039 DeepMind took a deep RL agent and let it analyze 00:19:11.039 --> 00:19:13.539 the historical data, all the temperatures, pump 00:19:13.539 --> 00:19:16.180 speeds, power consumption metrics. OK, so it 00:19:16.180 --> 00:19:17.940 learned the system. Yeah. And then the agent 00:19:17.940 --> 00:19:20.519 continuously interacted with the system, adjusting 00:19:20.519 --> 00:19:23.240 cool. configurations in real time, treating the 00:19:23.240 --> 00:19:25.980 facility like a massive continuous optimization 00:19:25.980 --> 00:19:28.380 game. And the result was it reduced Google's 00:19:28.380 --> 00:19:31.539 data center cooling bill by an astounding 40%. 00:19:31.539 --> 00:19:33.920 40%. Not through hardware upgrades, but purely 00:19:33.920 --> 00:19:37.000 by letting an AI figure out the optimal thermodynamic 00:19:37.000 --> 00:19:39.809 flow through trial and error. And the applications 00:19:39.809 --> 00:19:42.849 don't stop with physical systems or thermodynamics. 00:19:43.670 --> 00:19:46.049 The sources highlight a rapidly growing area 00:19:46.049 --> 00:19:49.390 of research deep RL for financial decision -making. 00:19:49.589 --> 00:19:51.269 Now that is interesting because Wall Street has 00:19:51.269 --> 00:19:54.269 been using quantitative algorithms for decades. 00:19:54.710 --> 00:19:56.789 Yeah. Wait, how is deep RL different from the 00:19:56.789 --> 00:20:00.000 trading bots that already exist? Well... Traditional 00:20:00.000 --> 00:20:02.500 financial approaches like modern portfolio theory 00:20:02.500 --> 00:20:05.500 rely heavily on static mean variance optimization. 00:20:06.059 --> 00:20:08.400 They look at historical averages to balance risk 00:20:08.400 --> 00:20:11.079 and return, basically assuming market returns 00:20:11.079 --> 00:20:13.380 follow a normal distribution. But the stock market 00:20:13.380 --> 00:20:16.000 isn't static. No, and it definitely doesn't follow 00:20:16.000 --> 00:20:19.099 normal distributions. It is wildly volatile with 00:20:19.099 --> 00:20:21.740 heavy tailed risks and non -stationary dynamics. 00:20:22.299 --> 00:20:24.359 Those traditional models struggle to adapt when 00:20:24.359 --> 00:20:26.779 the market behaves unexpectedly. Right. Deep 00:20:26.779 --> 00:20:29.259 RL, on the other hand, treats the entire stock 00:20:29.259 --> 00:20:32.140 market like a dynamic environment. Specifically, 00:20:32.420 --> 00:20:35.259 it frames it as a partially observable Markov 00:20:35.259 --> 00:20:39.220 decision process, or POMDP. Meaning the AI knows 00:20:39.220 --> 00:20:41.359 it doesn't have all the information. It can't 00:20:41.359 --> 00:20:43.059 see the hidden state of the market, just like 00:20:43.059 --> 00:20:45.099 you can't see the hidden cards in a game of poker. 00:20:45.390 --> 00:20:48.509 Exactly. It continuously interacts with the evolving 00:20:48.509 --> 00:20:51.630 financial data using its neural network to extract 00:20:51.630 --> 00:20:54.869 features from the noise. And crucially, advanced 00:20:54.869 --> 00:20:58.410 deep RL algorithms allow for continuous action 00:20:58.410 --> 00:21:00.829 spaces. Right. It's not just a discrete action 00:21:00.829 --> 00:21:03.049 like deciding to press buy or sell. Exactly. 00:21:03.250 --> 00:21:05.609 A continuous action space means the algorithm 00:21:05.609 --> 00:21:08.269 is deciding exactly what microscopic fraction 00:21:08.269 --> 00:21:11.130 of a percent of a portfolio to allocate to a 00:21:11.130 --> 00:21:13.869 specific asset at a specific microsecond. Wow. 00:21:14.860 --> 00:21:17.660 rebalances to maximize long -term returns. It 00:21:17.660 --> 00:21:20.660 is constant fluid adaptation to market shocks. 00:21:21.200 --> 00:21:23.140 So what does this all mean? When you step back 00:21:23.140 --> 00:21:25.460 and look at the sheer breadth of this technology, 00:21:25.920 --> 00:21:28.220 it becomes incredibly clear that deep reinforcement 00:21:28.220 --> 00:21:30.220 learning isn't just a parlor trick for beating 00:21:30.220 --> 00:21:32.779 grandmasters at chess or racking up high scores 00:21:32.779 --> 00:21:35.319 in breakout. No, definitely not. It is a fundamental 00:21:35.319 --> 00:21:38.730 dynamic decision -making engine. It thrives on 00:21:38.730 --> 00:21:40.549 continuous adaptation, whether it is steering 00:21:40.549 --> 00:21:42.630 a self -driving car through chaotic intersection, 00:21:43.190 --> 00:21:45.309 optimizing the thermodynamics of a massive server 00:21:45.309 --> 00:21:48.029 farm, or managing a complex retirement fund in 00:21:48.029 --> 00:21:50.579 a volatile global market. It really represents 00:21:50.579 --> 00:21:52.980 the transition from machines that merely compute 00:21:52.980 --> 00:21:55.259 instructions to machines that adapt to their 00:21:55.259 --> 00:21:57.880 environments. Let's do a quick recap of the journey 00:21:57.880 --> 00:22:00.220 we've taken you on today. We started with the 00:22:00.220 --> 00:22:02.519 basic ingredients, taking the pure trial and 00:22:02.519 --> 00:22:05.299 error point -scoring system of traditional reinforcement 00:22:05.299 --> 00:22:08.400 learning and supercharging it with the high -dimensional 00:22:08.400 --> 00:22:11.539 feature extraction of deep learning neural networks. 00:22:11.920 --> 00:22:14.000 Yeah, and we saw how it conquered the pixelated 00:22:14.000 --> 00:22:16.619 worlds of Atari by stacking frames to understand 00:22:16.619 --> 00:22:19.369 physics and then mastered the inf - complexity 00:22:19.369 --> 00:22:22.829 of Go, proving that a single algorithmic architecture 00:22:22.829 --> 00:22:25.869 could learn almost anything. We explored the 00:22:25.869 --> 00:22:28.849 complex algorithmic engine underneath, balancing 00:22:28.849 --> 00:22:31.089 the computationally heavy model -based planning 00:22:31.089 --> 00:22:33.730 against the brute force efficiency of model -free 00:22:33.730 --> 00:22:36.670 learning algorithms like PPO, which stabilize 00:22:36.670 --> 00:22:39.190 the learning process. And we looked at how researchers 00:22:39.190 --> 00:22:41.589 are solving the exploration -exploitation dilemma 00:22:41.589 --> 00:22:44.630 by artificially instilling curiosity and teaching 00:22:44.630 --> 00:22:47.589 agents to reverse engineer human mastery. Which 00:22:47.589 --> 00:22:50.069 all culminates in AI escaping the simulation, 00:22:50.730 --> 00:22:52.829 manipulating Rubik's cubes, cooling our internet 00:22:52.829 --> 00:22:55.470 infrastructure, and navigating the hidden variables 00:22:55.470 --> 00:22:57.970 of global finance. If we connect this to the 00:22:57.970 --> 00:23:00.950 bigger picture, the true promise of DeepRL is 00:23:00.950 --> 00:23:04.220 generalization. It is the unprecedented ability 00:23:04.220 --> 00:23:07.259 of a system to face completely unseen inputs, 00:23:07.960 --> 00:23:10.460 messy reality that it has never encountered before, 00:23:11.180 --> 00:23:13.339 and figure out the optimal path forward without 00:23:13.339 --> 00:23:16.059 a human having to handhold it or code a specific 00:23:16.059 --> 00:23:18.779 rule for that exact scenario. And as we wrap 00:23:18.779 --> 00:23:20.519 up this deep dive into our sources, I want to 00:23:20.519 --> 00:23:23.269 leave you with one final thought to ponder. We 00:23:23.269 --> 00:23:25.769 talked about how deep RL researchers use hindsight 00:23:25.769 --> 00:23:28.750 experience replay to force an AI to learn from 00:23:28.750 --> 00:23:31.470 its complete failures by simply relabeling the 00:23:31.470 --> 00:23:34.170 goal retroactively. Right. It makes you wonder, 00:23:34.829 --> 00:23:37.170 imagine if we apply that exact same algorithmic 00:23:37.170 --> 00:23:39.829 logic to our own human lives. What if the things 00:23:39.829 --> 00:23:42.529 we consider our daily failures aren't actually 00:23:42.529 --> 00:23:44.690 failures at all? What if they are just highly 00:23:44.690 --> 00:23:46.690 successful demonstrations of a goal we didn't 00:23:46.690 --> 00:23:49.089 even realize we were trying to learn? That is 00:23:49.089 --> 00:23:51.589 a fascinating, highly optimal way to reframe 00:23:51.589 --> 00:23:54.569 our own trial and error. Something to think about 00:23:54.569 --> 00:23:56.630 the next time you accidentally knock over your 00:23:56.630 --> 00:23:58.650 metaphorical blue cylinder. Thank you so much 00:23:58.650 --> 00:24:00.569 for joining us on this deep dive. Keep questioning, 00:24:00.809 --> 00:24:03.230 keep learning, and keep exploring the incredible, 00:24:03.430 --> 00:24:05.930 rapidly changing world of machine learning. We 00:24:05.930 --> 00:24:06.750 will see you next time.