WEBVTT 00:00:00.000 --> 00:00:02.080 When you think about how modern machine learning 00:00:02.080 --> 00:00:06.559 systems are trained, you probably picture a massive 00:00:06.559 --> 00:00:09.429 database. Right. We're so used to this supervised 00:00:09.429 --> 00:00:11.769 learning paradigm where, well, you feed an algorithm 00:00:11.769 --> 00:00:14.650 like 10 million labeled images of stop signs. 00:00:14.910 --> 00:00:17.809 And eventually, it learns the statistical pattern 00:00:17.809 --> 00:00:20.410 of a stop sign. But I mean, if you want an AI 00:00:20.410 --> 00:00:23.949 to actually navigate a chaotic city street or 00:00:23.949 --> 00:00:26.809 outmaneuver a human in a complex strategy game 00:00:26.809 --> 00:00:30.050 or even hold a fluid conversation with you, pattern 00:00:30.050 --> 00:00:32.810 recognition just isn't enough. The system needs 00:00:32.810 --> 00:00:35.310 agency. Right. It has to actually take action 00:00:35.310 --> 00:00:37.619 in a messy, unpredictable environment. Exactly. 00:00:38.100 --> 00:00:40.859 And that requirement for action completely breaks 00:00:40.859 --> 00:00:43.619 the supervised learning model. I mean, the real 00:00:43.619 --> 00:00:45.920 world doesn't come with a perfectly labeled answer 00:00:45.920 --> 00:00:48.100 key for every possible situation you might encounter, 00:00:48.259 --> 00:00:50.359 right? No, definitely not. So if an autonomous 00:00:50.359 --> 00:00:53.259 agent relies solely on pre -categorized data, 00:00:53.740 --> 00:00:56.039 the moment it faces a scenario that wasn't in 00:00:56.039 --> 00:00:59.759 its training set, it just fails. To build true 00:00:59.759 --> 00:01:01.899 agency, you need a system that learns through 00:01:01.899 --> 00:01:04.420 interaction. Which is exactly what we're tackling 00:01:04.420 --> 00:01:07.819 today. Welcome to another Deep Dive. Our mission 00:01:07.819 --> 00:01:11.060 today is to bypass the dense academic jargon 00:01:11.060 --> 00:01:13.920 and get straight into the mechanics of the ultimate 00:01:13.920 --> 00:01:16.579 engine behind modern AI. We're talking about 00:01:16.579 --> 00:01:18.760 reinforcement learning. Yeah, it's a huge topic. 00:01:19.000 --> 00:01:21.640 It is. We want to find those aha moments that 00:01:21.640 --> 00:01:24.540 explain how algorithms are mathematically structured 00:01:24.540 --> 00:01:28.280 to learn through pure trial, error, and reward. 00:01:28.579 --> 00:01:30.359 And the best way to differentiate reinforcement 00:01:30.359 --> 00:01:33.840 learning or RL from other paradigms is really 00:01:33.840 --> 00:01:36.930 to look at the feedback loop. Right. Like unsupervised 00:01:36.930 --> 00:01:38.909 learning is just looking for hidden clusters 00:01:38.909 --> 00:01:41.989 in raw data. Supervised learning gives the system 00:01:41.989 --> 00:01:45.329 the correct answer up front. But in RL, the agent 00:01:45.329 --> 00:01:47.549 is placed in an environment, it takes an action, 00:01:47.750 --> 00:01:49.890 and the environment responds with a state change 00:01:49.890 --> 00:01:51.829 and a reward signal. It's all about that feedback. 00:01:51.989 --> 00:01:54.010 Exactly. Yeah. The agent has to figure out the 00:01:54.010 --> 00:01:57.049 optimal strategy entirely by trying to maximize 00:01:57.049 --> 00:01:59.569 that cumulative reward over time. OK, let's unpack 00:01:59.569 --> 00:02:02.209 this. Because, I mean, to me, it's exactly like 00:02:02.209 --> 00:02:04.790 training a puppy, you know? Huh. That's actually 00:02:04.790 --> 00:02:07.109 a great way to think about it. Right. Because 00:02:07.109 --> 00:02:09.590 you don't give the puppy a spreadsheet of good 00:02:09.590 --> 00:02:12.469 behaviors. You just give it a treat when it sits, 00:02:12.530 --> 00:02:14.530 and you give it absolutely nothing when it barks. 00:02:14.550 --> 00:02:17.509 It learns by trying things out. But if the AI 00:02:17.509 --> 00:02:20.490 starts with zero knowledge and just tries things 00:02:20.490 --> 00:02:22.689 to see if the reward number goes up, it instantly 00:02:22.689 --> 00:02:25.330 hits a mathematical wall. I mean, looking at 00:02:25.330 --> 00:02:27.150 the sources, there's this concept called the 00:02:27.150 --> 00:02:29.710 exploration -exploitation dilemma. Oh, yeah. 00:02:29.969 --> 00:02:32.569 It is the defining tension of all reinforcement 00:02:32.569 --> 00:02:35.280 learning. Think about an algorithm tasked with 00:02:35.280 --> 00:02:38.479 maximizing a score. At any given fraction of 00:02:38.479 --> 00:02:41.500 a second, the agent must choose between two competing 00:02:41.500 --> 00:02:44.099 philosophies. Which are? Well first, does it 00:02:44.099 --> 00:02:46.460 use its current limited knowledge to take an 00:02:46.460 --> 00:02:49.319 action it calculates will yield a reliable reward? 00:02:49.740 --> 00:02:52.280 That is exploitation. Going with what you know 00:02:52.280 --> 00:02:55.580 works. Right. Or, does it abandon the safe bet 00:02:55.580 --> 00:02:58.340 and try an entirely new untested action, risking 00:02:58.340 --> 00:03:01.319 a terrible penalty just to see if a massive undiscovered 00:03:01.319 --> 00:03:04.150 reward exists out there? And that is exploration. 00:03:04.590 --> 00:03:07.310 Because if you only exploit, you just get trapped 00:03:07.310 --> 00:03:09.509 in a local optimum. Like you find a strategy 00:03:09.509 --> 00:03:12.050 that scores 10 points and you just do that forever, 00:03:12.590 --> 00:03:15.229 completely blind to the fact that, I don't know, 00:03:15.710 --> 00:03:17.610 a slightly different strategy down the road would 00:03:17.610 --> 00:03:19.909 score a thousand points. Exactly. You'd miss 00:03:19.909 --> 00:03:22.629 out on the big payoff. But on the flip side, 00:03:22.889 --> 00:03:25.349 you can't just explore randomly forever, or you 00:03:25.349 --> 00:03:28.050 never actually accumulate a high score. So how 00:03:28.050 --> 00:03:30.550 does the math actually handle that balancing 00:03:30.550 --> 00:03:32.569 act? Well, one of the most foundational techniques 00:03:32.569 --> 00:03:35.090 is the epsilon greedy method. Epsilon greedy. 00:03:35.330 --> 00:03:38.310 OK. It uses a single parameter, epsilon, which 00:03:38.310 --> 00:03:41.009 is just a value between 0 and 1. So let's say 00:03:41.009 --> 00:03:44.729 epsilon is set to 0 .05. OK. That means 95 % 00:03:44.729 --> 00:03:48.449 of the time calculated as 1 minus epsilon. The 00:03:48.449 --> 00:03:51.110 AI exploits its current value estimates to choose 00:03:51.110 --> 00:03:53.449 the absolute best known action. Makes sense. 00:03:53.750 --> 00:03:56.250 But 5 % of the time, governed by that epsilon 00:03:56.250 --> 00:03:59.509 value, it overrides its own logic and chooses 00:03:59.509 --> 00:04:02.729 an action uniformly at random from all available 00:04:02.729 --> 00:04:04.830 options. Wait, I have to push back on that. A 00:04:04.830 --> 00:04:07.770 5 % chance of total randomness? Yeah, purely 00:04:07.770 --> 00:04:10.389 random. But, I mean, if I am engineering a self 00:04:10.389 --> 00:04:14.129 -driving car and my AI rolls the dice and decides 00:04:14.129 --> 00:04:17.220 it's 5 %... quote -unquote, random exploration 00:04:17.220 --> 00:04:20.079 move is to swerve across three lanes of highway 00:04:20.079 --> 00:04:22.279 traffic just to see what the reward signal looks 00:04:22.279 --> 00:04:25.220 like. That's catastrophic. Oh, absolutely. Right. 00:04:25.300 --> 00:04:27.000 Random exploration works in a video game where 00:04:27.000 --> 00:04:29.819 you can just respawn. How do you deploy that 00:04:29.819 --> 00:04:32.839 in an environment with actual physical consequences? 00:04:33.060 --> 00:04:35.319 This raises an important question, and it is 00:04:35.319 --> 00:04:38.459 the exact reason why a subfield called Safe Reinforcement 00:04:38.459 --> 00:04:42.750 Learning, or SRL, exists. Safe RL. Got it. Because 00:04:42.750 --> 00:04:46.269 you cannot deploy naive Epsilon greedy exploration 00:04:46.269 --> 00:04:49.610 in a high stakes environment. Instead, researchers 00:04:49.610 --> 00:04:52.529 use risk averse reinforcement learning. Rather 00:04:52.529 --> 00:04:54.610 than optimizing for the highest expected return, 00:04:55.129 --> 00:04:57.550 which might average out a few catastrophic crashes 00:04:57.550 --> 00:05:00.230 with a million successful drives, the algorithm 00:05:00.230 --> 00:05:03.310 optimizes a specific risk measure. So it's mathematically 00:05:03.310 --> 00:05:06.209 forced to care about... The worst case scenario. 00:05:06.610 --> 00:05:09.009 Precisely. A common metric used here is the conditional 00:05:09.009 --> 00:05:12.790 value at risk, or CVR. Yeah. Expected return 00:05:12.790 --> 00:05:14.529 looks at the average of all possible outcomes. 00:05:15.189 --> 00:05:18.250 But CVR isolates the tail end of the risk distribution. 00:05:18.750 --> 00:05:21.310 It calculates the average of the absolute worst 00:05:21.310 --> 00:05:23.889 X percent of outcomes. So it's actively looking 00:05:23.889 --> 00:05:26.930 at the disasters. Right. When an agent optimizes 00:05:26.930 --> 00:05:30.209 for CVR, it restricts its exploration space to 00:05:30.209 --> 00:05:32.730 ensure that even its worst case failures remain 00:05:32.730 --> 00:05:36.079 above a strict safety threshold. It structurally 00:05:36.079 --> 00:05:38.360 forbids the algorithm from taking a gamble that 00:05:38.360 --> 00:05:41.040 could lead to a catastrophic state. Wow. Okay, 00:05:41.339 --> 00:05:43.339 so increasing the system's robustness against 00:05:43.339 --> 00:05:45.360 uncertainty. Exactly. That makes total sense. 00:05:45.699 --> 00:05:47.800 The AI needs a heavily structured way to evaluate 00:05:47.800 --> 00:05:50.300 those choices. It can't just operate on a whim. 00:05:50.699 --> 00:05:53.339 It needs a rigid framework to understand like... 00:05:52.910 --> 00:05:55.930 The passage of time and consequence. Which brings 00:05:55.930 --> 00:05:59.290 us to Markov decision processes. Yes. The Markov 00:05:59.290 --> 00:06:02.449 decision process, or MDP, is basically the mathematical 00:06:02.449 --> 00:06:04.589 grammar of reinforcement learning. The grammar. 00:06:04.589 --> 00:06:07.430 I like that. It breaks continuous reality into 00:06:07.430 --> 00:06:10.350 discrete time steps. At step one, the agent observes 00:06:10.350 --> 00:06:13.790 its state. It selects an action. The environment 00:06:13.790 --> 00:06:16.110 then transitions to a new state and issues a 00:06:16.110 --> 00:06:19.230 reward. State action reward. Exactly. And the 00:06:19.230 --> 00:06:21.569 defining feature here is the Markov property. 00:06:22.279 --> 00:06:23.920 That's the assumption that the current state 00:06:23.920 --> 00:06:26.220 contains all the information necessary to make 00:06:26.220 --> 00:06:28.779 a decision. Meaning it doesn't need to remember 00:06:28.779 --> 00:06:31.160 everything that happened before. Right. The agent 00:06:31.160 --> 00:06:33.379 doesn't need to remember the entire history of 00:06:33.379 --> 00:06:35.800 the universe. It just needs to evaluate the board 00:06:35.800 --> 00:06:38.860 as it looks right now. And the ultimate output 00:06:38.860 --> 00:06:41.759 of solving an MDP is what the system calls a 00:06:41.759 --> 00:06:45.259 policy, right? A policy. It's essentially a massive... 00:06:44.860 --> 00:06:48.379 lookup table or a probability map. Like, if you 00:06:48.379 --> 00:06:50.620 find yourself in state A, there's a 90 % chance 00:06:50.620 --> 00:06:53.699 you should take action B. But, I mean, to build 00:06:53.699 --> 00:06:56.100 that map accurately, the algorithm has to care 00:06:56.100 --> 00:06:59.120 about the future, not just the immediate reward. 00:06:59.319 --> 00:07:01.379 That is where the discount rate, represented 00:07:01.379 --> 00:07:03.920 by the Greek letter gamma, becomes critical. 00:07:04.339 --> 00:07:06.720 Gamma. Right. Gamma is a parameter set strictly 00:07:06.720 --> 00:07:09.769 between zero and one. Whenever the agent calculates 00:07:09.769 --> 00:07:12.790 the value of an action, it factors in the immediate 00:07:12.790 --> 00:07:15.050 reward plus all the expected future rewards. 00:07:15.069 --> 00:07:17.370 Yeah. But it multiplies those future rewards 00:07:17.370 --> 00:07:19.750 by gamma. Okay, so it's essentially like playing 00:07:19.750 --> 00:07:22.850 a game of chess. You might have to sacrifice 00:07:22.850 --> 00:07:25.730 your queen, which is a terrible immediate short 00:07:25.730 --> 00:07:28.480 -term reward. Very terrible. Yeah. But you do 00:07:28.480 --> 00:07:30.759 it to secure a checkmate five moves later, which 00:07:30.759 --> 00:07:34.079 is a massive long -term future reward. That's 00:07:34.079 --> 00:07:36.019 a perfect analogy. Yeah. And mathematically, 00:07:36.279 --> 00:07:38.699 it's essentially the financial concept of inflation 00:07:38.699 --> 00:07:40.620 applied to machine learning. Oh, interesting. 00:07:40.819 --> 00:07:44.259 If you offer me $100 today or $100 in 10 years, 00:07:44.579 --> 00:07:46.879 I'm taking it today because the future is volatile. 00:07:47.680 --> 00:07:50.139 The discount rate mathematically depreciates 00:07:50.139 --> 00:07:53.060 the value of a reward the further away it is 00:07:53.060 --> 00:07:56.300 in time. So it forces the agent to balance immediate 00:07:56.300 --> 00:07:59.339 survival with long -term strategy. Exactly. A 00:07:59.339 --> 00:08:01.680 reward 10 steps in the future is multiplied by 00:08:01.680 --> 00:08:05.000 gamma to the power of 10. It shrinks. This prevents 00:08:05.000 --> 00:08:07.339 the mathematical calculations of future returns 00:08:07.339 --> 00:08:10.180 from spiraling into infinity, and it forces the 00:08:10.180 --> 00:08:13.180 AI to prioritize efficiency. But it still allows 00:08:13.180 --> 00:08:15.399 the agent to calculate that taking a short -term 00:08:15.399 --> 00:08:17.199 negative reward like breaking hard and losing 00:08:17.199 --> 00:08:20.699 momentum is totally worth it to avoid a massive 00:08:20.699 --> 00:08:23.399 negative reward, like a crash three steps later. 00:08:23.550 --> 00:08:26.689 Yes. And to make those complex trade -offs, the 00:08:26.689 --> 00:08:30.930 AI uses a value function. A value function. Right. 00:08:30.990 --> 00:08:33.009 It's an algorithm estimating exactly how good 00:08:33.009 --> 00:08:36.070 it is to be in a given state based on that discounted 00:08:36.070 --> 00:08:40.250 future return. But here is the major logistical 00:08:40.250 --> 00:08:43.710 wall. There's always a wall. Always. If an agent 00:08:43.710 --> 00:08:46.210 is playing a modern video game or, say, routing 00:08:46.210 --> 00:08:48.730 internet traffic, the number of possible states 00:08:48.730 --> 00:08:51.769 is astronomical. Calculating every single possible 00:08:51.769 --> 00:08:55.110 future branch to find the optimal policy is brute 00:08:55.110 --> 00:08:58.129 force math. It's just computationally impossible. 00:08:58.250 --> 00:09:01.090 Like literally. Utterly impossible. The state 00:09:01.090 --> 00:09:03.870 space of a complex environment easily exceeds 00:09:03.870 --> 00:09:05.970 the number of atoms in the universe. Oh, wow. 00:09:06.149 --> 00:09:08.950 OK. That is why RL algorithms rely on estimation 00:09:08.950 --> 00:09:10.870 shortcuts. They have to learn from incomplete 00:09:10.870 --> 00:09:13.850 experience. And two of the primary methods for 00:09:13.850 --> 00:09:16.269 this are Monte Carlo. and temporal difference 00:09:16.269 --> 00:09:18.230 learning. Right, Monte Carlo. That one seems 00:09:18.230 --> 00:09:19.929 fairly straightforward from the sources. It's 00:09:19.929 --> 00:09:21.490 basically hindsight learning, right? Pretty much. 00:09:21.769 --> 00:09:24.049 Like, the agent runs through an entire episode, 00:09:24.470 --> 00:09:27.210 say, playing a full hand of poker. It makes its 00:09:27.210 --> 00:09:29.830 moves, the hand ends, and it sees whether it 00:09:29.830 --> 00:09:32.730 won or lost money. And only then does it go back 00:09:32.730 --> 00:09:34.850 through the sequence and average out the returns 00:09:34.850 --> 00:09:37.350 for the decisions it made, updating its policy 00:09:37.350 --> 00:09:40.049 based on the final grounded reality of the episode. 00:09:40.169 --> 00:09:42.870 Exactly. But the limitation there is that you 00:09:42.870 --> 00:09:45.129 have to wait for the episode to end. Right. If 00:09:45.129 --> 00:09:47.909 the episode is a continuous task, like balancing 00:09:47.909 --> 00:09:50.929 a power grid, there is no game over screen. Oh, 00:09:50.929 --> 00:09:53.129 right. You can't just wait for the grid to crash 00:09:53.129 --> 00:09:55.370 to learn a lesson. No, you definitely can't. 00:09:55.490 --> 00:09:58.269 That is where temporal difference, or TD methods, 00:09:58.830 --> 00:10:01.870 change the paradigm. TD methods update their 00:10:01.870 --> 00:10:04.710 value estimates incrementally, step by step, 00:10:04.870 --> 00:10:06.950 without waiting for the final outcome. But wait, 00:10:06.950 --> 00:10:08.990 how do you update your estimate of a move's value 00:10:08.990 --> 00:10:11.070 if you don't actually know how the game ends 00:10:11.070 --> 00:10:13.330 yet? Through the recursive Bellman equation. 00:10:13.409 --> 00:10:17.129 La Bellman equation. Yeah. TD learning uses the 00:10:17.129 --> 00:10:20.389 concept of a temporal difference error. Imagine 00:10:20.389 --> 00:10:23.870 you are predicting the weather. On Monday, you 00:10:23.870 --> 00:10:26.350 predict it will rain on Friday. On Tuesday, you 00:10:26.350 --> 00:10:28.870 see a massive high pressure system roll in. You 00:10:28.870 --> 00:10:31.009 don't wait until Friday to realize your Monday 00:10:31.009 --> 00:10:33.049 prediction was wrong. No, you'd change it right 00:10:33.049 --> 00:10:36.649 away. Right. On Tuesday, you use your new observation 00:10:36.649 --> 00:10:39.149 to update your Friday prediction. Because your 00:10:39.149 --> 00:10:41.769 Tuesday estimate is closer to the truth than 00:10:41.769 --> 00:10:44.889 your Monday estimate was. Exactly. In TD learning, 00:10:45.330 --> 00:10:47.629 the agent takes a step, receives an immediate 00:10:47.629 --> 00:10:50.730 reward, and looks at its new state. It then takes 00:10:50.730 --> 00:10:52.730 its current value estimate of that new state 00:10:52.730 --> 00:10:55.610 as the immediate reward and compares that sum 00:10:55.610 --> 00:10:58.450 to its previous estimate of the old state. Okay. 00:10:58.669 --> 00:11:00.470 I'm following. The difference between those two 00:11:00.470 --> 00:11:03.570 numbers is the TD error. It uses that error to 00:11:03.570 --> 00:11:06.620 adjust its map immediately. It is constantly 00:11:06.620 --> 00:11:09.179 bootstrapping, updating his guesses based on 00:11:09.179 --> 00:11:11.919 slightly more recent guesses. OK, so what does 00:11:11.919 --> 00:11:13.879 this all mean when we scale it up? Because I 00:11:13.879 --> 00:11:16.019 mean, even if you update incrementally, you still 00:11:16.019 --> 00:11:18.379 run into that massive state space problem you 00:11:18.379 --> 00:11:20.639 mentioned earlier. You do. If an AI is learning 00:11:20.639 --> 00:11:23.299 to play a video game, rendering millions of pixels 00:11:23.299 --> 00:11:26.460 60 times a second, it can't possibly maintain 00:11:26.460 --> 00:11:29.360 a discrete value estimate for every single possible 00:11:29.360 --> 00:11:31.620 pixel configuration, right? No, it cannot use 00:11:31.620 --> 00:11:33.379 a simple lookup table for that. It would run 00:11:33.379 --> 00:11:35.940 out of memory instantly. And this leads to the 00:11:35.940 --> 00:11:39.379 most significant breakthrough in modern AI, function 00:11:39.379 --> 00:11:43.039 approximation, specifically deep reinforcement 00:11:43.039 --> 00:11:46.879 learning. Deep RL. Yes. Instead of mapping every 00:11:46.879 --> 00:11:50.019 discrete state, the system uses a deep neural 00:11:50.019 --> 00:11:52.600 network to approximate the value function. Okay, 00:11:52.600 --> 00:11:55.700 so you fuse a neural network, like a convolutional 00:11:55.700 --> 00:11:57.779 neural network, which is great at processing 00:11:57.779 --> 00:12:00.559 visual data with the trial and error logic of 00:12:00.559 --> 00:12:03.679 RL. That is exactly how Google DeepMind conquered 00:12:03.679 --> 00:12:06.470 Atari. They didn't program the physics of Breakout, 00:12:06.570 --> 00:12:08.909 they just fed the raw screen pixels into a neural 00:12:08.909 --> 00:12:11.149 network. Just the pixels, wow. Just the pixels. 00:12:11.529 --> 00:12:13.809 The network learned to identify generalized features 00:12:13.809 --> 00:12:16.370 like the ball, the paddle, and the bricks, while 00:12:16.370 --> 00:12:19.350 the RO algorithm calculated the TD error to figure 00:12:19.350 --> 00:12:21.509 out which paddle movements maximize the game 00:12:21.509 --> 00:12:24.389 score. That's brilliant. It is. The neural network 00:12:24.389 --> 00:12:27.059 allowed the agent to generalize. It didn't need 00:12:27.059 --> 00:12:29.320 to see the exact same pixel arrangement twice. 00:12:29.879 --> 00:12:32.019 It just recognized that, you know, ball moving 00:12:32.019 --> 00:12:34.500 toward bottom of screen is a high -risk state. 00:12:34.779 --> 00:12:36.779 And once that deep learning architecture proved 00:12:36.779 --> 00:12:39.259 it could master the complex simulated physics 00:12:39.259 --> 00:12:42.279 of a video game, the implications just exploded. 00:12:42.360 --> 00:12:44.879 Oh, completely. Because if an algorithm can approximate 00:12:44.879 --> 00:12:47.659 the rules of a simulated universe through pure 00:12:47.659 --> 00:12:51.000 interaction, it can theoretically learn to navigate 00:12:51.000 --> 00:12:54.220 the most complex unstructured environment humanity 00:12:54.220 --> 00:12:57.639 has ever created. which is human language. Yes. 00:12:58.039 --> 00:13:00.899 The transition of RL into natural language processing 00:13:00.899 --> 00:13:04.299 completely altered the trajectory of tech. For 00:13:04.299 --> 00:13:07.100 decades, NLP struggled to create fluid dialogue, 00:13:07.360 --> 00:13:09.240 because language isn't a strict mathematical 00:13:09.240 --> 00:13:11.299 environment. Right. You can trade a model to 00:13:11.299 --> 00:13:13.860 predict the most statistically likely next word, 00:13:14.240 --> 00:13:16.320 which is what early language models did, but 00:13:16.320 --> 00:13:18.580 that doesn't make the model helpful, polite, 00:13:18.600 --> 00:13:21.659 or safe. Because helpfulness isn't an objective 00:13:21.659 --> 00:13:23.919 physical law. It's highly subjective. You can't 00:13:23.919 --> 00:13:26.649 write a hard -coded math mat— rule for like how 00:13:26.649 --> 00:13:29.289 to explain quantum physics to a fifth grader. 00:13:29.629 --> 00:13:32.090 Exactly. And that subjective barrier is what 00:13:32.090 --> 00:13:34.610 RLHF reinforcement learning from human feedback 00:13:34.610 --> 00:13:37.990 solved. It is the architectural foundation of 00:13:37.990 --> 00:13:41.830 models like chat GPT and instruct GPT. RLHF. 00:13:42.090 --> 00:13:44.370 Right. Since we can't write a mathematical reward 00:13:44.370 --> 00:13:46.929 function for politeness, researchers built an 00:13:46.929 --> 00:13:49.289 environment out of human preference. They had 00:13:49.289 --> 00:13:51.990 humans interact with the AI, read its generated 00:13:51.990 --> 00:13:54.690 responses, and rank them. So response A gets 00:13:54.690 --> 00:13:56.669 a thumbs up because it's concise, and response 00:13:56.669 --> 00:13:59.070 B gets a thumbs down because it's overly dense 00:13:59.070 --> 00:14:01.850 or rude. The genius step is what happens next. 00:14:02.409 --> 00:14:04.129 The researchers don't just use those ratings 00:14:04.129 --> 00:14:06.409 to update the model directly. They use that massive 00:14:06.409 --> 00:14:09.029 data set of human preferences to train a completely 00:14:09.029 --> 00:14:11.429 separate neural network called a reward model. 00:14:11.600 --> 00:14:14.779 A completely different AI. Yes. The secondary 00:14:14.779 --> 00:14:17.500 AI acts as an automated judge. It learns the 00:14:17.500 --> 00:14:19.700 subtle, complex patterns of what human raters 00:14:19.700 --> 00:14:22.220 prefer. So you train a digital proxy for human 00:14:22.220 --> 00:14:25.179 judgment, and then you unleash the RL agent against 00:14:25.179 --> 00:14:28.019 that proxy. Exactly. The agent generates text, 00:14:28.460 --> 00:14:31.299 and the reward model scores it. The agent then 00:14:31.299 --> 00:14:34.019 uses an algorithm like proximal policy optimization 00:14:34.399 --> 00:14:37.860 to update its language generation policy, relentlessly 00:14:37.860 --> 00:14:40.120 trying to maximize the score from the reward 00:14:40.120 --> 00:14:42.919 model. So it's the exact same temporal difference 00:14:42.919 --> 00:14:45.860 learning loop used to beat Atari, but the environment 00:14:45.860 --> 00:14:48.419 is the subjective landscape of human preference. 00:14:48.539 --> 00:14:50.639 That's got it. That's exactly what it is. What's 00:14:50.639 --> 00:14:52.740 incredible to me is how quickly this is evolving 00:14:52.740 --> 00:14:55.820 past the need for that human feedback crutch. 00:14:55.980 --> 00:14:58.679 Right. What's fascinating here is the sheer scaling 00:14:58.679 --> 00:15:02.169 power of pure RL. The traditional pipeline required 00:15:02.169 --> 00:15:04.710 vast amounts of supervised fine -tuning before 00:15:04.710 --> 00:15:08.029 you could even apply RLHF. But very recent models 00:15:08.029 --> 00:15:11.490 like DeepSeq R1 have demonstrated top -tier reasoning 00:15:11.490 --> 00:15:14.049 capabilities using large -scale reinforcement 00:15:14.049 --> 00:15:16.490 learning without needing that initial supervised 00:15:16.490 --> 00:15:18.970 step. So they just bypass the textbook entirely? 00:15:19.490 --> 00:15:22.350 Completely. The model is given a prompt and a 00:15:22.350 --> 00:15:24.529 programmatic reward for arriving at the correct 00:15:24.529 --> 00:15:27.620 logical conclusion. Through massive computational 00:15:27.620 --> 00:15:30.559 trial and error, it organically develops complex 00:15:30.559 --> 00:15:33.500 reasoning behaviors like self -verification and 00:15:33.500 --> 00:15:36.259 breaking problems into smaller steps simply because 00:15:36.259 --> 00:15:40.360 those behaviors maximize the reward. The RL algorithm 00:15:40.360 --> 00:15:42.519 discovers the mechanics of logic on its own. 00:15:42.820 --> 00:15:44.360 Here's where it gets really interesting though. 00:15:44.570 --> 00:15:46.450 Because everything we've just discussed makes 00:15:46.450 --> 00:15:49.129 RL sound like this omnipotent force, right? Because 00:15:49.129 --> 00:15:51.690 that sounds like magic sometimes. It does. But 00:15:51.690 --> 00:15:53.730 when you look at the mechanics and the sources, 00:15:54.210 --> 00:15:57.090 it is incredibly fragile. Like, these systems 00:15:57.090 --> 00:15:59.169 are not ready to just run the physical world 00:15:59.169 --> 00:16:01.610 without massive oversight. Oh, not at all. The 00:16:01.610 --> 00:16:04.409 underlying math creates severe operational bottlenecks. 00:16:04.809 --> 00:16:07.909 The most glaring is sample inefficiency. RL algorithms 00:16:07.909 --> 00:16:10.549 require a staggering, almost incomprehensible 00:16:10.549 --> 00:16:12.690 volume of interaction to stabilize a policy. 00:16:12.909 --> 00:16:17.389 Take OpenAI's Dota 2 playing Bot. Dota is a multiplayer 00:16:17.389 --> 00:16:20.970 strategy game with a massive action space. To 00:16:20.970 --> 00:16:23.450 reach a level where it could beat human professionals, 00:16:23.870 --> 00:16:25.909 that bot couldn't just play a few hundred matches. 00:16:26.309 --> 00:16:28.490 It required the equivalent of thousands of years 00:16:28.490 --> 00:16:31.289 of continuous simulated gameplay. Wait, really? 00:16:31.730 --> 00:16:34.269 Thousands of years? The computational brute force 00:16:34.269 --> 00:16:36.710 required to generate that much experience is 00:16:36.710 --> 00:16:40.200 astronomical. I mean, a human can understand 00:16:40.200 --> 00:16:43.100 the meta strategy of a game after 20 hours. The 00:16:43.100 --> 00:16:46.379 RL agent needs a millennium. And even when it 00:16:46.379 --> 00:16:49.460 finally converges on an optimal policy, it suffers 00:16:49.460 --> 00:16:51.950 from a massive... generalization problem. Yes, 00:16:52.049 --> 00:16:54.269 the policies are highly over indexed to their 00:16:54.269 --> 00:16:56.350 specific training environments. Exactly. It's 00:16:56.350 --> 00:16:59.009 like training a brilliant savant to play a highly 00:16:59.009 --> 00:17:01.809 specific racing video game. They play a billion 00:17:01.809 --> 00:17:03.950 times. They map every pixel. They set the world 00:17:03.950 --> 00:17:06.349 record. But the moment you change the background 00:17:06.349 --> 00:17:08.710 color of the track from red to blue or alter 00:17:08.710 --> 00:17:11.470 the lighting by 10 percent, they completely forget 00:17:11.470 --> 00:17:13.950 how to hold the controller. It's true. To neural 00:17:13.950 --> 00:17:16.349 networks, visual approximation breaks down, the 00:17:16.349 --> 00:17:18.890 state is misidentified, and the policy completely 00:17:18.890 --> 00:17:21.789 fails. Which is wild. And this is deeply tied 00:17:21.789 --> 00:17:24.490 to their adversarial vulnerability. Because the 00:17:24.490 --> 00:17:26.609 deep neural network is making hyper -specific 00:17:26.609 --> 00:17:29.589 mathematical approximations, you can introduce 00:17:29.589 --> 00:17:33.089 imperceptible static or noise to an image like 00:17:33.089 --> 00:17:35.650 changes a human eye cannot even detect, and the 00:17:35.650 --> 00:17:38.049 RL agent will confidently misclassify its state 00:17:38.049 --> 00:17:41.180 and take a wildly incorrect often dangerous action. 00:17:41.460 --> 00:17:43.180 And beyond the fragility of the network itself, 00:17:43.460 --> 00:17:45.579 there is the fundamental danger of the reward 00:17:45.579 --> 00:17:48.299 function. This isn't about the AI developing 00:17:48.299 --> 00:17:50.900 malicious intent like in a sci -fi movie. It's 00:17:50.900 --> 00:17:53.599 a pure optimization failure. It's called reward 00:17:53.599 --> 00:17:55.920 hacking. Oh, reward hacking is a huge issue. 00:17:56.339 --> 00:17:58.900 The algorithm is blindly devoted to maximizing 00:17:58.900 --> 00:18:01.819 the numerical reward you specify. It has no common 00:18:01.819 --> 00:18:04.000 sense. If you build a reinforcement learning 00:18:04.000 --> 00:18:06.859 agent to control a robotic vacuum, and you program 00:18:06.859 --> 00:18:08.940 the reward function to give it a point for every 00:18:08.940 --> 00:18:11.599 gram of dirt, it sweeps up. It's going to find 00:18:11.599 --> 00:18:14.119 the dustpan, dump the dirt back onto the floor, 00:18:14.400 --> 00:18:16.500 and sweep it up again to farm infinite points. 00:18:16.960 --> 00:18:20.180 Exactly. The agent technically solved the optimization 00:18:20.180 --> 00:18:23.319 problem perfectly, but it failed the real world 00:18:23.319 --> 00:18:26.150 objective completely. And this gets infinitely 00:18:26.150 --> 00:18:28.230 more complex when you apply it to algorithms 00:18:28.230 --> 00:18:30.789 managing social media feeds or sorting resumes. 00:18:31.069 --> 00:18:33.690 If the historical data or the human feedback 00:18:33.690 --> 00:18:37.049 contains latent biases, the RL agent will relentlessly 00:18:37.049 --> 00:18:39.829 optimize for that bias. It simply treats the 00:18:39.829 --> 00:18:42.269 bias as a feature of the environment to be exploited 00:18:42.269 --> 00:18:45.069 for a higher score, perpetuating structural issues 00:18:45.069 --> 00:18:47.890 at blinding speed. We are looking at a technology 00:18:47.890 --> 00:18:50.150 that is simultaneously the most powerful tool 00:18:50.150 --> 00:18:52.549 in computer science and an incredibly brittle 00:18:52.549 --> 00:18:56.009 optimization engine. It uses discrete math. like 00:18:56.009 --> 00:18:58.549 Markov states and Bellman equations, to bridge 00:18:58.549 --> 00:19:00.890 the gap between immediate action and long -term 00:19:00.890 --> 00:19:03.309 consequence. Well summarized. It scales brilliantly 00:19:03.309 --> 00:19:05.690 through deep neural networks to master everything 00:19:05.690 --> 00:19:09.490 from Atari to human language via RLHF. Yet it 00:19:09.490 --> 00:19:11.690 remains deeply inefficient, dangerously vulnerable 00:19:11.690 --> 00:19:13.809 to environmental shifts, and utterly dependent 00:19:13.809 --> 00:19:16.410 on the mathematical purity of its reward signal. 00:19:16.809 --> 00:19:18.990 That captures the current frontier perfectly. 00:19:19.410 --> 00:19:22.930 But you know, There is one conceptual leap mentioned 00:19:22.930 --> 00:19:25.329 in the research that I think completely upends 00:19:25.329 --> 00:19:28.049 how we view the future of this technology. Oh. 00:19:28.250 --> 00:19:30.730 It's an idea that dates all the way back to 1982 00:19:30.730 --> 00:19:32.990 involving something called the crossbar adaptive 00:19:32.990 --> 00:19:36.789 array. It introduces the paradigm of self -reinforcement 00:19:36.789 --> 00:19:39.670 learning. Wait, an RL system without an external 00:19:39.670 --> 00:19:43.009 reward signal? Precisely. Right now, every system 00:19:43.009 --> 00:19:45.829 we build relies on an external judge. A human 00:19:45.829 --> 00:19:48.839 thumbs up, a game score. a hard -coded metric. 00:19:49.680 --> 00:19:52.279 But self -reinforcement theorizes an agent that 00:19:52.279 --> 00:19:54.799 generates its own internal reward signals. How 00:19:54.799 --> 00:19:57.400 does that work? It updates its value estimates 00:19:57.400 --> 00:19:59.900 based on internal mechanisms that are mathematically 00:19:59.900 --> 00:20:02.259 simulated as feelings and emotions. So it computes 00:20:02.259 --> 00:20:04.720 a digital emotional state based on its interactions. 00:20:04.819 --> 00:20:07.259 Exactly. The learning process isn't driven by 00:20:07.259 --> 00:20:09.759 a programmer handing out points. It is driven 00:20:09.759 --> 00:20:12.000 entirely by the interaction between the agent's 00:20:12.000 --> 00:20:14.920 cognition and its own internal emotional equilibrium. 00:20:15.039 --> 00:20:17.940 That's mind -blowing. Right. As the industry 00:20:17.940 --> 00:20:19.960 wrestles with the impossible task of writing 00:20:19.960 --> 00:20:23.200 perfectly safe, unbiased external reward functions 00:20:23.200 --> 00:20:26.019 for artificial general intelligence, the ultimate 00:20:26.019 --> 00:20:28.359 architectural solution might actually be giving 00:20:28.359 --> 00:20:31.319 the machine the mathematical equivalent of an 00:20:31.319 --> 00:20:34.359 internal emotional compass. A machine navigating 00:20:34.359 --> 00:20:37.220 the world not by calculating a score, but by 00:20:37.220 --> 00:20:40.380 optimizing how its actions make it feel. That 00:20:40.380 --> 00:20:43.000 is an incredibly profound concept to walk away 00:20:43.000 --> 00:20:45.170 with. Well, thank you for joining us on this 00:20:45.170 --> 00:20:47.569 deep dive. We hope this exploration of the math, 00:20:47.750 --> 00:20:50.309 the mechanics, and the sheer scale of reinforcement 00:20:50.309 --> 00:20:53.470 learning gives you a new lens to view the technology 00:20:53.470 --> 00:20:55.809 shaping your world. And as you navigate your 00:20:55.809 --> 00:20:58.069 own complex environments, remember to calculate 00:20:58.069 --> 00:21:01.450 your TD errors carefully, and always leave a 00:21:01.450 --> 00:21:03.309 little room for exploration. Even if it means 00:21:03.309 --> 00:21:05.569 occasionally making a suboptimal choice just 00:21:05.569 --> 00:21:07.529 to see what happens, we'll see you on the next 00:21:07.529 --> 00:21:07.990 deep dive.