WEBVTT 00:00:00.000 --> 00:00:03.379 You know that classic scene in almost every true 00:00:03.379 --> 00:00:06.759 crime documentary or police procedural? Oh, the 00:00:06.759 --> 00:00:10.259 giant cork board. Yes, exactly. The detective 00:00:10.259 --> 00:00:12.880 is standing in front of this massive cork board. 00:00:13.380 --> 00:00:15.759 And it's just completely covered in photos and 00:00:15.759 --> 00:00:18.359 maps and sticky notes. And they're all frantically 00:00:18.359 --> 00:00:22.899 connected by this chaotic web of red string. 00:00:23.160 --> 00:00:25.679 Right. They're looking for the signal in the 00:00:25.679 --> 00:00:27.870 noise. Yeah. They want to know what caused what. 00:00:28.190 --> 00:00:30.070 But I mean, here's the problem for the rest of 00:00:30.070 --> 00:00:32.609 us. Life doesn't really come with red string. 00:00:32.929 --> 00:00:35.210 No, unfortunately, it doesn't. So when you are 00:00:35.210 --> 00:00:37.890 looking at a messy, totally unpredictable world, 00:00:38.329 --> 00:00:40.649 how do you actually figure out that invisible 00:00:40.649 --> 00:00:43.609 web of cause and effect? Well, I mean, it's really 00:00:43.609 --> 00:00:45.250 the ultimate puzzle, right? We're constantly 00:00:45.250 --> 00:00:47.310 having to make decisions based on incomplete 00:00:47.310 --> 00:00:49.329 information. Yeah, all the time. We see the effects 00:00:49.329 --> 00:00:51.750 all around us, but the causes are, you know, 00:00:51.810 --> 00:00:54.310 they're often completely hidden from view. So 00:00:54.310 --> 00:00:57.000 we're basically forced to guess. Right. And today 00:00:57.000 --> 00:01:00.100 we are going to look at the mathematical blueprint 00:01:00.100 --> 00:01:02.799 for how to actually solve that puzzle. So welcome 00:01:02.799 --> 00:01:05.640 to today's deep dive. Thanks for having me. We 00:01:05.640 --> 00:01:09.299 are looking at a pretty dense, highly technical 00:01:09.299 --> 00:01:12.900 concept called Bayesian networks. Now, if you. 00:01:13.619 --> 00:01:16.140 If you don't spend your days in a computer science 00:01:16.140 --> 00:01:18.000 lab, that might sound like a pretty intimidating 00:01:18.000 --> 00:01:20.140 wall of statistical jargon. Oh, it definitely 00:01:20.140 --> 00:01:22.819 has a reputation for being very heavy on the 00:01:22.819 --> 00:01:25.620 equation. Yeah, but our mission today is to totally 00:01:25.620 --> 00:01:27.780 decode it for you. We're going to strip away 00:01:27.780 --> 00:01:29.799 the jargon and show you exactly how computers, 00:01:30.340 --> 00:01:33.739 and honestly even how human brains, model uncertainty. 00:01:33.900 --> 00:01:36.260 Exactly. We're going to explore how we figure 00:01:36.260 --> 00:01:39.620 out causes from effects and how we map out the 00:01:39.620 --> 00:01:42.040 messy realities of the world without getting 00:01:42.040 --> 00:01:44.400 just entirely overwhelmed by the math. Yeah. 00:01:44.700 --> 00:01:47.579 And to kind of set the stage here, the term Bayesian 00:01:47.579 --> 00:01:49.939 network was actually coined by a computer scientist 00:01:49.939 --> 00:01:53.480 named Judea Pearl back in 1985. 1985. Yeah. And 00:01:53.480 --> 00:01:55.950 he did this to highlight a really crucial distinction 00:01:55.950 --> 00:01:58.629 in artificial intelligence. He wanted to mathematically 00:01:58.629 --> 00:02:02.290 separate pure causal reasoning from just simple 00:02:02.290 --> 00:02:04.290 gathering of evidence. Like separating the actual 00:02:04.290 --> 00:02:06.290 mechanism from just observing things happening. 00:02:06.549 --> 00:02:08.889 Right. And he also wanted to really emphasize 00:02:08.889 --> 00:02:12.169 the subjective nature of the input information 00:02:12.169 --> 00:02:14.669 we use to build these models. It's not just about 00:02:14.669 --> 00:02:18.289 crunching numbers. It's about how we choose to 00:02:18.289 --> 00:02:20.610 structure our understanding of reality in the 00:02:20.610 --> 00:02:22.830 first place. OK, let's unpack this because that's 00:02:22.830 --> 00:02:25.330 a big idea. Let's start with the basic architecture. 00:02:25.469 --> 00:02:28.430 How do we actually build this digital corkboard? 00:02:29.030 --> 00:02:31.650 Well, at its core, a Bayesian network is what 00:02:31.650 --> 00:02:34.110 we call a probabilistic graphical model. OK. 00:02:34.430 --> 00:02:37.469 It represents a set of variables and their conditional 00:02:37.469 --> 00:02:40.509 dependencies. But to actually work, it requires 00:02:40.509 --> 00:02:42.889 a very specific structure. It has to be a directed 00:02:42.889 --> 00:02:47.669 acyclic graph, or a DAG for short. Wait, a directed 00:02:47.669 --> 00:02:51.009 acyclic graph? Okay, for those who don't spend 00:02:51.009 --> 00:02:53.289 their days mapping graph theory, this basically 00:02:53.289 --> 00:02:55.650 just means there are strict traffic laws for 00:02:55.650 --> 00:02:57.530 our red string, right? Yeah, that's actually 00:02:57.530 --> 00:02:59.229 a perfect way to look at it. So, directed just 00:02:59.229 --> 00:03:01.050 means the connections between things have to 00:03:01.050 --> 00:03:03.569 flow in a specific direction. Think of, like, 00:03:03.710 --> 00:03:06.030 one -way streets. Okay, one -way streets. And 00:03:06.030 --> 00:03:08.830 acyclic means those streets can never loop back 00:03:08.830 --> 00:03:11.050 on themselves. You absolutely cannot have an 00:03:11.050 --> 00:03:13.870 infinite loop where, you know, A causes B and 00:03:13.870 --> 00:03:16.969 B causes C and then C causes A. Oh, right, so 00:03:16.969 --> 00:03:19.400 no time travel paradox. is allowed. Exactly. 00:03:19.639 --> 00:03:21.860 The flow of causality has to keep moving forward 00:03:21.860 --> 00:03:24.340 downstream. Got it. And in this graph, we have 00:03:24.340 --> 00:03:27.020 nodes and we have edges. The nodes represent 00:03:27.020 --> 00:03:29.919 our variables. And these could be observable 00:03:29.919 --> 00:03:32.719 facts, like things you can clearly see and measure, 00:03:32.900 --> 00:03:35.039 like the weather, where they could be hitting, 00:03:35.180 --> 00:03:37.659 or latent variables, things that you strongly 00:03:37.659 --> 00:03:40.219 suspect exist, but you can't directly observe, 00:03:40.979 --> 00:03:43.800 like a suspect's underlying motive. Or they could 00:03:43.800 --> 00:03:46.500 even be entirely unknown parameters. And so the 00:03:46.500 --> 00:03:48.520 edges, those are the one red strings connecting 00:03:48.520 --> 00:03:51.180 all these nodes. Right. The edges represent direct 00:03:51.180 --> 00:03:54.319 conditional dependencies. If node A has an arrow 00:03:54.319 --> 00:03:57.340 pointing to node B, then B depends on A. But 00:03:57.340 --> 00:03:59.020 here is a really crucial detail that's going 00:03:59.020 --> 00:04:01.699 to save us later. Okay. If two nodes are not 00:04:01.699 --> 00:04:04.819 connected by any path whatsoever, they are conditionally 00:04:04.819 --> 00:04:07.159 independent of each other. Okay, so I am picturing 00:04:07.159 --> 00:04:09.300 that detective string board again, but with a 00:04:09.300 --> 00:04:12.080 twist. The red strings only go in one direction. 00:04:12.250 --> 00:04:15.909 And instead of just arbitrarily showing a connection, 00:04:16.170 --> 00:04:17.930 saying like, oh, this suspect knows this suspect, 00:04:18.569 --> 00:04:21.300 each string acts like a pipe. A pipe. Yeah, like 00:04:21.300 --> 00:04:23.620 a pipe carrying a specific volume of mathematical 00:04:23.620 --> 00:04:26.379 probability based entirely on its parent nodes, 00:04:26.560 --> 00:04:29.500 the nodes pointing at it. Yes. And what's fascinating 00:04:29.500 --> 00:04:32.360 here is how elegant that math actually is. Each 00:04:32.360 --> 00:04:35.240 node uses a probability function. It takes inputs 00:04:35.240 --> 00:04:38.279 from its parent variables, and it outputs a precise 00:04:38.279 --> 00:04:41.040 probability distribution. It's quantifying the 00:04:41.040 --> 00:04:43.860 chaos. Literally. It takes a chaotic, uncertain 00:04:43.860 --> 00:04:47.319 world and forces it into this structured mathematical 00:04:47.319 --> 00:04:50.379 hierarchy. It quantifies exactly how much one 00:04:50.379 --> 00:04:52.740 event influences another. That is incredibly 00:04:52.740 --> 00:04:55.300 cool in theory. But I mean, abstract graphs are 00:04:55.300 --> 00:04:57.439 just that. They're abstract. So to see how this 00:04:57.439 --> 00:04:59.259 actually solves a real world problem for you 00:04:59.259 --> 00:05:02.240 and me, let's look at a classic everyday scenario 00:05:02.240 --> 00:05:05.399 from the text. The wet grass example. Yes. Imagine 00:05:05.399 --> 00:05:07.720 a really simple system with just three variables. 00:05:07.779 --> 00:05:10.060 You have a sprinkler, you have rain, and you 00:05:10.060 --> 00:05:12.879 have wet grass. A very familiar situation for 00:05:12.879 --> 00:05:15.259 anyone with a lawn, really. Right. So imagine 00:05:15.259 --> 00:05:17.439 you listening to this right now. You wake up 00:05:17.439 --> 00:05:19.759 and you look out your front window. You see that 00:05:19.759 --> 00:05:22.379 the grass is wet. That is your effect. That's 00:05:22.379 --> 00:05:24.759 a known fact. Right. So you have two potential 00:05:24.759 --> 00:05:27.600 causes on your mental corkboard right now. Rain 00:05:27.600 --> 00:05:31.560 and an active sprinkler. But, and this is the 00:05:31.560 --> 00:05:34.500 kicker, rain also has a direct effect on the 00:05:34.500 --> 00:05:38.000 sprinkler. because logically your sprinkler system 00:05:38.000 --> 00:05:40.240 is on a timer that shuts off when it's raining 00:05:40.240 --> 00:05:43.040 outside. Yeah, so on our graph we have an arrow 00:05:43.040 --> 00:05:45.199 from rain to wet grass, we have an arrow from 00:05:45.199 --> 00:05:47.379 sprinkler to wet grass, and then we have a third 00:05:47.379 --> 00:05:49.420 arrow from rain to sprinkler. Okay, I see the 00:05:49.420 --> 00:05:52.579 triangle. So you observe the wet grass. The network's 00:05:52.579 --> 00:05:55.139 job now is to calculate the inverse probability, 00:05:55.519 --> 00:05:57.480 which basically means determining the likelihood 00:05:57.480 --> 00:05:59.759 of a specific cause given that observed effect. 00:05:59.800 --> 00:06:02.420 So the network asks, given that the grass is 00:06:02.420 --> 00:06:04.600 definitely wet, what are the chances it rained? 00:06:04.750 --> 00:06:07.430 How does it actually weigh those odds without 00:06:07.430 --> 00:06:09.810 just completely guessing? Well, the network uses 00:06:09.810 --> 00:06:12.170 the joint probability function of all those variables 00:06:12.170 --> 00:06:14.550 It doesn't just look at how often it rains in 00:06:14.550 --> 00:06:16.829 general It looks at the total universe of times 00:06:16.829 --> 00:06:18.990 the grass is wet whether that's from rain or 00:06:18.990 --> 00:06:21.889 the sprinkler then it isolates just the slice 00:06:21.889 --> 00:06:24.839 of that pie where rain was the actual culprit. 00:06:25.579 --> 00:06:28.000 And it mathematically factors in that rain actually 00:06:28.000 --> 00:06:30.040 suppresses the sprinkler from turning on. Right, 00:06:30.079 --> 00:06:32.860 the sensor. Right. So it computes the specific 00:06:32.860 --> 00:06:35.620 weight of that reality to give you a very precise 00:06:35.620 --> 00:06:38.740 percentage chance that rain is the cause. The 00:06:38.740 --> 00:06:41.220 text actually notes that in this specific mathematical 00:06:41.220 --> 00:06:45.860 setup, it evaluates to exactly 891 over 2491, 00:06:46.060 --> 00:06:51.819 which is about 35 .77%. OK, but wait. If we have 00:06:51.819 --> 00:06:53.779 years of weather records and sprinkler logs, 00:06:53.860 --> 00:06:56.540 why do we need this whole complex network? And 00:06:56.540 --> 00:06:58.540 the text mentions this specific mathematical 00:06:58.540 --> 00:07:01.100 concept called the dew operator. Can a computer 00:07:01.100 --> 00:07:03.180 just look at a giant spreadsheet of historical 00:07:03.180 --> 00:07:05.420 correlation to figure out what happens when grass 00:07:05.420 --> 00:07:07.480 gets wet? That is a very, very common assumption, 00:07:07.480 --> 00:07:09.920 but it is exactly why Judeo Pearl developed this 00:07:09.920 --> 00:07:11.680 whole framework. There is a massive difference 00:07:11.680 --> 00:07:14.319 between passive observation and active intervention. 00:07:14.800 --> 00:07:17.120 Passive versus active, okay. Right, and this 00:07:17.120 --> 00:07:19.500 is where the dew calculus comes in. The dew calculus, 00:07:19.860 --> 00:07:23.180 meaning literally doing an action. Yes. Suppose 00:07:23.180 --> 00:07:25.660 you don't just sit at your window and passively 00:07:25.660 --> 00:07:28.759 observe the wet grass. Suppose you intervene. 00:07:29.680 --> 00:07:32.220 You walk outside with a hose and you purposely 00:07:32.220 --> 00:07:35.060 water the lawn yourself. In the network's language, 00:07:35.259 --> 00:07:37.939 you are applying the do operator. You do the 00:07:37.939 --> 00:07:40.540 action of making the grass wet. Oh, I see. I 00:07:40.540 --> 00:07:43.819 am forcing the node to activate. Exactly. And 00:07:43.819 --> 00:07:46.180 when you do that, the network actively removes 00:07:46.180 --> 00:07:48.759 the links from the parent nodes to the wet grass 00:07:48.759 --> 00:07:51.199 node. It literally cuts the red string. Wait, 00:07:51.199 --> 00:07:54.970 why? Because you intervening and making the grass 00:07:54.970 --> 00:07:57.490 wet does not change the probability of it raining. 00:07:57.709 --> 00:07:59.910 Your hose doesn't control the weather. Oh, wow. 00:08:00.050 --> 00:08:02.170 OK. If an algorithm just looked at historical 00:08:02.170 --> 00:08:04.769 data without understanding this flow of causality, 00:08:05.170 --> 00:08:07.509 it might see a strong correlation between wet 00:08:07.509 --> 00:08:10.050 grass and rain and then falsely conclude that 00:08:10.050 --> 00:08:12.050 wetting the grass makes it more likely to rain. 00:08:12.170 --> 00:08:14.129 Which means the machine would think my garden 00:08:14.129 --> 00:08:17.829 hose is a weather control device. Ah, that's 00:08:17.829 --> 00:08:21.290 hilarious. Right. And algorithms make that exact 00:08:21.290 --> 00:08:23.470 kind of mistake all the time if they only look 00:08:23.470 --> 00:08:26.089 at raw correlations. By cutting those parent 00:08:26.089 --> 00:08:28.490 links during an intervention, Bayesian networks 00:08:28.490 --> 00:08:30.810 allow us to satisfy what's called the backdoor 00:08:30.810 --> 00:08:34.090 criterion. The backdoor criterion. Yeah, it basically 00:08:34.090 --> 00:08:37.049 prevents us from being fooled by spurious correlations, 00:08:37.049 --> 00:08:40.409 like the famous Simpson's paradox. Oh, yeah, 00:08:40.450 --> 00:08:43.470 Simpson's paradox. That is a great example of 00:08:43.470 --> 00:08:46.610 data totally lying to us. That is when a trend 00:08:46.610 --> 00:08:49.440 appears in different groups of data. but it disappears 00:08:49.440 --> 00:08:51.779 or even reverses when those groups are combined. 00:08:51.820 --> 00:08:54.480 Exactly. Like a medical study where a new drug 00:08:54.480 --> 00:08:57.179 looks highly effective for men and highly effective 00:08:57.179 --> 00:08:59.480 for women. But when you put the whole population 00:08:59.480 --> 00:09:02.080 together, the drug looks like a complete failure 00:09:02.080 --> 00:09:04.480 just because the sample sizes of the groups were 00:09:04.480 --> 00:09:06.919 unevenly distributed. Precisely. The overall 00:09:06.919 --> 00:09:09.840 correlation hides the true causal mechanism. 00:09:10.299 --> 00:09:13.139 The do -operator isolates the pure causal effect 00:09:13.139 --> 00:09:15.960 by mathematically severing those confusing background 00:09:15.960 --> 00:09:18.840 influences. OK, so if I turn on the I sever the 00:09:18.840 --> 00:09:21.179 natural causal link. Okay, I get that. Three 00:09:21.179 --> 00:09:24.059 variables, rain, sprinkler, grass. That is easy 00:09:24.059 --> 00:09:26.919 enough for my brain to process. Sure. But let's 00:09:26.919 --> 00:09:30.139 scale this up. A medical diagnosis network might 00:09:30.139 --> 00:09:33.340 have... hundreds or thousands of symptoms, test 00:09:33.340 --> 00:09:37.100 results, and diseases. If the math requires mapping 00:09:37.100 --> 00:09:39.700 the joint probability of everything, wouldn't 00:09:39.700 --> 00:09:42.860 a massive network require storing just millions 00:09:42.860 --> 00:09:44.779 or billions of possible combinations? Oh, it 00:09:44.779 --> 00:09:46.679 absolutely would, if everything were connected 00:09:46.679 --> 00:09:48.500 to everything else. This is really the scale 00:09:48.500 --> 00:09:50.559 problem in Bayesian networks. Because it just 00:09:50.559 --> 00:09:53.179 gets too big. Yeah. Normally, calculating the 00:09:53.179 --> 00:09:55.539 conditional probabilities of just 10 simple yes 00:09:55.539 --> 00:09:59.340 or no variables requires storing 1024 different 00:09:59.340 --> 00:10:02.129 values. Every new variable doubles the complexity. 00:10:02.590 --> 00:10:05.409 20 variables pushes you over a million values. 00:10:05.570 --> 00:10:08.850 An exhaustive probability table quickly becomes 00:10:08.850 --> 00:10:11.289 too massive for any computer to even hold in 00:10:11.289 --> 00:10:13.570 its memory. So if the math grows exponentially 00:10:13.570 --> 00:10:16.049 like that, brute force is obviously impossible. 00:10:16.210 --> 00:10:18.070 There has to be some kind of mathematical cheat 00:10:18.070 --> 00:10:20.629 code. What data are they cutting out to shrink 00:10:20.629 --> 00:10:23.230 the problem? Well, they beat the math by relying 00:10:23.230 --> 00:10:26.129 on sparse conditional dependencies. Because in 00:10:26.129 --> 00:10:28.490 the real world, not everything influences everything 00:10:28.490 --> 00:10:31.509 else directly. If no variable in our 10 variable 00:10:31.509 --> 00:10:34.070 network depends on more than three parent variables, 00:10:34.570 --> 00:10:38.110 you don't need to store 1 ,024 values. You only 00:10:38.110 --> 00:10:41.409 need to store at most 80 values. 80 values. From 00:10:41.409 --> 00:10:44.429 1 ,000 down to 80, that is a massive reduction. 00:10:45.110 --> 00:10:48.509 But how does the network know what it is allowed 00:10:48.509 --> 00:10:51.049 to just ignore? It relies on a really beautiful 00:10:51.049 --> 00:10:53.830 concept called the Markov blanket. OK, I love 00:10:53.830 --> 00:10:55.549 this concept. Let me try an analogy here that 00:10:55.549 --> 00:10:57.610 I was thinking about. Go for it. I like to think 00:10:57.610 --> 00:11:00.169 of your Markov blanket as your immediate gossip 00:11:00.169 --> 00:11:03.690 circle in a small town. OK. So your Markov blanket 00:11:03.690 --> 00:11:06.509 consists of your parents, your children. and 00:11:06.509 --> 00:11:08.470 the other parents of your children. Yeah, that 00:11:08.470 --> 00:11:10.649 is actually a surprisingly accurate translation 00:11:10.649 --> 00:11:12.730 of the graph theory. Right, because once you 00:11:12.730 --> 00:11:14.909 know exactly what your specific little gossip 00:11:14.909 --> 00:11:17.509 circle is doing, once you know their exact state, 00:11:17.830 --> 00:11:19.970 you are totally insulated from the rest of the 00:11:19.970 --> 00:11:21.590 network. You really don't need to listen to the 00:11:21.590 --> 00:11:23.250 rest of the town to know what your state is. 00:11:23.490 --> 00:11:26.090 Yeah, exactly. The joint distribution of just 00:11:26.090 --> 00:11:29.210 the variables in your Markov blanket is all the 00:11:29.210 --> 00:11:31.649 knowledge needed to calculate your nodes distribution. 00:11:32.129 --> 00:11:34.570 Everything outside that blanket is rendered mathematically 00:11:34.570 --> 00:11:36.929 irrelevant to you. Okay, but what if we want 00:11:36.929 --> 00:11:39.750 to zoom out from that gossip circle? We can actually 00:11:39.750 --> 00:11:42.230 look at the whole town using a broader concept 00:11:42.230 --> 00:11:45.070 called deseparation, which stands for directional 00:11:45.070 --> 00:11:47.809 separation. It's how we determine if two variables 00:11:47.809 --> 00:11:50.049 anywhere in the vast network are independent 00:11:50.049 --> 00:11:51.909 of each other, given what we currently know. 00:11:52.250 --> 00:11:54.490 Let's track the red string on that. How does 00:11:54.490 --> 00:11:57.649 deseparation block the flow of information? It 00:11:57.649 --> 00:12:01.149 looks at the trails or paths between nodes. A 00:12:01.149 --> 00:12:02.929 trail can be blocked in a few ways. So if you 00:12:02.929 --> 00:12:05.549 have a simple chain, like A causes B, which causes 00:12:05.549 --> 00:12:08.250 C. OK. If you already know the state of B, then 00:12:08.250 --> 00:12:11.330 A and C are separated. They're independent. Knowing 00:12:11.330 --> 00:12:13.389 more about A won't tell you anything new about 00:12:13.389 --> 00:12:15.870 C, because B is already locked in as the middleman. 00:12:15.909 --> 00:12:17.629 Oh, that makes perfect sense. And the same is 00:12:17.629 --> 00:12:20.070 true for a fork. That's where A causes both B 00:12:20.070 --> 00:12:22.610 and C. If you already know the shared cause, 00:12:22.789 --> 00:12:25.210 A, then learning about B tells you nothing new 00:12:25.210 --> 00:12:28.679 about C. The trail is blocked. But what if two 00:12:28.679 --> 00:12:31.759 arrows point at the same node, like our sprinkler 00:12:31.759 --> 00:12:34.639 and rain both pointing at the wet grass? They're 00:12:34.639 --> 00:12:36.919 both parents of the same child. Right. That is 00:12:36.919 --> 00:12:39.519 called an inverted fork or a collider. And it 00:12:39.519 --> 00:12:41.700 behaves exactly the opposite way. Wait, really? 00:12:41.879 --> 00:12:45.779 Yeah. If A and B both point to C, A and B are 00:12:45.779 --> 00:12:47.679 totally independent of each other until you learn 00:12:47.679 --> 00:12:51.100 the state of C. Once you know C, A and B suddenly 00:12:51.100 --> 00:12:53.059 become dependent. Okay, let me trace that out 00:12:53.059 --> 00:12:55.740 in my head. If I know the grass is wet, that's 00:12:55.740 --> 00:12:58.720 our child node, C. Yes. And I suddenly find out 00:12:58.720 --> 00:13:00.860 the sprinkler system is completely broken, that's 00:13:00.860 --> 00:13:04.360 node A. Then the probability that it rained... 00:13:04.399 --> 00:13:08.419 Node B suddenly skyrockets. Learning about one 00:13:08.419 --> 00:13:10.480 parent tells me about the other, but only because 00:13:10.480 --> 00:13:12.320 I already know the outcome of their shared child 00:13:12.320 --> 00:13:15.000 node. Exactly. Understanding these trails, the 00:13:15.000 --> 00:13:16.899 chains, forks, and colliders, and knowing how 00:13:16.899 --> 00:13:19.220 they get blocked, is how the network isolates 00:13:19.220 --> 00:13:22.440 variables. It safely ignores huge chunks of the 00:13:22.440 --> 00:13:24.779 graph, which drastically reduces the computational 00:13:24.779 --> 00:13:27.940 power needed. The structure of the network, basically 00:13:27.940 --> 00:13:30.759 who is pointing at whom, is what makes it highly 00:13:30.759 --> 00:13:34.159 efficient. But that raises a massive logistical 00:13:34.159 --> 00:13:36.519 question for me. What's that? If these networks 00:13:36.519 --> 00:13:40.399 can map hundreds or thousands of variables, who 00:13:40.399 --> 00:13:43.659 is actually dueling them? Does a human expert 00:13:43.659 --> 00:13:45.879 have to sit there and manually string up the 00:13:45.879 --> 00:13:48.980 corkboard, drawing every single arrow and typing 00:13:48.980 --> 00:13:51.899 in every single probability? Well, for very simple 00:13:51.899 --> 00:13:54.600 models, yes. A doctor might easily define the 00:13:54.600 --> 00:13:57.019 relationships between a few diseases and symptoms. 00:13:57.629 --> 00:14:01.210 But for complex applications, the task is vastly 00:14:01.210 --> 00:14:03.669 too complicated for a human mind. I would imagine. 00:14:04.110 --> 00:14:06.629 The network structure and the mathematical parameters 00:14:06.629 --> 00:14:08.950 within it must be learned automatically from 00:14:08.950 --> 00:14:10.909 raw data. And this is where machine learning 00:14:10.909 --> 00:14:13.289 takes the wheel. Precisely. And the learning 00:14:13.289 --> 00:14:15.789 is actually split into two parts. First, there 00:14:15.789 --> 00:14:18.370 is parameter learning. Imagine you know the shape 00:14:18.370 --> 00:14:20.210 of the graph. You know where the strings are, 00:14:20.330 --> 00:14:22.350 but you just don't know the exact probabilities. 00:14:23.289 --> 00:14:25.730 You use algorithms to estimate the unknown numbers 00:14:25.730 --> 00:14:28.080 from your data set. They use techniques like 00:14:28.080 --> 00:14:31.480 the expectation maximization algorithm. And mechanically, 00:14:31.840 --> 00:14:34.580 what is that algorithm actually doing? Think 00:14:34.580 --> 00:14:38.259 of it as making a highly educated baseline guess, 00:14:38.740 --> 00:14:41.820 checking its own work against the data, and automatically 00:14:41.820 --> 00:14:44.440 tweaking its own dials until the puzzle pieces 00:14:44.440 --> 00:14:47.870 fit. It iteratively guesses the values of unobserved 00:14:47.870 --> 00:14:50.929 variables, updates its probabilities, and repeats 00:14:50.929 --> 00:14:53.830 the cycle until it finds the optimal mathematical 00:14:53.830 --> 00:14:56.330 fit for the data it has. Okay, but what if you 00:14:56.330 --> 00:14:58.870 don't even know the shape of the graph? What 00:14:58.870 --> 00:15:01.450 if you just have a giant spreadsheet of chaotic 00:15:01.450 --> 00:15:04.190 data and you have zero idea what causes what? 00:15:04.649 --> 00:15:07.250 How does the machine pin the strings on the corkboard? 00:15:07.419 --> 00:15:10.120 entirely by itself. That is structure learning, 00:15:10.240 --> 00:15:13.000 and it is a massive challenge in artificial intelligence. 00:15:13.460 --> 00:15:15.700 But there was a genius insight developed by a 00:15:15.700 --> 00:15:18.179 recovery algorithm from Rabane and Pearl. Remember 00:15:18.179 --> 00:15:19.940 the collider we just talked about? Where two 00:15:19.940 --> 00:15:21.799 independent parents point to a single child. 00:15:22.379 --> 00:15:24.320 So rain and sprinkler colliding at wet grass. 00:15:24.480 --> 00:15:27.940 Yes. When an algorithm is looking at raw, unorganized 00:15:27.940 --> 00:15:30.460 data, it searches for junction patterns, a chain, 00:15:30.679 --> 00:15:33.399 so A to B to C, and a fork, where B and C are 00:15:33.399 --> 00:15:36.460 both caused by A. Those look absolutely identical 00:15:36.460 --> 00:15:39.240 purely in terms of statistical dependency. You 00:15:39.240 --> 00:15:41.840 can't easily tell which way the arrows should 00:15:41.840 --> 00:15:44.820 point just from looking at the raw numbers, but 00:15:44.820 --> 00:15:48.000 a collider is uniquely identifiable. So the collider 00:15:48.000 --> 00:15:50.080 is the one string on the board that the machine 00:15:50.080 --> 00:15:53.519 can orient all by itself. Why is that? Because 00:15:53.519 --> 00:15:56.039 the two parent nodes are marginally independent 00:15:56.039 --> 00:15:58.899 until you condition on the child node, if the 00:15:58.899 --> 00:16:01.240 algorithm spots two variables in the wild that 00:16:01.240 --> 00:16:03.879 seem totally unrelated but suddenly become highly 00:16:03.879 --> 00:16:06.059 correlated when a third variable is factored 00:16:06.059 --> 00:16:09.440 in, the algorithm knows instantly. It knows those 00:16:09.440 --> 00:16:12.379 two variables are the causes and they are colliding 00:16:12.379 --> 00:16:14.860 at that third variable. Identifying colliders 00:16:14.860 --> 00:16:17.240 allows the algorithm to start orienting the arrows 00:16:17.240 --> 00:16:19.779 of causality automatically. It builds the board 00:16:19.779 --> 00:16:21.559 from the inside out. But wait, here's where it 00:16:21.559 --> 00:16:24.039 gets really interesting, though. If the network 00:16:24.039 --> 00:16:26.840 has to check every possible combination of these 00:16:26.840 --> 00:16:29.720 arrows, constantly searching for colliders across 00:16:29.720 --> 00:16:33.559 thousands of variables, doesn't the math eventually 00:16:33.559 --> 00:16:35.940 just break? Well, in the text, it notes that 00:16:35.940 --> 00:16:38.639 in 1990, a researcher named Cooper mathematically 00:16:38.639 --> 00:16:41.559 proved that exact inference in Bayesian networks 00:16:41.559 --> 00:16:46.779 is NP -hard. And then in 1993, Degum and Luby 00:16:46.779 --> 00:16:49.340 proved that even approximating the inference 00:16:49.340 --> 00:16:53.980 is NP -hard. Oh, yeah. Those were pivotal, totally 00:16:53.980 --> 00:16:56.240 paradigm -shifting moments in the field of computer 00:16:56.240 --> 00:16:58.960 science. For you listening, NP -hard basically 00:16:58.960 --> 00:17:02.059 means the problem is so incredibly complex that 00:17:02.059 --> 00:17:04.299 the time it takes a computer to solve it grows 00:17:04.299 --> 00:17:06.960 exponentially with the size of the problem. Exactly. 00:17:07.240 --> 00:17:09.700 It implies that for large networks, computing 00:17:09.700 --> 00:17:11.920 the exact probabilities would take longer than 00:17:11.920 --> 00:17:14.480 the lifespan of the universe. So if the math 00:17:14.480 --> 00:17:17.180 is literally proven to be too hard for computers 00:17:17.180 --> 00:17:20.140 to solve efficiently, is this whole system basically 00:17:20.140 --> 00:17:22.579 broken for large -scale use? It definitely seemed 00:17:22.579 --> 00:17:24.799 that way at first, but if we connect this to 00:17:24.799 --> 00:17:26.839 the bigger picture, you have to understand that 00:17:26.839 --> 00:17:28.680 computer science doesn't just give up when it 00:17:28.680 --> 00:17:31.039 hits a theoretical wall, it adapts. Okay, so 00:17:31.039 --> 00:17:33.200 what do they do? The complexity proofs simply 00:17:33.200 --> 00:17:35.700 meant that we couldn't use brute force for massive, 00:17:36.039 --> 00:17:39.420 unconstrained networks. We needed clever workarounds. 00:17:39.779 --> 00:17:41.640 How did they work around a proven mathematical 00:17:41.640 --> 00:17:44.420 impossibility? by changing the rules of the game 00:17:44.420 --> 00:17:47.690 slightly. Degum and Luby, the same researchers 00:17:47.690 --> 00:17:51.529 who proved approximation was NP -hard, they actually 00:17:51.529 --> 00:17:54.849 developed the bounded variance algorithm. It 00:17:54.849 --> 00:17:58.349 was the first fast algorithm to efficiently approximate 00:17:58.349 --> 00:18:01.410 probabilistic inference with mathematical guarantees 00:18:01.410 --> 00:18:04.470 on the error margin. It just required a minor 00:18:04.470 --> 00:18:06.250 restriction on the conditional probabilities. 00:18:07.109 --> 00:18:09.890 We also developed local search strategies like 00:18:09.890 --> 00:18:12.789 Markov chain Monte Carlo. Markov chain Monte 00:18:12.789 --> 00:18:15.910 Carlo. How does that bypass the wall? So instead 00:18:15.910 --> 00:18:19.069 of forcing the computer to meticulously map every 00:18:19.069 --> 00:18:22.130 single inch of a massive mathematical maze, it's 00:18:22.130 --> 00:18:24.630 like sending out 1 ,000 randomized scouts to 00:18:24.630 --> 00:18:26.509 wander through the possibilities. Oh, I like 00:18:26.509 --> 00:18:29.049 that analogy. Right. They report back on where 00:18:29.049 --> 00:18:31.309 they end up, which allows the system to find 00:18:31.309 --> 00:18:34.049 highly accurate approximations without getting 00:18:34.049 --> 00:18:36.609 stuck calculating literally every dead end. That 00:18:36.609 --> 00:18:39.289 is incredibly clever. It really is. We also started 00:18:39.289 --> 00:18:41.150 restricting what's called the tree width of the 00:18:41.150 --> 00:18:43.569 graphs. The tree width? Yeah, basically by forcing 00:18:43.569 --> 00:18:45.930 the network to keep its interconnectedness slightly 00:18:45.930 --> 00:18:48.789 contain, preventing it from becoming a totally 00:18:48.789 --> 00:18:51.589 tangled ball of yarn, we can find optimal structures 00:18:51.589 --> 00:18:54.450 and run highly accurate approximations, even 00:18:54.450 --> 00:18:57.150 with thousands of variables. It is honestly marvelous. 00:18:57.329 --> 00:19:00.430 It's this incredible blend of hard mathematical 00:19:00.430 --> 00:19:04.130 limitations, graph theory, and these immensely 00:19:04.130 --> 00:19:06.769 practical workarounds. And this architecture 00:19:06.769 --> 00:19:09.150 forms the absolute backbone of modern machine 00:19:09.150 --> 00:19:11.589 learning and automated decision -making systems 00:19:11.589 --> 00:19:14.099 today. It really is the ghost in the machine 00:19:14.099 --> 00:19:16.900 of modern AI. So what does this all actually 00:19:16.900 --> 00:19:19.480 mean for you listening right now? Why should 00:19:19.480 --> 00:19:22.240 you care about DAGs and Markov blankets and do 00:19:22.240 --> 00:19:25.220 calculus? Because Bayesian networks are not just 00:19:25.220 --> 00:19:27.319 abstract theories sitting in a computer science 00:19:27.319 --> 00:19:30.200 textbook somewhere. They represent a fundamental 00:19:30.200 --> 00:19:33.670 logical way to navigate an unpredictable world. 00:19:33.930 --> 00:19:36.569 Yeah, they formalize how human intuition naturally 00:19:36.569 --> 00:19:38.769 tries to make sense of uncertainty. Exactly. 00:19:38.849 --> 00:19:41.069 They teach us a very practical lesson. You don't 00:19:41.069 --> 00:19:42.670 need to know everything to make a good decision. 00:19:42.829 --> 00:19:44.470 You don't need to hold the entire universe in 00:19:44.470 --> 00:19:47.109 your head. No, you really don't. You just need 00:19:47.109 --> 00:19:49.829 to understand what your variables are, what depends 00:19:49.829 --> 00:19:52.750 on what, and what is safely isolated in your 00:19:52.750 --> 00:19:56.170 own Markov blanket. If you can figure out what 00:19:56.170 --> 00:19:58.430 information actually influences your immediate 00:19:58.430 --> 00:20:00.890 circle, you can tune out the noise of the rest 00:20:00.890 --> 00:20:03.309 of the network. That's very true. But there is 00:20:03.309 --> 00:20:05.630 a final crucial caveat to all of this that I 00:20:05.630 --> 00:20:08.309 think is really worth mulling over. Oh. Yeah. 00:20:08.730 --> 00:20:12.019 Remember. Judea Pearl coined the term Bayesian 00:20:12.019 --> 00:20:14.440 network partially to emphasize the subjective 00:20:14.440 --> 00:20:17.240 nature of the input information. Right, the subjective 00:20:17.240 --> 00:20:20.019 nature of the input. Yes, because no matter how 00:20:20.019 --> 00:20:22.500 advanced the math becomes, no matter how elegant 00:20:22.500 --> 00:20:25.079 the algorithms, or how powerful the machine learning 00:20:25.079 --> 00:20:29.000 gets, the entire architecture still rests on 00:20:29.000 --> 00:20:31.240 the initial framing of the problem. Interesting. 00:20:31.619 --> 00:20:33.819 The machine only knows about the variables you 00:20:33.819 --> 00:20:36.940 give it. Someone, or something, has to decide 00:20:36.940 --> 00:20:38.920 what goes on the corkboard in the first place. 00:20:39.160 --> 00:20:42.019 So the nodes don't define themselves. They don't. 00:20:42.259 --> 00:20:44.579 And as we move into a world that is increasingly 00:20:44.579 --> 00:20:47.619 run by automated decision networks, it raises 00:20:47.619 --> 00:20:50.339 an important question about how we model reality. 00:20:51.160 --> 00:20:53.700 The math will flawlessly execute whatever structure 00:20:53.700 --> 00:20:56.619 it is handed. But if a critical variable is left 00:20:56.619 --> 00:20:59.400 off the board, the machine will never know it 00:20:59.400 --> 00:21:02.839 exists. Wow. The perfection of the math is entirely 00:21:02.839 --> 00:21:05.519 bound by the subjectivity of the modeler. That 00:21:05.519 --> 00:21:08.339 is a fascinating thought to leave on. What you 00:21:08.339 --> 00:21:10.420 choose to measure is ultimately the reality you 00:21:10.420 --> 00:21:13.180 get. When you look out the window to check the 00:21:13.180 --> 00:21:15.400 weather, you're building a tiny Bayesian network 00:21:15.400 --> 00:21:17.759 in your mind. Just make sure you know what variables 00:21:17.759 --> 00:21:19.839 you're actually paying attention to. We want 00:21:19.839 --> 00:21:21.480 to thank you for bringing such an incredible, 00:21:21.519 --> 00:21:23.940 challenging source to the table today. Keep pulling 00:21:23.940 --> 00:21:26.460 at those red strings. Keep exploring the hidden 00:21:26.460 --> 00:21:28.779 structures around you. And we will see you on 00:21:28.779 --> 00:21:29.779 the next Deep Dive.