WEBVTT 00:00:00.000 --> 00:00:03.319 Imagine you're sitting across a polished mahogany 00:00:03.319 --> 00:00:06.040 desk from a loan officer. Oh, stressful already. 00:00:06.379 --> 00:00:09.300 Right. You've applied for a mortgage to buy your 00:00:09.300 --> 00:00:12.660 dream home, but the officer slides a piece of 00:00:12.660 --> 00:00:14.839 paper across the desk and tells you your application 00:00:14.839 --> 00:00:17.280 has been denied. Yeah. And obviously you ask 00:00:17.280 --> 00:00:20.140 why. Exactly. But they just shrug and say, well. 00:00:20.280 --> 00:00:23.500 The algorithm said no. Wow. Yeah. No reasons, 00:00:23.679 --> 00:00:27.280 no feedback, just a completely opaque, impenetrable 00:00:27.280 --> 00:00:30.519 black box making a life -altering choice on your 00:00:30.519 --> 00:00:33.439 behalf. Which is terrifying. It really is. And 00:00:33.439 --> 00:00:35.780 in a world increasingly dominated by incredibly 00:00:35.780 --> 00:00:39.060 complex artificial intelligence, that scenario 00:00:39.060 --> 00:00:41.520 is becoming a terrifying reality for a lot of 00:00:41.520 --> 00:00:44.159 people. Because we feed these massive neural 00:00:44.159 --> 00:00:47.179 networks millions of data points. And we often 00:00:47.179 --> 00:00:49.509 just have to, you know, blindly trust whatever 00:00:49.509 --> 00:00:51.530 prediction pops out the other side. Even when 00:00:51.530 --> 00:00:53.429 the programmers themselves can't fully trace 00:00:53.429 --> 00:00:56.210 the logic, right? Exactly. I mean, they literally 00:00:56.210 --> 00:00:58.369 can't tell you how the machine arrived at that 00:00:58.369 --> 00:01:00.250 specific note. But what if I told you that one 00:01:00.250 --> 00:01:03.270 of the most foundational, powerful ways a computer 00:01:03.270 --> 00:01:06.170 learns to make predictions doesn't rely on a 00:01:06.170 --> 00:01:09.010 mysterious black box at all? Oh, I like where 00:01:09.010 --> 00:01:11.409 this is going. What if it operates almost exactly 00:01:11.409 --> 00:01:14.269 like the game of 20 questions? That is a very 00:01:14.269 --> 00:01:16.569 different vibe from a neural network. Definitely. 00:01:16.989 --> 00:01:19.670 Welcome to today's deep dive. Whether you are 00:01:19.670 --> 00:01:22.069 prepping for a big meeting or you're simply curious 00:01:22.069 --> 00:01:24.650 about how machine learning actually categorizes 00:01:24.650 --> 00:01:27.670 the world, our mission today is to explore the 00:01:27.670 --> 00:01:30.790 ultimate transparent algorithm. The anti -black 00:01:30.790 --> 00:01:33.930 box. Essentially. Precisely. We are pulling our 00:01:33.930 --> 00:01:36.390 insights today from a comprehensive Wikipedia 00:01:36.390 --> 00:01:39.689 article on decision tree learning, which is this 00:01:39.689 --> 00:01:42.549 absolute cornerstone concept in data mining. 00:01:42.890 --> 00:01:45.379 Okay, let's unpack this. Let's do it. Before 00:01:45.379 --> 00:01:47.599 we look at the complex math running under the 00:01:47.599 --> 00:01:49.900 hood, I feel like we need to establish a visual 00:01:49.900 --> 00:01:53.040 baseline. Good idea. When we say decision tree, 00:01:53.219 --> 00:01:56.500 I immediately picture a flow chart. Like, you 00:01:56.500 --> 00:01:58.340 know one of those quizzes in the back of a magazine 00:01:58.340 --> 00:02:00.099 that tells you what kind of personality you have? 00:02:00.180 --> 00:02:01.920 Oh, definitely. Like, what kind of pizza topping 00:02:01.920 --> 00:02:04.480 are you? Yes, exactly. You start at the very 00:02:04.480 --> 00:02:07.200 top, you answer a single question, and you follow 00:02:07.200 --> 00:02:09.460 the arrow down to the next question until you 00:02:09.460 --> 00:02:11.919 eventually reach the final result. And that visual 00:02:11.919 --> 00:02:14.599 maps perfectly. to what data scientists call 00:02:14.599 --> 00:02:18.080 a supervised learning approach. Supervised learning. 00:02:18.319 --> 00:02:20.860 Right. The algorithm builds a predictive model 00:02:20.860 --> 00:02:23.919 to draw concrete conclusions about a set of observations. 00:02:24.599 --> 00:02:27.199 And under this big umbrella term called CART. 00:02:27.580 --> 00:02:31.219 Wait, CART? Like a shopping cart. C -A -R -T. 00:02:31.539 --> 00:02:34.139 It stands for Classification and Regression Trees. 00:02:34.930 --> 00:02:37.909 Under CART, we see two main ways this flow chart 00:02:37.909 --> 00:02:40.189 structure is actually applied. Okay, break those 00:02:40.189 --> 00:02:42.449 down for me. So, classification trees handle 00:02:42.449 --> 00:02:45.090 discrete categories. In your flow chart visual, 00:02:45.509 --> 00:02:47.969 the leaves at the very end of the branches represent 00:02:47.969 --> 00:02:50.569 class labels. Like the pizza topping. Exactly. 00:02:50.849 --> 00:02:53.169 And the branches themselves are the conjunctions 00:02:53.169 --> 00:02:55.870 of features that guide you there. Yes or no paths. 00:02:55.949 --> 00:02:58.530 Got it. And the regression part. Regression trees 00:02:58.530 --> 00:03:00.990 handle continuous values, like if you're predicting 00:03:00.990 --> 00:03:02.889 a real number. So if you're trying to estimate 00:03:02.889 --> 00:03:05.469 the price of a house or calculate how many days 00:03:05.469 --> 00:03:07.110 a patient will need to stay in a hospital bed, 00:03:07.509 --> 00:03:09.430 you use a regression tree. OK, that makes sense. 00:03:09.569 --> 00:03:11.610 The source material actually offers a fascinating 00:03:11.610 --> 00:03:14.430 historical example of a classification tree using 00:03:14.430 --> 00:03:17.580 passengers on the Titanic. Oh, the Titanic data 00:03:17.580 --> 00:03:20.259 set is a classic in machine learning. Is it? 00:03:20.500 --> 00:03:23.319 Well, the algorithm analyzes the passenger data 00:03:23.319 --> 00:03:26.680 and constructs the most efficient way to categorize 00:03:26.680 --> 00:03:29.819 who survived and who didn't. And the resulting 00:03:29.819 --> 00:03:32.680 tree is completely transparent. You can see exactly 00:03:32.680 --> 00:03:35.080 what it prioritized. Right. It essentially reveals 00:03:35.080 --> 00:03:37.639 that your chances of survival were highest if 00:03:37.639 --> 00:03:40.840 you were female. Or if you were a male, you needed 00:03:40.840 --> 00:03:43.240 to be under nine and a half years old with fewer 00:03:43.240 --> 00:03:46.159 than three siblings aboard. which perfectly aligns 00:03:46.159 --> 00:03:48.360 with the whole women and children first protocol. 00:03:49.000 --> 00:03:51.819 Exactly. And modern medicine applies that exact 00:03:51.819 --> 00:03:54.939 same structural logic to predict patient outcomes 00:03:54.939 --> 00:03:58.180 today. Oh, absolutely. The source actually highlights 00:03:58.180 --> 00:04:01.180 a specific decision tree used to estimate the 00:04:01.180 --> 00:04:03.340 probability of a spinal deformity developing 00:04:03.340 --> 00:04:05.860 after surgery. Kyphosis, right? Yes, kyphosis. 00:04:06.180 --> 00:04:07.919 The beauty of it is that the algorithm doesn't 00:04:07.919 --> 00:04:10.599 need a thousand complex variables to figure this 00:04:10.599 --> 00:04:13.240 out. Like a neural net would. Right. It focuses 00:04:13.240 --> 00:04:15.639 primarily on just the age of the patient and 00:04:15.639 --> 00:04:18.160 the specific vertebro that was operated on. Wow. 00:04:18.279 --> 00:04:21.399 Just those two main things. Mainly, yeah. Following 00:04:21.399 --> 00:04:24.240 those splits down the branches lands the doctor 00:04:24.240 --> 00:04:27.339 on a leaf that provides a very clear statistical 00:04:27.339 --> 00:04:29.639 probability of the condition occurring. It really 00:04:29.639 --> 00:04:32.519 is the ultimate game of 20 questions. The computer 00:04:32.519 --> 00:04:35.019 just asks a series of yes or no questions to 00:04:35.019 --> 00:04:37.060 progressively narrow down the possibilities. 00:04:37.540 --> 00:04:41.259 Except this player is completely blind at the 00:04:41.259 --> 00:04:43.839 start. True. So it relies on a process called 00:04:43.839 --> 00:04:46.860 top -down induction of decision trees. TDIDT 00:04:46.860 --> 00:04:49.720 for short. Exactly. And the core engine driving 00:04:49.720 --> 00:04:53.180 TDIDT is a technique called recursive partitioning. 00:04:53.439 --> 00:04:56.139 Recursive partitioning, okay. Rather than just 00:04:56.139 --> 00:04:58.180 dropping a textbook definition on everyone, let's 00:04:58.180 --> 00:05:00.199 visualize this. I love a good visualization. 00:05:00.560 --> 00:05:03.060 Imagine the algorithm puts the entire data set, 00:05:03.060 --> 00:05:05.500 let's say a thousand medical patients, into a 00:05:05.500 --> 00:05:08.040 single massive room. Okay, everyone is mingled 00:05:08.040 --> 00:05:10.319 together. Right. That is the root node at the 00:05:10.319 --> 00:05:13.379 very top of the tree. To partition them, it draws 00:05:13.379 --> 00:05:15.379 a line down the middle of the room based on a 00:05:15.379 --> 00:05:19.620 specific role. Like, are you over 50 years old? 00:05:19.759 --> 00:05:21.480 And then it sends people to opposite corners 00:05:21.480 --> 00:05:24.519 based on their answer. And once those two smaller 00:05:24.519 --> 00:05:27.899 subsets are formed, it repeats the process recursively. 00:05:28.420 --> 00:05:31.060 It draws another line in each corner, splitting 00:05:31.060 --> 00:05:33.459 the groups again and again. Until when? When 00:05:33.459 --> 00:05:35.810 does it stop? Well, the algorithm only halts 00:05:35.810 --> 00:05:38.370 this splitting when a subset has all the exact 00:05:38.370 --> 00:05:41.029 same values, meaning everyone in that corner 00:05:41.029 --> 00:05:44.290 has the exact same diagnosis. Oh, I see. Or it 00:05:44.290 --> 00:05:46.490 stops when drawing another line, just doesn't 00:05:46.490 --> 00:05:48.769 add any measurable value to the prediction. Which 00:05:48.769 --> 00:05:51.290 actually exposes a massive blind spot if you 00:05:51.290 --> 00:05:53.790 think about it. How so? Well, when you and I 00:05:53.790 --> 00:05:56.750 play 20 questions, human intuition guides our 00:05:56.750 --> 00:05:59.740 opening moves. We start incredibly broad with 00:05:59.740 --> 00:06:01.660 something like, is it an animal? Right, you want 00:06:01.660 --> 00:06:03.959 to cut the field in half. Yeah. We would never 00:06:03.959 --> 00:06:06.839 start the game by asking, does it have a slightly 00:06:06.839 --> 00:06:09.759 crooked left toe? That would be a terrible first 00:06:09.759 --> 00:06:12.180 question. A machine doesn't have that common 00:06:12.180 --> 00:06:14.199 sense, though. So how does it mathematically 00:06:14.199 --> 00:06:17.040 determine which variable creates the actual best 00:06:17.040 --> 00:06:19.959 possible split at the very top of the tree? Ah, 00:06:20.220 --> 00:06:22.439 well, algorithms use several different mathematical 00:06:22.439 --> 00:06:25.180 metrics to define what best actually means. OK, 00:06:25.300 --> 00:06:27.100 like what? One of the more straightforward methods 00:06:27.100 --> 00:06:30.430 is the estimate of positive correctness. Or EP. 00:06:30.750 --> 00:06:33.810 EP. The source breaks down the math for ET as 00:06:33.810 --> 00:06:36.129 taking the true positives and subtracting the 00:06:36.129 --> 00:06:38.550 false positives. Right. But let's ditch the raw 00:06:38.550 --> 00:06:40.930 arithmetic for a second and use an analogy. Go 00:06:40.930 --> 00:06:44.069 for it. Imagine you're managing a nightclub and 00:06:44.069 --> 00:06:47.089 you're trying to hire a bouncer. OK. Feature 00:06:47.089 --> 00:06:50.410 A is a bouncer who catches eight underage kids 00:06:50.410 --> 00:06:53.189 trying to sneak in. But he accidentally kicks 00:06:53.189 --> 00:06:55.529 out two adults who are actually of legal age. 00:06:55.709 --> 00:06:59.800 Oops. Right. So his EQ score is six. Eight true 00:06:59.800 --> 00:07:02.540 positives minus two false positives. Makes sense. 00:07:03.060 --> 00:07:05.060 Now, feature B is a different bouncer. He only 00:07:05.060 --> 00:07:08.120 catches six underage kids, but he also accidentally 00:07:08.120 --> 00:07:10.980 kicks out two adults. His EP is four. Because 00:07:10.980 --> 00:07:14.139 six minus two is four. Exactly. So just looking 00:07:14.139 --> 00:07:16.459 at the EP score, bouncer A seems like the clear 00:07:16.459 --> 00:07:18.980 winner, right? Six is higher than four. Sure, 00:07:18.980 --> 00:07:22.220 but the EP metric just provides a fast, raw volume 00:07:22.220 --> 00:07:24.980 estimate. It completely lacks critical context. 00:07:25.120 --> 00:07:27.319 What kind of context? Well, what if bouncer A... 00:07:27.389 --> 00:07:30.370 missed 100 underage kids that slipped past him, 00:07:30.730 --> 00:07:33.110 while bouncer B was guarding a smaller door and 00:07:33.110 --> 00:07:35.550 caught every single underage person who tried 00:07:35.550 --> 00:07:37.970 to enter. Oh, wow. Yeah, bouncer A suddenly looks 00:07:37.970 --> 00:07:40.269 terrible in that scenario. Exactly. Which is 00:07:40.269 --> 00:07:42.269 why data scientists often prefer a metric called 00:07:42.269 --> 00:07:45.529 the true positive rate, or TPR, to solve this. 00:07:45.750 --> 00:07:48.449 TPR. Yeah, TPR measures proportions instead of 00:07:48.449 --> 00:07:51.810 raw volume. So in your example, feature A might 00:07:51.810 --> 00:07:56.300 have an EP of 6, but its TPR is only 0 .73. Meaning 00:07:56.300 --> 00:07:59.819 he caught 73 % of the kids. Right. Feature B 00:07:59.819 --> 00:08:03.939 has a lower EP of 4, but a higher TPR of 0 .75. 00:08:04.279 --> 00:08:07.720 Ah, he caught 75%. So an experienced user recognizes 00:08:07.720 --> 00:08:09.899 that Feature B is proportionally more accurate 00:08:09.899 --> 00:08:12.240 across the whole data set. So EP gives you a 00:08:12.240 --> 00:08:14.899 quick and dirty volume check, while TPR reveals 00:08:14.899 --> 00:08:17.259 the deeper proportional truth. Exactly. But those 00:08:17.259 --> 00:08:19.439 are just two ways to evaluate a split, right? 00:08:19.779 --> 00:08:21.699 The CART algorithm we mentioned earlier relies 00:08:21.699 --> 00:08:24.660 on a completely different metric. Yes. CART uses 00:08:24.660 --> 00:08:27.160 something called Genine Impurity. Genine Impurity. 00:08:27.339 --> 00:08:29.860 Sounds like a magic lamp situation. Not quite. 00:08:30.500 --> 00:08:33.139 Genie Impurity measures the probability that 00:08:33.139 --> 00:08:36.100 a randomly chosen element from a set would be 00:08:36.100 --> 00:08:38.759 mislabeled if it were randomly assigned a label. 00:08:38.940 --> 00:08:41.460 based on the distribution in that specific node. 00:08:41.679 --> 00:08:44.919 That is a mouthful. It is. Basically, the algorithm's 00:08:44.919 --> 00:08:47.299 goal is to drive this number as close to zero 00:08:47.299 --> 00:08:49.840 as possible. Why zero? Because a score of zero 00:08:49.840 --> 00:08:52.559 means every single item in that node falls into 00:08:52.559 --> 00:08:55.440 a single target category. The room is completely 00:08:55.440 --> 00:08:58.340 pure. OK, the room is pure. But here's the crazy 00:08:58.340 --> 00:09:01.759 part. The source connects genie impurity to a 00:09:01.759 --> 00:09:04.240 mind blowing concept in physics called salus 00:09:04.240 --> 00:09:07.019 entropy. Yes, it does. How does sorting data 00:09:07.019 --> 00:09:09.399 relate to quantum systems? That just blew my 00:09:09.399 --> 00:09:11.179 mind. It's actually really elegant. Think of 00:09:11.179 --> 00:09:13.559 entropy as a measure of chaos or randomness. 00:09:14.220 --> 00:09:16.779 Like a teenager's messy bedroom is in a state 00:09:16.779 --> 00:09:18.600 of high entropy. Oh, I have kids. I know high 00:09:18.600 --> 00:09:20.519 entropy. Right. In thermodynamics and quantum 00:09:20.519 --> 00:09:23.240 mechanics, systems naturally lean toward chaos. 00:09:23.879 --> 00:09:26.320 OK. Salus entropy specifically measures a lack 00:09:26.320 --> 00:09:28.440 of information and out of equilibrium systems. 00:09:28.740 --> 00:09:31.179 So when a decision tree calculates Gini impurity, 00:09:31.539 --> 00:09:34.639 it is essentially applying the physics of thermodynamics 00:09:34.639 --> 00:09:38.309 to structure data. It is looking for the specific 00:09:38.309 --> 00:09:40.909 question that cleans the messy room the fastest. 00:09:42.669 --> 00:09:45.429 Reducing the chaos and forcing the data into 00:09:45.429 --> 00:09:48.500 an organized state. That is wild. A computer 00:09:48.500 --> 00:09:50.779 approving a credit card application is literally 00:09:50.779 --> 00:09:53.100 using the mathematical principles of quantum 00:09:53.100 --> 00:09:55.899 chaos to organize your financial history. It 00:09:55.899 --> 00:09:59.559 really is. And other algorithms like ID3 or C4 00:09:59.559 --> 00:10:02.899 .5 tackle this chaos using a different metric 00:10:02.899 --> 00:10:05.559 called information gain. Which stems from the 00:10:05.559 --> 00:10:08.019 Shannon index in information theory, right? Exactly. 00:10:08.419 --> 00:10:11.320 But the underlying goal remains the same. Reduce 00:10:11.320 --> 00:10:13.909 randomness. The source illustrates information 00:10:13.909 --> 00:10:15.970 gain with a classic weather example, which I 00:10:15.970 --> 00:10:17.970 really like. Oh yeah, the tennis one. Yeah. Imagine 00:10:17.970 --> 00:10:20.269 you have 14 days of historical data and you want 00:10:20.269 --> 00:10:22.269 to build a decision tree to predict whether you 00:10:22.269 --> 00:10:24.269 should go outside and play tennis. Right. You 00:10:24.269 --> 00:10:26.230 have four variables to choose from. The overall 00:10:26.230 --> 00:10:28.789 outlook, the temperature, the humidity, and whether 00:10:28.789 --> 00:10:31.929 or not it is windy. So the algorithm evaluates 00:10:31.929 --> 00:10:35.549 the entropy, the total randomness of those 14 00:10:35.549 --> 00:10:38.090 days before any split happens at all. It looks 00:10:38.090 --> 00:10:41.769 at the big messy room first. Exactly. Then... 00:10:41.840 --> 00:10:45.759 It imagines splitting the data by, let's say, 00:10:45.919 --> 00:10:48.700 the windy variable. It creates a pile for windy 00:10:48.700 --> 00:10:51.460 is true, and a pile for windy is false. And it 00:10:51.460 --> 00:10:54.080 checks how many yes we played and no we didn't 00:10:54.080 --> 00:10:57.259 days end up in each of those new piles. So it's 00:10:57.259 --> 00:10:59.840 basically testing out a question to see how clean 00:10:59.840 --> 00:11:03.220 the piles get. Yes. Information gain is calculated 00:11:03.220 --> 00:11:06.919 by taking the original chaos of the 14 days and 00:11:06.919 --> 00:11:09.120 subtracting the weighted sum of the chaos in 00:11:09.120 --> 00:11:11.360 the two new piles. And it does this for all the 00:11:11.360 --> 00:11:13.759 variables. It runs this identical calculation 00:11:13.759 --> 00:11:16.580 for all four variables. Whichever variable yields 00:11:16.580 --> 00:11:19.299 the highest information gain. Meaning it annihilates 00:11:19.299 --> 00:11:21.639 the most randomness. Right. That one is crowned 00:11:21.639 --> 00:11:24.019 the winner and becomes the very first split at 00:11:24.019 --> 00:11:26.840 the root of the tree. Okay, so we have EP, TPR, 00:11:27.379 --> 00:11:30.250 genie impurity, and information gain. But the 00:11:30.250 --> 00:11:32.769 source introduces one more metric from the original 00:11:32.769 --> 00:11:36.090 1984 CART publication. Right. It's simply called 00:11:36.090 --> 00:11:39.049 the measure of goodness. Goodness. I love how 00:11:39.049 --> 00:11:40.929 simple that sounds compared to the solace entropy. 00:11:41.370 --> 00:11:43.629 Seriously? This metric acts as a diplomat, actually. 00:11:43.769 --> 00:11:46.970 It's a diplomat? How so? Well, rather than aggressively 00:11:46.970 --> 00:11:49.450 pursuing absolute purity like information game 00:11:49.450 --> 00:11:52.809 does, the goodness metric seeks to balance creating 00:11:52.809 --> 00:11:56.370 pure children nodes with creating equally sized 00:11:56.370 --> 00:11:59.940 children nodes. Ah, okay. The source frames this 00:11:59.940 --> 00:12:02.980 with a credit risk scenario. You have eight bank 00:12:02.980 --> 00:12:05.720 customers and you know their savings, assets, 00:12:06.000 --> 00:12:07.840 income, and whether their final credit risk was 00:12:07.840 --> 00:12:10.240 labeled good or bad. Right. And the goodness 00:12:10.240 --> 00:12:12.740 function methodically evaluates every possible 00:12:12.740 --> 00:12:15.159 split. So if it considers dividing the customers 00:12:15.159 --> 00:12:17.700 by a low savings threshold? It runs a specialized 00:12:17.700 --> 00:12:20.659 formula. This formula weighs the proportion of 00:12:20.659 --> 00:12:22.960 records sent down the left branch versus the 00:12:22.960 --> 00:12:25.700 right branch while simultaneously evaluating 00:12:25.700 --> 00:12:28.159 the purity of the good and bad credit labels 00:12:28.159 --> 00:12:30.559 within those new branches. So it deliberately 00:12:30.559 --> 00:12:33.539 builds a more balanced symmetrical tree. Exactly. 00:12:33.799 --> 00:12:36.039 It is willing to sacrifice a tiny bit of purity 00:12:36.039 --> 00:12:38.500 if it means keeping the branches relatively even. 00:12:39.019 --> 00:12:41.720 Which I assume ensures the algorithm's decision 00:12:41.720 --> 00:12:44.340 time remains consistent across different predictions. 00:12:44.460 --> 00:12:47.340 You've got it. But... This raises an important 00:12:47.340 --> 00:12:49.139 question about the fundamental nature of these 00:12:49.139 --> 00:12:51.360 calculations. Actually, hold on. I need to push 00:12:51.360 --> 00:12:53.059 back on this entire process for a second. Oh, 00:12:53.059 --> 00:12:56.480 right. Let's hear it. Every single metric we 00:12:56.480 --> 00:13:00.000 just discussed, GNA, information gain, the goodness 00:13:00.000 --> 00:13:03.460 function, they all calculate the absolute best 00:13:03.460 --> 00:13:06.460 move for the immediate next split. Correct. Isn't 00:13:06.460 --> 00:13:08.840 that incredibly short -sighted? I mean, it feels 00:13:08.840 --> 00:13:11.220 like moving a pawn in chess just to capture a 00:13:11.220 --> 00:13:13.460 knight, completely blind to the fact that you 00:13:13.460 --> 00:13:15.940 are leaving your queen totally exposed three 00:13:15.940 --> 00:13:18.500 moves down the line. That is a very apt analogy. 00:13:18.799 --> 00:13:21.080 Right. If it only looks at the immediate step, 00:13:21.259 --> 00:13:24.399 it must be missing better overall tree configurations. 00:13:24.700 --> 00:13:26.960 Well, your skepticism hits on the central flaw 00:13:26.960 --> 00:13:30.299 of this entire architecture. Computer scientists 00:13:30.299 --> 00:13:32.980 literally classify these algorithms as greedy. 00:13:33.149 --> 00:13:36.350 Greedy algorithms. Yes. They make locally optimal 00:13:36.350 --> 00:13:39.149 choices at each individual node, crossing their 00:13:39.149 --> 00:13:41.509 fingers that it somehow leads to a globally optimal 00:13:41.509 --> 00:13:43.970 tree. Which it probably doesn't always do. Definitely 00:13:43.970 --> 00:13:47.559 not. Because finding the absolute mathematically 00:13:47.559 --> 00:13:50.980 perfect decision tree is a problem known as NP 00:13:50.980 --> 00:13:53.440 -complete. NP -complete? What does that mean 00:13:53.440 --> 00:13:56.419 in practical terms? In practical terms, it means 00:13:56.419 --> 00:13:59.259 if you gave a supercomputer a moderately sized 00:13:59.259 --> 00:14:02.980 data set and asked it to test every single possible 00:14:02.980 --> 00:14:05.600 tree configuration to find the undisputed perfect 00:14:05.600 --> 00:14:08.299 one, the sun would burn out before your laptop 00:14:08.299 --> 00:14:11.580 finished the math. Wait, really? The sun would 00:14:11.580 --> 00:14:14.360 burn out. The computational load is astronomically 00:14:14.360 --> 00:14:17.379 impossible. Because we cannot calculate the perfect 00:14:17.379 --> 00:14:20.779 tree, we rely on these greedy heuristics to rapidly 00:14:20.779 --> 00:14:24.200 generate a tree that is just good enough. But 00:14:24.200 --> 00:14:27.419 that greedy nature introduces severe vulnerabilities, 00:14:27.860 --> 00:14:30.179 right? Like, the source notes that decision trees 00:14:30.179 --> 00:14:33.200 are notoriously non -robust. The highly non -robust. 00:14:33.320 --> 00:14:36.019 It says a tiny, seemingly insignificant typo 00:14:36.019 --> 00:14:38.500 or alteration in the initial training data can 00:14:38.500 --> 00:14:40.960 cascade through all those greedy splits, resulting 00:14:40.960 --> 00:14:43.279 in a completely different tree structure. And 00:14:43.279 --> 00:14:45.860 wildly different final predictions. That's terrifying. 00:14:46.039 --> 00:14:49.039 And it gets worse. They also suffer from a fatal 00:14:49.039 --> 00:14:51.980 flaw called overfitting. Overfitting? A greedy 00:14:51.980 --> 00:14:54.320 algorithm will sometimes try too hard to reduce 00:14:54.320 --> 00:14:56.860 entropy. Like trying to make the room perfectly 00:14:56.860 --> 00:14:59.919 clean. Exactly. It continues to split the data 00:14:59.919 --> 00:15:03.240 until it creates an insanely complex convoluted 00:15:03.240 --> 00:15:06.779 tree that perfectly memorizes every single anomaly, 00:15:07.259 --> 00:15:11.590 outlier, and typo in the training data. It builds 00:15:11.590 --> 00:15:13.909 a custom rule for every single data point. Yes. 00:15:14.090 --> 00:15:16.009 But the moment you release that overfitted model 00:15:16.009 --> 00:15:18.470 into the real world to evaluate brand new data, 00:15:18.970 --> 00:15:21.549 it just fails miserably. Because it memorized 00:15:21.549 --> 00:15:23.789 the past instead of learning the underlying patterns. 00:15:24.029 --> 00:15:26.029 It's like relying on a single decision tree is 00:15:26.029 --> 00:15:28.529 like asking that one friend for advice who totally 00:15:28.529 --> 00:15:30.450 overthinks everything. We all have that friend. 00:15:30.610 --> 00:15:33.029 Right. They give you a ridiculously specific, 00:15:33.289 --> 00:15:36.169 brittle plan that falls apart the second it rains 00:15:36.169 --> 00:15:39.309 or traffic is bad. That is exactly what an over 00:15:39.309 --> 00:15:41.250 -fitted tree does. So how do data scientists 00:15:41.250 --> 00:15:43.970 fix this? The source mentions pruning? Yeah, 00:15:44.070 --> 00:15:46.470 pruning is literally cutting back the hyper -specific 00:15:46.470 --> 00:15:48.649 branches that don't provide broad predictive 00:15:48.649 --> 00:15:51.250 power. You just chop them off. Basically. But 00:15:51.250 --> 00:15:54.610 pruning only helps so much. The most robust solution 00:15:54.610 --> 00:15:56.830 to these inherent pitfalls is the application 00:15:56.830 --> 00:16:00.029 of ensemble methods. Ensemblement? Instead of 00:16:00.029 --> 00:16:03.210 relying on a single brittle tree, data scientists 00:16:03.210 --> 00:16:05.789 harness the collective power of multiple algorithms 00:16:05.789 --> 00:16:08.720 working in tandem. OK, let's contextualize how 00:16:08.720 --> 00:16:10.899 these ensemble methods actually work in the real 00:16:10.899 --> 00:16:13.100 world. Sure. Imagine a hospital is trying to 00:16:13.100 --> 00:16:15.600 build a reliable model to detect a rare disease 00:16:15.600 --> 00:16:18.659 and their single decision tree keeps overfitting 00:16:18.659 --> 00:16:21.419 to the specific patients in their trial. A very 00:16:21.419 --> 00:16:24.159 common problem. Right. So they decide to bring 00:16:24.159 --> 00:16:27.879 in an ensemble approach. How does a boosted tree 00:16:27.879 --> 00:16:31.820 like add a boost tackle this? So boosting introduces 00:16:31.820 --> 00:16:34.899 an element of iterative learning. The hospital 00:16:34.899 --> 00:16:38.169 builds a standard decision tree. Predictably, 00:16:38.289 --> 00:16:40.470 it makes some mistakes and misdiagnoses a few 00:16:40.470 --> 00:16:42.370 patients. Because it's a greedy little tree. 00:16:42.690 --> 00:16:45.070 Exactly. The Adaboost algorithm then builds a 00:16:45.070 --> 00:16:47.549 second tree, but it heavily weights the data 00:16:47.549 --> 00:16:49.789 from the patients the first tree got wrong. Oh, 00:16:49.789 --> 00:16:52.309 that's clever. Yeah, it forces the new tree to 00:16:52.309 --> 00:16:55.129 focus specifically on previous failures. So it 00:16:55.129 --> 00:16:57.509 learns from its mistakes. It builds a whole sequence 00:16:57.509 --> 00:16:59.769 of trees, actually, each compensating for the 00:16:59.769 --> 00:17:02.110 blind spots of its predecessor. Okay, that's 00:17:02.110 --> 00:17:04.809 Adaboost. What if the hospital uses a committee 00:17:04.809 --> 00:17:08.890 approach? The source also called it KDDT. The 00:17:08.890 --> 00:17:11.369 committee approach relies on randomized diversity. 00:17:12.009 --> 00:17:14.309 The hospital feeds the exact same patient data 00:17:14.309 --> 00:17:16.730 into several different randomized algorithms, 00:17:16.950 --> 00:17:19.049 generating a diverse array of decision trees. 00:17:19.230 --> 00:17:21.710 Like getting second, third, and fourth opinions. 00:17:21.990 --> 00:17:24.890 Exactly. When a new patient arrives, every tree 00:17:24.890 --> 00:17:27.750 in the committee evaluates the data and the final 00:17:27.750 --> 00:17:30.049 diagnosis is determined by a simple majority 00:17:30.049 --> 00:17:32.609 vote. That makes a lot of sense. Which brings 00:17:32.609 --> 00:17:35.990 us to bootstrap aggregated trees, which are commonly 00:17:35.990 --> 00:17:38.670 referred to as bagged trees. And the most famous 00:17:38.670 --> 00:17:41.289 application of this is the random forest classifier. 00:17:41.710 --> 00:17:43.470 Random forest? I've actually heard that term 00:17:43.470 --> 00:17:46.609 before. It's very popular. A random forest utilizes 00:17:46.609 --> 00:17:49.089 a technique called resampling with replacement. 00:17:49.309 --> 00:17:51.150 Okay, stick with the hospital analogy. How does 00:17:51.150 --> 00:17:53.190 that work? The hospital takes their original 00:17:53.190 --> 00:17:55.609 stack of a thousand patient files. They create 00:17:55.609 --> 00:17:58.329 a new stack of a thousand files by drawing randomly 00:17:58.329 --> 00:18:01.059 for the original pile. But... Because they replace 00:18:01.059 --> 00:18:03.740 the file after drawing it, the new stack might 00:18:03.740 --> 00:18:06.920 have, say, three duplicates of patient A, while 00:18:06.920 --> 00:18:10.119 patient B is completely missing. Oh, weird. Why 00:18:10.119 --> 00:18:12.000 would they want duplicates? Because they do this 00:18:12.000 --> 00:18:14.720 hundreds of times, building a massive forest 00:18:14.720 --> 00:18:17.900 of unique decision trees, each trained on a slightly 00:18:17.900 --> 00:18:20.720 skewed version of reality. That is wild. And 00:18:20.720 --> 00:18:23.960 then, when a new case appears, the entire forest 00:18:23.960 --> 00:18:26.609 votes on the outcome. So instead of asking your 00:18:26.609 --> 00:18:29.009 overthinking friend, it's like polling your entire 00:18:29.009 --> 00:18:31.230 group of friends and going with the reliable 00:18:31.230 --> 00:18:33.369 majority vote? That's exactly it. Okay, the last 00:18:33.369 --> 00:18:35.450 ensemble method detailed in the source is the 00:18:35.450 --> 00:18:38.250 rotation forest. How does that differ from the 00:18:38.250 --> 00:18:41.089 random resampling we just saw? A rotation forest 00:18:41.089 --> 00:18:43.769 fundamentally alters the data itself before the 00:18:43.769 --> 00:18:46.190 tree ever even looks at it. Alters it how? It 00:18:46.190 --> 00:18:48.210 applies a mathematical technique called principal 00:18:48.210 --> 00:18:51.930 component analysis, or PCA, to random subsets 00:18:51.930 --> 00:18:54.170 of the input features. So instead of just looking 00:18:54.170 --> 00:18:56.309 at raw variables like blood pressure and age 00:18:56.309 --> 00:18:59.589 separately? PCA might mathematically fuse them 00:18:59.589 --> 00:19:02.809 together into a new hybrid variable. It physically 00:19:02.809 --> 00:19:05.759 shifts the perspective of the data. So it forces 00:19:05.759 --> 00:19:08.579 the tree to look at the medical charts from completely 00:19:08.579 --> 00:19:11.079 different angles before making a split. Yes. 00:19:11.480 --> 00:19:13.660 By combining these diverse, slightly rotated 00:19:13.660 --> 00:19:16.559 perspectives, the final ensemble prediction becomes 00:19:16.559 --> 00:19:19.740 incredibly resilient to the noise and typos that 00:19:19.740 --> 00:19:22.549 would completely shatter a single... greedy tree. 00:19:23.190 --> 00:19:26.210 OK, so we have dissected a flow chart that uses 00:19:26.210 --> 00:19:29.130 quantum entropy metrics, struggles with short 00:19:29.130 --> 00:19:32.289 -sighted greed, and requires entire simulated 00:19:32.289 --> 00:19:35.069 forests of algorithms to just remain stable. 00:19:35.329 --> 00:19:36.730 It sounds like a lot of work when you summarize 00:19:36.730 --> 00:19:39.029 it like that. It does. Which makes me wonder, 00:19:39.369 --> 00:19:41.569 why do data scientists still bother with this 00:19:41.569 --> 00:19:44.410 architecture when we possess incredibly advanced 00:19:44.410 --> 00:19:47.470 artificial neural networks? Well, the primary 00:19:47.470 --> 00:19:49.769 advantage of a decision tree and it is a massive 00:19:49.769 --> 00:19:52.690 irreplaceable advantage, is its status as a white 00:19:52.690 --> 00:19:56.250 box model. A white box or open box? Yes, which 00:19:56.250 --> 00:19:58.450 directly counters the terrifying black box we 00:19:58.450 --> 00:20:00.089 discussed at the beginning of this deep dive. 00:20:00.269 --> 00:20:02.690 Oh, right. The loan officer algorithm. Exactly. 00:20:03.109 --> 00:20:05.009 An artificial neural network might deliver a 00:20:05.009 --> 00:20:06.789 highly accurate prediction regarding a medical 00:20:06.789 --> 00:20:10.250 diagnosis. But if a doctor asks the network why 00:20:10.250 --> 00:20:12.710 it reached that conclusion, the machine just 00:20:12.710 --> 00:20:16.519 shrugs. The output is just millions of incomprehensible 00:20:16.519 --> 00:20:19.779 mathematical weights. A human cannot audit the 00:20:19.779 --> 00:20:22.539 neural network's internal logic. But a decision 00:20:22.539 --> 00:20:25.400 tree's logic is entirely observable. Because 00:20:25.400 --> 00:20:27.880 it is grounded in simple Boolean logic. True 00:20:27.880 --> 00:20:31.119 or false, yes or no. You can trace the path from 00:20:31.119 --> 00:20:33.849 the root to the leaf with your own finger. The 00:20:33.849 --> 00:20:36.529 source notes several practical engineering advantages 00:20:36.529 --> 00:20:39.710 as well, besides just transparency. Like, decision 00:20:39.710 --> 00:20:42.910 trees require remarkably little data preparation. 00:20:43.150 --> 00:20:45.190 Oh, they're a dream to work with in that regard. 00:20:45.829 --> 00:20:47.849 Yeah. Data scientists don't have to spend hours 00:20:47.849 --> 00:20:50.750 normalizing data or creating complex dummy variables. 00:20:50.910 --> 00:20:53.210 It just takes the raw info. Yeah. A decision 00:20:53.210 --> 00:20:56.190 tree seamlessly processes numerical data, like 00:20:56.190 --> 00:20:58.990 a patient's exact income, and categorical data, 00:20:59.009 --> 00:21:01.619 like a patient's eye color, simultaneously. They 00:21:01.619 --> 00:21:03.940 also perform built -in feature selection, right? 00:21:03.980 --> 00:21:06.420 They do. Because the hierarchy inherently reveals 00:21:06.420 --> 00:21:08.720 which attributes carry the most weight. Exactly. 00:21:08.819 --> 00:21:10.660 The feature sitting at the very top of the tree 00:21:10.660 --> 00:21:13.839 providing the highest information gain are demonstrably 00:21:13.839 --> 00:21:16.740 the most important variables. So irrelevant data 00:21:16.740 --> 00:21:19.420 simply falls away without needing to be manually 00:21:19.420 --> 00:21:21.779 scrubbed by a human. Right. And if we connect 00:21:21.779 --> 00:21:24.440 this to the bigger picture, the enduring relevance 00:21:24.440 --> 00:21:27.660 of decision trees really lies in how closely 00:21:27.660 --> 00:21:31.009 they mirror actual human decision making. Which 00:21:31.009 --> 00:21:33.009 brings us back around. So what does this all 00:21:33.009 --> 00:21:35.430 mean for you, the listener, when you find yourself 00:21:35.430 --> 00:21:38.009 sitting across from that loan officer? The stakes 00:21:38.009 --> 00:21:41.069 are real. Very real. If an algorithm decides 00:21:41.069 --> 00:21:43.730 to deny your mortgage or if a machine learning 00:21:43.730 --> 00:21:47.109 model flags your medical scam for a biopsy, you 00:21:47.109 --> 00:21:50.009 desperately want a white box decision tree operating 00:21:50.009 --> 00:21:52.109 behind the scenes. You need to know why. You 00:21:52.109 --> 00:21:54.529 want a model capable of printing out the exact 00:21:54.529 --> 00:21:57.950 flow chart to say, hey, We denied your loan specifically 00:21:57.950 --> 00:22:00.150 because your liquid assets were low and your 00:22:00.150 --> 00:22:02.849 debt to income ratio exceeded this exact threshold. 00:22:03.109 --> 00:22:06.130 You demand intelligibility, not just raw algorithmic 00:22:06.130 --> 00:22:08.569 power. Right. What's fascinating here is the 00:22:08.569 --> 00:22:11.410 ongoing tension between raw predictive capability 00:22:11.410 --> 00:22:14.529 and human accountability. It's a huge debate 00:22:14.529 --> 00:22:17.250 in the tech world right now. In critical industries 00:22:17.250 --> 00:22:19.849 like finance, healthcare, and criminal justice, 00:22:20.369 --> 00:22:22.710 transparency is not just a preference, it's a 00:22:22.710 --> 00:22:25.559 necessity. Absolutely. Decision trees provide 00:22:25.559 --> 00:22:28.380 that crucial why in a landscape that is just 00:22:28.380 --> 00:22:31.799 increasingly crowded with unexplainable AI. They 00:22:31.799 --> 00:22:34.660 anchor complex predictions in verifiable logic. 00:22:34.980 --> 00:22:37.420 They really do. So to briefly recap our journey 00:22:37.420 --> 00:22:40.009 today. We started with the simple, relatable 00:22:40.009 --> 00:22:42.829 concept of an algorithmic 20 questions game. 00:22:42.990 --> 00:22:45.349 The visual baseline. Right. Then we challenged 00:22:45.349 --> 00:22:48.049 the short -sighted, greedy mathematics of genie 00:22:48.049 --> 00:22:50.890 impurities and information gain that desperately 00:22:50.890 --> 00:22:53.410 tried to reduce chaos in the data room. Using 00:22:53.410 --> 00:22:56.549 salus entropy. Which still blows my mind. Then 00:22:56.549 --> 00:22:59.349 we explored how easily a single tree can become 00:22:59.349 --> 00:23:02.109 brittle and overfit to its training data. Like 00:23:02.109 --> 00:23:04.970 the overthinking friend. Exactly. And we saw 00:23:04.970 --> 00:23:07.430 how data scientists overcome those inherent flaws 00:23:07.430 --> 00:23:09.940 by harnessing the collective democratic power 00:23:09.940 --> 00:23:12.599 of ensembles and random forests. Bring in the 00:23:12.599 --> 00:23:14.599 committee. Ultimately, we discovered why the 00:23:14.599 --> 00:23:17.019 transparent white box nature of this architecture 00:23:17.019 --> 00:23:19.220 makes it absolutely essential for maintaining 00:23:19.220 --> 00:23:21.920 human trust and predictive models. Which honestly 00:23:21.920 --> 00:23:24.259 leaves us with a rather provocative thought to 00:23:24.259 --> 00:23:27.619 mull over. Oh, lay it on me. Well, throughout 00:23:27.619 --> 00:23:30.920 the source material, decision trees are repeatedly 00:23:30.920 --> 00:23:33.599 praised for perfectly mirroring human decision 00:23:33.599 --> 00:23:37.089 making. Right. Yeah, splitting complex, overwhelming 00:23:37.089 --> 00:23:41.190 data into manageable binary choices. We naturally 00:23:41.190 --> 00:23:43.809 build mental flow charts to navigate our lives 00:23:43.809 --> 00:23:46.670 all the time. But given everything we've just 00:23:46.670 --> 00:23:48.869 discussed regarding the brittleness of a single 00:23:48.869 --> 00:23:51.650 decision tree... Uh -oh. And how achieving a 00:23:51.650 --> 00:23:55.329 truly robust, accurate prediction actually requires 00:23:55.329 --> 00:23:58.490 an entire random forest of subconscious algorithms 00:23:58.490 --> 00:24:01.289 constantly voting in the background. Oh, wow. 00:24:01.609 --> 00:24:03.930 I see where you're going. It fundamentally challenges 00:24:03.930 --> 00:24:06.470 our perception of ourselves. Does human intuition 00:24:06.470 --> 00:24:09.529 actually exist as a single clear thought? Probably 00:24:09.529 --> 00:24:12.349 not. Or are our brains just running incredibly 00:24:12.349 --> 00:24:15.190 deep, unpruned, random forests that we simply 00:24:15.190 --> 00:24:17.990 cannot consciously interpret, delivering a final 00:24:17.990 --> 00:24:20.170 gut feeling that we mistakenly call intuition? 00:24:20.470 --> 00:24:22.630 Are we all just random forests pretending to 00:24:22.630 --> 00:24:25.690 be a single logical tree? It's entirely possible. 00:24:25.890 --> 00:24:27.509 Keep that in mind the next time you make a snap 00:24:27.509 --> 00:24:29.329 decision. Thanks for joining us and we'll catch 00:24:29.329 --> 00:24:30.269 you on the next Deep Dive.