WEBVTT 00:00:00.000 --> 00:00:03.720 Imagine someone like looking at a chart, tapping 00:00:03.720 --> 00:00:07.080 a pen, and confidently predicting a sudden spike 00:00:07.080 --> 00:00:09.640 in venomous spider bites across the United States. 00:00:09.800 --> 00:00:13.179 Which is a terrifying thought. Right. And they're 00:00:13.179 --> 00:00:15.220 making this prediction simply by looking at the 00:00:15.220 --> 00:00:17.710 winning word of the national spelling bee. It 00:00:17.710 --> 00:00:20.410 just, it sounds completely unhinged. Totally 00:00:20.410 --> 00:00:22.609 unhinged. You'd assume they were joking. But, 00:00:22.910 --> 00:00:24.969 well, in the world of invisible data trailing 00:00:24.969 --> 00:00:27.629 behind you every single day, mathematical models 00:00:27.629 --> 00:00:31.769 can actually draw those exact, terrifyingly precise 00:00:31.769 --> 00:00:34.280 conclusions. Yeah, they really can. Welcome to 00:00:34.280 --> 00:00:36.840 the deep dive. Today we're stripping away the, 00:00:36.840 --> 00:00:38.719 you know, the corporate marketing jargon from 00:00:38.719 --> 00:00:40.899 a term you have definitely heard but almost certainly 00:00:40.899 --> 00:00:43.359 misunderstand. We are diving into the complex 00:00:43.359 --> 00:00:47.140 history, the underlying math, and the legal battle 00:00:47.140 --> 00:00:50.159 surrounding data mining. It's a phrase that absolutely 00:00:50.159 --> 00:00:52.649 dominates business intelligence, right? and, 00:00:52.649 --> 00:00:54.909 you know, tech keynotes. Oh, everywhere. Yeah. 00:00:55.189 --> 00:00:57.390 But when you look at it through a strict academic 00:00:57.390 --> 00:01:01.409 lens, data mining is this very specific, highly 00:01:01.409 --> 00:01:04.510 technical intersection. It's the exact point 00:01:04.510 --> 00:01:09.000 where machine learning, complex statistics, and 00:01:09.000 --> 00:01:12.079 massive database systems all just collide. Okay, 00:01:12.079 --> 00:01:14.099 let's unpack this because right out of the gate 00:01:14.099 --> 00:01:16.239 we have to clear up a massive misconception. 00:01:16.560 --> 00:01:19.439 The name itself. Exactly. The very term data 00:01:19.439 --> 00:01:23.079 mining is a complete misnomer. It paints like 00:01:23.079 --> 00:01:25.359 the completely wrong picture in your head. It 00:01:25.359 --> 00:01:27.579 really does. I mean the name suggests a totally 00:01:27.579 --> 00:01:30.340 different physical action than what the algorithms 00:01:30.340 --> 00:01:32.099 are actually doing. Right, because if you think 00:01:32.099 --> 00:01:33.980 about traditional mining like, I don't know, 00:01:34.099 --> 00:01:35.709 panning for gold, you might assume data Again 00:01:35.709 --> 00:01:38.230 mining means digging around to find new data. 00:01:38.370 --> 00:01:40.829 Which it isn't. No. You aren't mining for dirt. 00:01:40.950 --> 00:01:43.090 You already own the mountain of dirt. It's less 00:01:43.090 --> 00:01:45.689 like swinging a pickaxe to find new data and 00:01:45.689 --> 00:01:50.390 more like running a giant highly calibrated electromagnet 00:01:50.390 --> 00:01:52.790 over the mountain of data you already have just 00:01:52.790 --> 00:01:55.129 to pull out the hidden iron shavings. That's 00:01:55.129 --> 00:01:57.469 a great way to put it. And those iron shavings 00:01:57.469 --> 00:02:01.370 are the patterns. The goal isn't extracting information. 00:02:01.760 --> 00:02:04.579 It's extracting relationships from within the 00:02:04.579 --> 00:02:07.620 information. That magnet analogy really hits 00:02:07.620 --> 00:02:09.159 the nail on the head. You're pulling out the 00:02:09.159 --> 00:02:11.840 invisible structures. And to really grasp the 00:02:11.840 --> 00:02:14.560 mechanics here, I think we need to draw a hard 00:02:14.560 --> 00:02:18.020 definitive line between standard data analysis 00:02:18.020 --> 00:02:21.090 and actual data mining. Because they're not the 00:02:21.090 --> 00:02:23.469 same thing. Not at all. They operate on entirely 00:02:23.469 --> 00:02:25.310 reversed philosophies. OK, let's play that out. 00:02:25.349 --> 00:02:28.830 Say I run a retail chain, right, and I just ran 00:02:28.830 --> 00:02:32.030 a massive holiday promotion. If I'm doing just 00:02:32.030 --> 00:02:34.030 standard data analysis, how am I approaching 00:02:34.030 --> 00:02:36.650 that? Well, with standard data analysis, you 00:02:36.650 --> 00:02:39.050 approach the database with a pre -existing hypothesis. 00:02:39.270 --> 00:02:43.210 OK. You ask a direct question like, did the holiday 00:02:43.210 --> 00:02:45.050 promotion increase sales in the footwear department 00:02:45.050 --> 00:02:47.650 by 20 percent? Got it. The system runs the numbers 00:02:47.650 --> 00:02:50.449 and the data gives you a yes or a no. You knew 00:02:50.449 --> 00:02:51.990 exactly what you were looking for before you 00:02:51.990 --> 00:02:53.969 ever even touched the keyboard. I had the question 00:02:53.969 --> 00:02:56.849 before I had the answer. Precisely. Now data 00:02:56.849 --> 00:02:59.810 mining completely flips that dynamic. Data mining 00:02:59.810 --> 00:03:03.389 deploys machine learning models to uncover clandestine 00:03:03.389 --> 00:03:06.349 hidden patterns that no human being even knew 00:03:06.349 --> 00:03:09.509 to look for. Wow. Yeah, you aren't testing a 00:03:09.509 --> 00:03:12.479 hypothesis at all. You're setting up the algorithms 00:03:12.479 --> 00:03:15.659 to generate entirely new hypotheses from scratch. 00:03:15.939 --> 00:03:17.580 So you're just handing over the mountain of dirt 00:03:17.580 --> 00:03:19.400 and saying, you know, show me what I don't know, 00:03:19.479 --> 00:03:22.219 which is wild because historically approaching 00:03:22.219 --> 00:03:24.759 science that way was considered practically unethical. 00:03:24.780 --> 00:03:26.919 Oh, it was a massive faux pas. Yeah. If we look 00:03:26.919 --> 00:03:30.120 back at the 1960s, statisticians and economists 00:03:30.120 --> 00:03:33.060 had incredibly negative terminology for this. 00:03:33.539 --> 00:03:37.270 They refer to it as data phishing or Data dredging. 00:03:37.330 --> 00:03:39.430 Data dredging sounds like you're dragging a swamp 00:03:39.430 --> 00:03:41.629 for a body. I mean, that was exactly the intended 00:03:41.629 --> 00:03:44.509 tone. Really? Yeah. The prevailing academic wisdom 00:03:44.509 --> 00:03:48.150 was that analyzing data without an a priori hypothesis, 00:03:48.409 --> 00:03:50.310 meaning without a specific defined question you 00:03:50.310 --> 00:03:52.330 were trying to answer, was just terrible practice. 00:03:52.469 --> 00:03:54.250 Because you're just looking for random stuff. 00:03:54.469 --> 00:03:56.770 Right. The assumption was that you were merely 00:03:56.770 --> 00:03:58.889 fishing for random coincidences and trying to 00:03:58.889 --> 00:04:01.210 pass them off as real science. There was actually 00:04:01.210 --> 00:04:04.270 an economist, Michael Lovell, who published a 00:04:04.270 --> 00:04:06.710 pretty famous critique in the early 80s where 00:04:06.710 --> 00:04:09.310 he explicitly attacked the practice. What did 00:04:09.310 --> 00:04:12.189 he say? He argued that it masqueraded under aliases, 00:04:12.530 --> 00:04:14.509 calling it experimentation if you're being generous 00:04:14.509 --> 00:04:18.430 or snooping if you're being honest. data snooping. 00:04:18.850 --> 00:04:20.970 So how do we get from data snooping being this 00:04:20.970 --> 00:04:25.209 like statistical sin to data mining being a cornerstone 00:04:25.209 --> 00:04:27.829 of modern tech? Well what's fascinating here 00:04:27.829 --> 00:04:31.120 is how the database community essentially staged 00:04:31.120 --> 00:04:34.459 a philosophical coup in the 1990s. How so? Computing 00:04:34.459 --> 00:04:36.879 powers were just exploding. Datasets were growing 00:04:36.879 --> 00:04:39.420 exponentially, moving from megabytes to gigabytes 00:04:39.420 --> 00:04:42.259 to terabytes. And researchers quickly realized 00:04:42.259 --> 00:04:44.800 that human beings simply could not form hypotheses 00:04:44.800 --> 00:04:47.500 fast enough. Or complex enough, probably. Exactly. 00:04:47.759 --> 00:04:49.639 We couldn't process all this information manually. 00:04:49.839 --> 00:04:52.139 The algorithms had to do the heavy lifting. Human 00:04:52.139 --> 00:04:54.379 limitation basically transformed the practice 00:04:54.379 --> 00:04:57.160 from an academic insult into an absolute necessity. 00:04:57.319 --> 00:05:00.699 And the rebranding of the field has this brilliant 00:05:00.699 --> 00:05:02.800 piece of trivia attached to it from the source 00:05:02.800 --> 00:05:05.699 material. Oh, the trademark issue. Yes. When 00:05:05.699 --> 00:05:08.180 researchers in the 80s wanted to legitimize this 00:05:08.180 --> 00:05:11.129 new science, they actually tried to name it database 00:05:11.129 --> 00:05:13.829 mining. But they couldn't. Legally couldn't. 00:05:13.930 --> 00:05:17.810 A company down in San Diego called HNC had already 00:05:17.810 --> 00:05:20.589 trademarked the phrase database mining to market 00:05:20.589 --> 00:05:23.370 their specific workstation. The whole academic 00:05:23.370 --> 00:05:25.970 community was boxed out by a trademark. Exactly. 00:05:26.069 --> 00:05:28.610 They literally didn't own the rights to the phrase. 00:05:29.200 --> 00:05:31.319 So, researchers were forced to shorten it, and 00:05:31.319 --> 00:05:34.500 they settled on data mining. The new name stuck, 00:05:34.899 --> 00:05:36.759 the whole phishing thing was completely forgotten, 00:05:37.120 --> 00:05:40.040 and the modern era of the algorithms began. But 00:05:40.040 --> 00:05:42.319 shedding that negative reputation meant the industry 00:05:42.319 --> 00:05:44.439 needed to prove its math was actually sound. 00:05:44.540 --> 00:05:46.420 Right. You need proof. Yeah. If you're going 00:05:46.420 --> 00:05:49.079 to let machines hunt for hidden patterns autonomously, 00:05:49.439 --> 00:05:51.800 you need an ironclad framework to ensure the 00:05:51.800 --> 00:05:54.319 output isn't just, you know, statistical garbage. 00:05:54.420 --> 00:05:56.639 Which brings us to how the sausage is actually 00:05:56.639 --> 00:06:01.420 made. is a concept called KDD. That's knowledge 00:06:01.420 --> 00:06:05.819 discovery in databases. Yes. And KDD is crucial 00:06:05.819 --> 00:06:08.459 to understand because the actual mining... part, 00:06:08.600 --> 00:06:11.279 the algorithm running the numbers, is just one 00:06:11.279 --> 00:06:13.879 single step in a much larger, highly rigorous 00:06:13.879 --> 00:06:15.680 workflow. Right, because you don't just point 00:06:15.680 --> 00:06:17.759 a machine learning model at a server farm and 00:06:17.759 --> 00:06:19.699 press a big green button. I wish it were that 00:06:19.699 --> 00:06:21.759 easy. There's a whole methodology behind it. 00:06:22.079 --> 00:06:24.899 The most dominant framework used by professionals, 00:06:25.100 --> 00:06:28.000 it's consistently topped industry polls for decades, 00:06:28.360 --> 00:06:31.240 is something called CrispusDM. Cross industry 00:06:31.240 --> 00:06:33.500 standard process for data mining. That's a mouthful. 00:06:33.680 --> 00:06:36.459 It is, but it's a brilliant structural approach 00:06:36.459 --> 00:06:39.920 because it breaks the entire chaos of data discovery 00:06:39.920 --> 00:06:43.680 down into six distinct manageable phases. And 00:06:43.680 --> 00:06:45.199 I think it's important to walk through them, 00:06:45.379 --> 00:06:47.720 because it demystifies what data scientists are 00:06:47.720 --> 00:06:49.819 actually doing all day. Let's do it. Walk me 00:06:49.819 --> 00:06:52.100 through the six phases. OK, so phase one is business 00:06:52.100 --> 00:06:54.819 understanding. OK. Before touching a single piece 00:06:54.819 --> 00:06:57.699 of data, you have to define the objective. What 00:06:57.699 --> 00:07:01.819 is the real world problem? Are we trying to stop 00:07:01.819 --> 00:07:03.899 credit card fraud, or are we trying to figure 00:07:03.899 --> 00:07:06.120 out why subscribers are canceling their streaming 00:07:06.120 --> 00:07:08.629 service? That makes total sense. You need a North 00:07:08.629 --> 00:07:12.089 Star to guide you. Phase two is data understanding. 00:07:12.410 --> 00:07:14.529 This is where you audit what you actually have. 00:07:14.629 --> 00:07:17.170 What's taking inventory. Yeah. You look at the 00:07:17.170 --> 00:07:20.269 raw information and ask, is this data sufficient 00:07:20.269 --> 00:07:22.509 to solve the problem we just defined? OK, so 00:07:22.509 --> 00:07:24.329 once you know your goal and you have your raw 00:07:24.329 --> 00:07:26.870 materials, you hit phase three, which I imagine 00:07:26.870 --> 00:07:30.449 is where things get really messy. Data preparation. 00:07:30.730 --> 00:07:33.170 Oh, it is notoriously the most time -consuming 00:07:33.170 --> 00:07:36.170 part of the entire KDD process. Really? Yeah. 00:07:36.509 --> 00:07:38.449 This is the pre -processing and cleaning phase. 00:07:39.009 --> 00:07:42.269 You're dealing with missing values, corrupt files, 00:07:42.829 --> 00:07:45.430 formatting inconsistencies. Oh, I see. To return 00:07:45.430 --> 00:07:47.810 to your earlier analogy, if your mountain of 00:07:47.810 --> 00:07:50.250 dirt is heavily contaminated with toxic waste, 00:07:50.790 --> 00:07:52.589 your electromagnet isn't going to find any clean 00:07:52.589 --> 00:07:55.290 iron shavings. The machine learning models require 00:07:55.290 --> 00:07:57.610 meticulously structured data to function. Okay, 00:07:57.610 --> 00:07:59.430 so we've defined the business goal, we audited 00:07:59.430 --> 00:08:01.370 the data, and we spent weeks cleaning it up. 00:08:01.529 --> 00:08:04.110 We're finally at phase four. Phase four is modeling. 00:08:04.589 --> 00:08:08.470 Yes. This is the actual data mining. This is 00:08:08.470 --> 00:08:11.250 where you deploy specific mathematical tasks 00:08:11.250 --> 00:08:13.209 to uncover the patterns. And this is where the 00:08:13.209 --> 00:08:15.230 math gets really interesting because there are 00:08:15.230 --> 00:08:17.350 different types of modeling tasks depending on 00:08:17.350 --> 00:08:19.269 what you actually want the machine to do. Right. 00:08:19.470 --> 00:08:21.670 Let's look at one called association rule learning. 00:08:22.009 --> 00:08:25.449 The classic real -world translation of this is 00:08:25.449 --> 00:08:28.990 market basket analysis. This is how the supermarket 00:08:28.990 --> 00:08:31.889 magically knows to put the tortilla chips right 00:08:31.889 --> 00:08:33.950 next to the salsa even though they belong an 00:08:33.950 --> 00:08:36.970 entirely different food category. Yes, but the 00:08:36.970 --> 00:08:41.409 mechanism behind how it knows that is pure probability. 00:08:41.809 --> 00:08:44.370 How so? Well, the algorithm isn't thinking about 00:08:44.490 --> 00:08:47.309 taste profiles or what makes a good snack. Right. 00:08:47.470 --> 00:08:50.350 It's analyzing millions of individual checkout 00:08:50.350 --> 00:08:53.289 receipts. It calculates the statistical probability 00:08:53.289 --> 00:08:55.789 that an item will appear in a transaction, and 00:08:55.789 --> 00:08:57.629 then it establishes a mathematical confidence 00:08:57.629 --> 00:08:59.769 score that two items will appear together. OK, 00:08:59.769 --> 00:09:02.769 so it's just pure numbers. Pure numbers. If millions 00:09:02.769 --> 00:09:05.309 of receipts show that whenever product A is purchased, 00:09:05.730 --> 00:09:08.509 product B has an 85 % probability of also being 00:09:08.509 --> 00:09:11.490 in the cart, the algorithm flags a high confidence 00:09:11.490 --> 00:09:15.820 association. The math proves a relationship that 00:09:15.820 --> 00:09:18.860 beats random chance. So association finds connections 00:09:18.860 --> 00:09:21.919 in a messy unstructured pile of data. But what 00:09:21.919 --> 00:09:23.899 if we already know the categories we care about 00:09:23.899 --> 00:09:27.620 and we just need the machine to sort new incoming 00:09:27.620 --> 00:09:30.120 items for us? That transitions us to a task called 00:09:30.120 --> 00:09:33.000 classification. OK. In classification, the algorithm 00:09:33.000 --> 00:09:35.879 learns the hidden structural rules of known data 00:09:35.879 --> 00:09:39.340 so it can categorize unseen data. The universal 00:09:39.340 --> 00:09:42.019 example here is your email spam filter. Right, 00:09:42.019 --> 00:09:43.860 but how is that fundamentally different from 00:09:43.860 --> 00:09:45.820 association? It's still just looking for patterns, 00:09:45.879 --> 00:09:48.039 isn't it? It is, but the difference is the training 00:09:48.039 --> 00:09:51.059 mechanism. With classification, you feed the 00:09:51.059 --> 00:09:53.059 algorithm millions of emails that have already 00:09:53.059 --> 00:09:55.580 been manually tagged by humans as either spam 00:09:55.580 --> 00:09:58.120 or legitimate. So it has an answer key. Exactly. 00:09:58.299 --> 00:10:00.139 The algorithm breaks down the vocabulary, the 00:10:00.139 --> 00:10:02.799 routing headers, the link structures, and mathematically 00:10:02.799 --> 00:10:04.899 defines what spam email looks like. And then 00:10:04.899 --> 00:10:07.639 it uses that rule. Yeah. Once the model is trained, 00:10:07.840 --> 00:10:11.179 it can instantly classify a brand new unseen 00:10:11.179 --> 00:10:13.779 email based on those learned mathematical weights. 00:10:14.179 --> 00:10:16.259 OK, so classification is sorting the knowns. 00:10:16.379 --> 00:10:18.919 But then we have anomaly detection. If I'm guessing 00:10:18.919 --> 00:10:21.820 based on the name, this is the algorithm looking 00:10:21.820 --> 00:10:23.860 for the one thing that doesn't fit the pattern. 00:10:24.100 --> 00:10:26.279 It's essentially the inverse of everything we 00:10:26.279 --> 00:10:28.899 just discussed. Instead of establishing the common 00:10:28.899 --> 00:10:31.960 pattern to sort things, the algorithm establishes 00:10:31.960 --> 00:10:35.539 the common pattern specifically to hunt for the 00:10:35.539 --> 00:10:38.419 extreme outlier. Like finding the needle by learning 00:10:38.419 --> 00:10:41.440 what the haystack looks like. Precisely. It measures 00:10:41.440 --> 00:10:43.500 the statistical distance of every data point 00:10:43.500 --> 00:10:46.200 from the mean average. When a record deviates 00:10:46.200 --> 00:10:48.700 so wildly from the standard range that the math 00:10:48.700 --> 00:10:51.620 flags it as an anomaly, it demands immediate 00:10:51.620 --> 00:10:54.000 investigation. Like when my credit card company 00:10:54.000 --> 00:10:56.320 freezes my account because I suddenly bought 00:10:56.320 --> 00:10:58.340 three laptops in a country I've never visited. 00:10:58.379 --> 00:11:00.059 I think exactly. The algorithm saw my normal 00:11:00.059 --> 00:11:02.360 pattern, saw the massive deviation, and just 00:11:02.360 --> 00:11:04.759 slammed on the brakes. That is anomaly detection. 00:11:04.960 --> 00:11:08.159 in action. Now, compare all of that to the final 00:11:08.159 --> 00:11:10.379 major task we should cover, which is clustering. 00:11:10.779 --> 00:11:13.799 Clustering. How does that differ from classification? 00:11:14.360 --> 00:11:16.559 Because they kind of sound like the exact same 00:11:16.559 --> 00:11:19.759 sorting process. They do. But in classification, 00:11:20.259 --> 00:11:22.559 remember, you gave the machine predefined labels 00:11:22.559 --> 00:11:25.980 spam or not spam. In clustering, the algorithm 00:11:25.980 --> 00:11:28.899 goes in completely blind. There are zero predefined 00:11:28.899 --> 00:11:31.759 labels. Really? So what does it do? The model 00:11:31.759 --> 00:11:34.820 just plots millions of data points across multiple 00:11:34.820 --> 00:11:38.000 dimensions and groups them entirely based on 00:11:38.000 --> 00:11:40.440 mathematical proximity. Okay. So it might analyze 00:11:40.440 --> 00:11:43.080 a city's demographics and plot out five highly 00:11:43.080 --> 00:11:46.080 distinct clusters of consumer behavior based 00:11:46.080 --> 00:11:49.779 on, say, income, commute times, and grocery habits. 00:11:50.120 --> 00:11:52.399 And sociologists might never realize those five 00:11:52.399 --> 00:11:55.000 specific groups even existed in the real world. 00:11:55.240 --> 00:11:58.169 Precisely. It discovers organic categories rather 00:11:58.169 --> 00:12:00.149 than imposing human ones. The machine just found 00:12:00.149 --> 00:12:02.929 the structural clumps naturally. Okay? So we 00:12:02.929 --> 00:12:04.970 have these incredibly powerful tools. They can 00:12:04.970 --> 00:12:08.070 associate the chips and salsa, classify our spam, 00:12:08.690 --> 00:12:11.269 catch the anomalous credit card thief, and cluster 00:12:11.269 --> 00:12:13.409 whole cities into new demographic groups. It's 00:12:13.409 --> 00:12:15.870 a very robust toolkit. But knowing how ruthless 00:12:15.870 --> 00:12:18.450 these algorithms are at finding patterns brings 00:12:18.450 --> 00:12:20.929 us back to that terrifying spelling bee example 00:12:20.929 --> 00:12:24.190 from the beginning. Ah, yes. the spiders. What 00:12:24.190 --> 00:12:26.590 happens when the model finds a pattern that is 00:12:26.590 --> 00:12:29.909 mathematically perfect but fundamentally meaningless? 00:12:30.289 --> 00:12:32.370 If we connect this to the bigger picture, you 00:12:32.370 --> 00:12:35.389 are touching on the single greatest danger in 00:12:35.389 --> 00:12:38.250 the entire field of data mining. It was a trap 00:12:38.250 --> 00:12:41.070 called overfitting. Overfitting. Let's dig into 00:12:41.070 --> 00:12:43.049 the mechanics of that because that spelling bee 00:12:43.049 --> 00:12:45.809 example from the source is just incredible. There 00:12:45.809 --> 00:12:49.070 was this statistician named Tyler Vigeon who 00:12:49.070 --> 00:12:52.899 operated a bot that literally just dredged massive 00:12:52.899 --> 00:12:55.419 public data sets looking for anything with a 00:12:55.419 --> 00:12:57.519 matching statistical curve. Just hunting for 00:12:57.519 --> 00:12:59.980 any math that lined up. Exactly. And it found 00:12:59.980 --> 00:13:02.759 an identical mathematically perfect correlation 00:13:02.759 --> 00:13:04.940 between the winning word of the national spelling 00:13:04.940 --> 00:13:07.399 bee and the number of people killed by venomous 00:13:07.399 --> 00:13:10.240 spiders. It's wild. As the letters in the winning 00:13:10.240 --> 00:13:12.840 word increased, the spider fatalities increased. 00:13:13.059 --> 00:13:16.419 The line graphs matched perfectly. And obviously, 00:13:16.940 --> 00:13:19.100 spelling complex words does not summon venomous 00:13:19.100 --> 00:13:21.929 spiders. We certainly hope not, but I mean... 00:13:21.919 --> 00:13:24.179 Doesn't this imply that data mining can essentially 00:13:24.179 --> 00:13:26.679 hallucinate scientific facts? Oh, absolutely. 00:13:26.879 --> 00:13:29.559 If you let an algorithm run wild, won't it just 00:13:29.559 --> 00:13:31.740 create totally misleading science because the 00:13:31.740 --> 00:13:34.399 math happens to align? It absolutely will, and 00:13:34.399 --> 00:13:37.120 it happens constantly in poorly designed models. 00:13:37.799 --> 00:13:40.299 To understand overfitting, think of it like a 00:13:40.299 --> 00:13:43.299 student who memorizes the exact answers to a 00:13:43.299 --> 00:13:45.899 specific practice test, rather than actually 00:13:45.899 --> 00:13:47.879 learning the underlying subject. Oh, that's a 00:13:47.879 --> 00:13:50.720 great analogy. When an algorithm analyzes a data 00:13:50.720 --> 00:13:53.129 set, it can examine the numbers so intensely 00:13:53.129 --> 00:13:56.289 that it begins to mathematically encode statistical 00:13:56.289 --> 00:14:00.250 noise. It encodes random flukes and bizarre coincidences, 00:14:00.470 --> 00:14:03.110 like the spiders, as if they are universal laws. 00:14:03.289 --> 00:14:05.509 It fits its logic way too tightly to that one 00:14:05.509 --> 00:14:08.190 specific scenario. Exactly. It overfits. The 00:14:08.190 --> 00:14:10.330 machine thinks it found a profound truth, but 00:14:10.330 --> 00:14:12.629 it merely memorized the quirks of that specific 00:14:12.629 --> 00:14:15.659 pile of dirt. Wow. If you try to apply that overfitted 00:14:15.659 --> 00:14:18.220 pattern to the real world, it possesses zero 00:14:18.220 --> 00:14:20.259 predictive value. It completely falls apart. 00:14:20.480 --> 00:14:22.679 This raises an important question, though. If 00:14:22.679 --> 00:14:25.360 these multi -million dollar machines can be entirely 00:14:25.360 --> 00:14:28.399 fooled by a random coincidence, how do we ever 00:14:28.399 --> 00:14:32.000 trust the algorithms? That is exactly why KDD 00:14:32.000 --> 00:14:34.580 and Christium do not end with the modeling phase. 00:14:35.320 --> 00:14:38.419 Phase five is evaluation, and it is arguably 00:14:38.419 --> 00:14:40.740 the most critical step of all. How does it work? 00:14:40.919 --> 00:14:43.220 It relies on a mechanism called the train -test 00:14:43.220 --> 00:14:47.620 split. You never, ever accept the pattern an 00:14:47.620 --> 00:14:50.399 algorithm finds at face value. So you test it, 00:14:50.419 --> 00:14:52.159 but how do you do that without just feeding it 00:14:52.159 --> 00:14:54.379 more noise? Well, before you even start modeling, 00:14:54.539 --> 00:14:56.899 you take your total data set and you literally 00:14:56.899 --> 00:14:59.539 chop it in half. OK. You train the algorithm 00:14:59.539 --> 00:15:02.120 on the first chunk of data. You hide the second 00:15:02.120 --> 00:15:04.120 chunk completely. Like locking the answers in 00:15:04.120 --> 00:15:07.120 a desk drawer. Exactly. Once the algorithm finishes 00:15:07.120 --> 00:15:08.960 mining the first set and says, hey, here are 00:15:08.960 --> 00:15:11.340 the hidden rules I found, you introduce it to 00:15:11.340 --> 00:15:13.519 the hidden data. It's the ultimate pop quiz. 00:15:13.659 --> 00:15:16.539 You say, let's see if your rules hold up on information 00:15:16.539 --> 00:15:19.389 you have never seen before. So if my spam filter 00:15:19.389 --> 00:15:21.929 works perfectly on the training data but suddenly 00:15:21.929 --> 00:15:24.789 starts flagging, like, all of my boss's emails 00:15:24.789 --> 00:15:28.070 as spam in the hidden test data, I know the algorithm 00:15:28.070 --> 00:15:30.750 overfitted. Yes. It memorized the practice test, 00:15:30.769 --> 00:15:34.289 but it failed the real exam. Precisely. And data 00:15:34.289 --> 00:15:37.330 scientists use complex visual tools like ROC 00:15:37.330 --> 00:15:40.850 curves to measure this. ROC curves. Yeah, ROC 00:15:40.850 --> 00:15:43.230 curve essentially graphs the tradeoff between 00:15:43.230 --> 00:15:45.870 the model catching a true pattern and the model 00:15:45.870 --> 00:15:48.519 triggering a false alarm. You can physically 00:15:48.519 --> 00:15:50.600 look at the curve and see if the model is actually 00:15:50.600 --> 00:15:53.860 smart or if it's just guessing wildly. That's 00:15:53.860 --> 00:15:55.659 fascinating. If the pattern doesn't replicate 00:15:55.659 --> 00:15:58.980 cleanly on new data, it isn't knowledge. It's 00:15:58.980 --> 00:16:01.659 just noise. Which brings us to the final phase 00:16:01.659 --> 00:16:05.259 of CrispDM, right? Phase 6 deployment. Yes. You 00:16:05.259 --> 00:16:07.559 release the validated algorithm into the real 00:16:07.559 --> 00:16:09.899 world. And when the validation is done correctly, 00:16:10.419 --> 00:16:13.039 these models are terrifyingly accurate. I mean, 00:16:13.039 --> 00:16:16.159 they predict human behavior with shocking precision. 00:16:16.360 --> 00:16:19.100 They really do. Which pivots us away from the 00:16:19.100 --> 00:16:21.580 mathematics and directly into the human cost. 00:16:21.960 --> 00:16:24.960 Because what happens when those incredibly accurate 00:16:24.960 --> 00:16:28.039 hidden patterns are about your private personal 00:16:28.039 --> 00:16:31.379 life? That's the big question. The privacy implications 00:16:31.379 --> 00:16:34.860 of data mining are profound. Primarily because 00:16:34.860 --> 00:16:36.860 of a fundamental misunderstanding most people 00:16:36.860 --> 00:16:39.320 have about anonymity. Right, because most people 00:16:39.320 --> 00:16:41.419 think, you know, I didn't put my name on that 00:16:41.419 --> 00:16:43.919 search query or that movie rating. My data is 00:16:43.919 --> 00:16:46.320 totally anonymous. That is the privacy illusion. 00:16:46.490 --> 00:16:49.309 The issue stems from a mechanism called data 00:16:49.309 --> 00:16:51.690 aggregation, which happens constantly during 00:16:51.690 --> 00:16:54.009 the data preparation phase. Let's walk through 00:16:54.009 --> 00:16:56.889 how aggregation actually breaks anonymity. Sure. 00:16:57.210 --> 00:16:59.789 Imagine you have a data set of supposedly anonymous 00:16:59.789 --> 00:17:02.169 movie ratings. Just titles and star ratings. 00:17:02.549 --> 00:17:04.849 Completely harmless. Okay, sure. Now imagine 00:17:04.849 --> 00:17:06.890 a separate data set containing anonymous zip 00:17:06.890 --> 00:17:09.410 codes, genders, and birth dates. Also seemingly 00:17:09.410 --> 00:17:12.619 harmless. Right. But when a data miner aggregates 00:17:12.619 --> 00:17:14.640 those data sets when they merge them together, 00:17:15.039 --> 00:17:17.420 the combination of those specific variables acts 00:17:17.420 --> 00:17:19.380 like a digital fingerprint. Because there might 00:17:19.380 --> 00:17:21.539 be thousands of people who liked a certain movie, 00:17:21.920 --> 00:17:24.480 but there's likely only one person born on your 00:17:24.480 --> 00:17:27.160 exact birthday, living in your exact zip code, 00:17:27.619 --> 00:17:31.319 who also rated that specific obscure 1980s film 00:17:31.319 --> 00:17:34.500 five stars. Exactly. The intersection of generalized 00:17:34.500 --> 00:17:37.039 data creates hyper -specific identification. 00:17:37.259 --> 00:17:39.779 Here's where it gets really interesting and deeply 00:17:39.779 --> 00:17:42.400 unsettling. We have historical examples from 00:17:42.400 --> 00:17:44.619 the source material where this exact mechanism 00:17:44.619 --> 00:17:47.960 caused catastrophic breaches. Take the infamous 00:17:47.960 --> 00:17:52.119 AOL search history incident. Oh, a classic, devastating 00:17:52.119 --> 00:17:55.480 case study in the failure of anonymization. AOL 00:17:55.480 --> 00:17:57.380 genuinely thought they were doing the academic 00:17:57.380 --> 00:18:00.539 community a huge favor. They took a massive data 00:18:00.539 --> 00:18:02.640 set of user search histories, stripped off all 00:18:02.640 --> 00:18:04.500 the usernames, replaced them with random numbers, 00:18:04.740 --> 00:18:06.799 and released the data for researchers to mine. 00:18:06.859 --> 00:18:09.319 Which sounds safe in theory. Right. But journalists 00:18:09.319 --> 00:18:11.660 simply looked at the aggregated patterns of what 00:18:11.660 --> 00:18:14.519 specific numbered users were searching for. If, 00:18:14.519 --> 00:18:18.680 say, user 4417749 searches for a specific local 00:18:18.680 --> 00:18:21.259 high school, a specific rare medical condition, 00:18:21.519 --> 00:18:24.420 and a specific local landscaping business. You 00:18:24.420 --> 00:18:26.480 no longer need a username, the pattern is the 00:18:26.480 --> 00:18:30.079 name. Journalists literally physically tracked 00:18:30.079 --> 00:18:32.359 down the actual individuals sitting in their 00:18:32.359 --> 00:18:34.619 homes just based on their search queries. It's 00:18:34.619 --> 00:18:37.279 incredible. And it isn't just accidental academic 00:18:37.279 --> 00:18:40.559 releases. The historical record includes massive 00:18:40.559 --> 00:18:44.119 legal battles like the 2011 lawsuit against Walgreens. 00:18:44.519 --> 00:18:47.339 Customers sued the pharmacy chain because Walgreens 00:18:47.339 --> 00:18:49.579 was allegedly taking their private prescription 00:18:49.579 --> 00:18:53.279 information, anonymizing it, and selling it to 00:18:53.279 --> 00:18:56.000 data mining companies. And those companies then 00:18:56.000 --> 00:18:58.730 mined the aggregated data for prescribing patterns 00:18:58.730 --> 00:19:02.269 and sold those highly lucrative insights to big 00:19:02.269 --> 00:19:04.329 pharmaceutical manufacturers. Which just feels 00:19:04.329 --> 00:19:06.950 like a profound violation of the social contract. 00:19:07.009 --> 00:19:09.869 And it begs a massive question. Who is regulating 00:19:09.869 --> 00:19:12.269 this? If the algorithms can see right through 00:19:12.269 --> 00:19:15.130 the illusion of anonymity, who legally owns the 00:19:15.130 --> 00:19:17.170 hidden patterns of our lives? The answer to that 00:19:17.170 --> 00:19:19.630 depends entirely on your GPS coordinates. Really? 00:19:19.910 --> 00:19:22.160 Yeah. The international legal landscape regarding 00:19:22.160 --> 00:19:25.339 data mining is incredibly fractured. It relies 00:19:25.339 --> 00:19:27.339 on fundamentally opposing philosophies depending 00:19:27.339 --> 00:19:29.000 on where you are. Let's start with Europe, then. 00:19:29.200 --> 00:19:31.440 Well, Europe takes a highly protective stance. 00:19:32.079 --> 00:19:34.380 The European Union recognizes something called 00:19:34.380 --> 00:19:37.559 a database right. So the actual collection of 00:19:37.559 --> 00:19:40.279 the data is protected as intellectual property. 00:19:40.440 --> 00:19:42.799 Yes, it's based on the sweat of the brow doctrine. 00:19:43.440 --> 00:19:46.240 Compiling the database takes effort and investment. 00:19:46.759 --> 00:19:49.019 Therefore, the database itself is protected. 00:19:49.140 --> 00:19:52.789 OK. Mining an in copyright data set in the EU. 00:19:53.130 --> 00:19:56.349 generally requires explicit negotiated permission 00:19:56.349 --> 00:19:59.170 from the owner. Now, they did pass a directive 00:19:59.170 --> 00:20:02.470 in 2019 that created specific exceptions, for 00:20:02.470 --> 00:20:04.970 instance, allowing text and data mining specifically 00:20:04.970 --> 00:20:07.730 for scientific research without permission. But 00:20:07.730 --> 00:20:10.009 commercially, the protection of the data is incredibly 00:20:10.009 --> 00:20:11.910 robust. And what about the UK? Did they adopt 00:20:11.910 --> 00:20:14.349 that same model? The UK took a slightly nuanced 00:20:14.349 --> 00:20:17.710 path. In 2014, following an extensive government 00:20:17.710 --> 00:20:20.250 review of intellectual property, they amended 00:20:20.250 --> 00:20:22.730 their copyright law. To do what? They actually 00:20:22.730 --> 00:20:24.410 became the second country in the world right 00:20:24.410 --> 00:20:27.230 after Japan to create a specific legal limitation 00:20:27.230 --> 00:20:30.009 and exception for content mining. Oh, wow. But 00:20:30.009 --> 00:20:32.710 there's a massive caveat. The UK exception only 00:20:32.710 --> 00:20:34.609 allows for non -commercial purposes. Got it. 00:20:34.849 --> 00:20:38.190 You can mine data for academic research, but 00:20:38.190 --> 00:20:40.849 you cannot legally mine a protected data set 00:20:40.849 --> 00:20:43.750 just to build a commercial product or make a 00:20:43.750 --> 00:20:46.410 quick buck. OK, so Europe operates on strict 00:20:46.410 --> 00:20:49.640 permission and non -commercial exceptions. Let's 00:20:49.640 --> 00:20:52.339 look at the United States, which operates on 00:20:52.339 --> 00:20:55.839 a totally different wavelength. The US approach 00:20:55.839 --> 00:20:57.940 is vastly different because it relies on the 00:20:57.940 --> 00:21:00.539 legal doctrine of fair use. Fair use. Right. 00:21:01.019 --> 00:21:03.779 In the US, data mining is largely upheld by the 00:21:03.779 --> 00:21:06.180 courts because the act of mining is considered 00:21:06.180 --> 00:21:08.680 transformative. Transformative. Let's define 00:21:08.680 --> 00:21:11.660 that legally. It means the algorithm isn't just 00:21:11.660 --> 00:21:13.700 copying the original expression of the data, 00:21:13.880 --> 00:21:16.579 it's creating entirely new knowledge out of it. 00:21:16.799 --> 00:21:19.390 Exactly. You aren't republishing the original 00:21:19.390 --> 00:21:21.690 data. You are publishing the mathematical patterns 00:21:21.690 --> 00:21:24.930 you found inside it. Therefore, it doesn't supplant 00:21:24.930 --> 00:21:26.809 the market for the original work. That makes 00:21:26.809 --> 00:21:29.450 sense. The clearest legal precedent is the famous 00:21:29.450 --> 00:21:32.650 Google Books settlement. Google systematically 00:21:32.650 --> 00:21:35.690 digitized millions of in -copyright books without 00:21:35.690 --> 00:21:38.450 permission. I remember that. And the presiding 00:21:38.450 --> 00:21:41.369 judge ruled it was fundamentally lawful, largely 00:21:41.369 --> 00:21:44.309 because one of the primary transformative uses 00:21:44.309 --> 00:21:47.990 was allowing text and data mining. across the 00:21:47.990 --> 00:21:50.710 corpus of literature. So U .S. copyright law 00:21:50.710 --> 00:21:52.950 essentially encourages data mining. But what 00:21:52.950 --> 00:21:55.849 about privacy laws? We hear about HEPA every 00:21:55.849 --> 00:21:58.170 time we go to the doctor's office. Right. Surely 00:21:58.170 --> 00:22:00.369 that stops companies from mining our personal 00:22:00.369 --> 00:22:04.089 details. Laws like HEPA for health care or FERPA 00:22:04.089 --> 00:22:07.069 for educational records do provide strict protections. 00:22:07.309 --> 00:22:09.210 They require informed consent for how your data 00:22:09.210 --> 00:22:12.250 is utilized. Okay. But here is the critical, 00:22:12.549 --> 00:22:15.289 often misunderstood reality of the American legal 00:22:15.289 --> 00:22:18.690 system. Those laws are strictly sector -specific. 00:22:18.849 --> 00:22:20.990 Wait, meaning they only apply to the doctor or 00:22:20.990 --> 00:22:23.930 the school? Yes. If a retail app tracks your 00:22:23.930 --> 00:22:26.190 location or a website logs your browsing habits, 00:22:26.650 --> 00:22:29.230 Hypea does not apply. Oh, wow. The use of data 00:22:29.230 --> 00:22:31.349 mining by the vast majority of commercial businesses 00:22:31.349 --> 00:22:33.789 in the U .S. is not controlled by any overarching 00:22:33.789 --> 00:22:36.829 federal privacy legislation. So outside of very 00:22:36.829 --> 00:22:39.490 specific sectors, it's essentially the Wild West. 00:22:39.670 --> 00:22:41.849 It's an invisible economy operating with very 00:22:41.849 --> 00:22:44.569 few speed limits. So what does this all mean? 00:22:44.829 --> 00:22:46.950 Let's zoom out and look at the journey we've 00:22:46.950 --> 00:22:50.190 just taken. We started with an annoying corporate 00:22:50.190 --> 00:22:52.990 buzzword that actually began its life as an academic 00:22:52.990 --> 00:22:55.990 insult data fishing, a practice that was deeply 00:22:55.990 --> 00:22:57.950 frowned upon because it lacked a hypothesis. 00:22:58.450 --> 00:23:01.069 But as the digital world exploded, the philosophy 00:23:01.069 --> 00:23:04.420 flipped. we realized we needed the algorithms 00:23:04.420 --> 00:23:07.380 to find the hypotheses for us. We established 00:23:07.380 --> 00:23:11.299 the rigorous, six -phase, crisp DM process to 00:23:11.299 --> 00:23:13.880 clean the dirt, run the math, and most importantly, 00:23:14.019 --> 00:23:17.279 validate the results to avoid the trap of overfitting 00:23:17.279 --> 00:23:19.779 and venomous spiders. And we ended up looking 00:23:19.779 --> 00:23:22.579 at the very real legal reality of your digital 00:23:22.579 --> 00:23:25.539 life. Because of data aggregation, your anonymity 00:23:25.539 --> 00:23:28.039 is largely an illusion. And depending on where 00:23:28.039 --> 00:23:30.140 you live, the patterns of your behavior might 00:23:30.140 --> 00:23:32.759 be legally extracted. transformed and monetized 00:23:32.759 --> 00:23:35.200 without you ever knowing it. Every single tap, 00:23:35.660 --> 00:23:37.880 swipe, and purchase you make today is quite literally 00:23:37.880 --> 00:23:40.859 a piece of raw material being fed into an electromagnet. 00:23:40.940 --> 00:23:42.859 Sifting for the iron shavings of your habits. 00:23:43.480 --> 00:23:46.339 Exactly. And as we close out today's deep dive, 00:23:46.579 --> 00:23:49.799 I want to leave you with one final slightly provocative 00:23:49.799 --> 00:23:52.799 thought to mull over. Manual pattern extraction 00:23:52.799 --> 00:23:56.400 is not a new concept. No, not at all. Human beings 00:23:56.400 --> 00:23:59.420 have been doing it for centuries. The mathematician 00:23:59.420 --> 00:24:02.420 Thomas Bayes was extracting probability patterns 00:24:02.420 --> 00:24:05.900 in the 1700s. Regression analysis was being used 00:24:05.900 --> 00:24:08.900 to find relationships and data in the 1800s. 00:24:09.220 --> 00:24:11.759 We have always been pattern seekers. It's fundamental 00:24:11.759 --> 00:24:15.569 to human progress. It is. But historically, humans 00:24:15.569 --> 00:24:18.490 drove the process. We did the math. We understood 00:24:18.490 --> 00:24:20.769 the exact mechanism of how we got from point 00:24:20.769 --> 00:24:23.910 A to point B. We understood the why behind the 00:24:23.910 --> 00:24:26.359 connection. Because we built it. Exactly. But 00:24:26.359 --> 00:24:28.859 as these data sets grow infinitely larger and 00:24:28.859 --> 00:24:31.299 complex artificial neural networks completely 00:24:31.299 --> 00:24:33.660 take over the mining process, we are entering 00:24:33.660 --> 00:24:36.079 an era of the black box. Which is a very different 00:24:36.079 --> 00:24:38.440 paradigm. It really is. We have to wonder, will 00:24:38.440 --> 00:24:40.720 we soon reach a point where the patterns these 00:24:40.720 --> 00:24:43.380 algorithms discover are so incredibly complex 00:24:43.380 --> 00:24:46.000 and so deeply layered across thousands of dimensions 00:24:46.000 --> 00:24:49.000 that the human brain simply can no longer comprehend 00:24:49.000 --> 00:24:51.640 the math behind them? That's a very real possibility. 00:24:51.920 --> 00:24:54.789 Right. If the machine gives us the perfect validated 00:24:54.789 --> 00:24:56.990 answer, but the underlying logic is too vast 00:24:56.990 --> 00:24:59.910 for any human to understand, are we effectively 00:24:59.910 --> 00:25:02.089 locking ourselves out of our own knowledge discovery 00:25:02.089 --> 00:25:04.509 process? It's something to think about the next 00:25:04.509 --> 00:25:06.569 time your phone's algorithm auto -predicts exactly 00:25:06.569 --> 00:25:07.809 what you are about to type.