WEBVTT 00:00:00.000 --> 00:00:02.960 Welcome to the deep dive. You hear the word data 00:00:02.960 --> 00:00:05.320 everywhere these days, don't you? Oh, constantly. 00:00:05.440 --> 00:00:07.860 Especially in fields like education. It's kind 00:00:07.860 --> 00:00:10.660 of sold as this key to making smart, objective 00:00:10.660 --> 00:00:13.820 decisions. Sounds great on paper. It really does. 00:00:14.160 --> 00:00:16.600 But that's where the core issue that our source 00:00:16.600 --> 00:00:21.280 material points out really lies. Data gets thrown 00:00:21.280 --> 00:00:25.179 around a lot, but often pretty imprecisely, especially 00:00:25.179 --> 00:00:28.350 in education settings. Imprecisely how? Well, 00:00:28.489 --> 00:00:30.829 there's often a really big gap, a massive difference, 00:00:30.989 --> 00:00:33.250 actually, between what gets called data -driven 00:00:33.250 --> 00:00:36.250 and what, you know, researchers actually do when 00:00:36.250 --> 00:00:38.450 they handle data and evidence. Okay, that's interesting. 00:00:38.490 --> 00:00:40.450 So that's our mission for this deep dive, then, 00:00:40.530 --> 00:00:42.609 to unpack that difference, to figure out what 00:00:42.609 --> 00:00:45.250 really counts as solid evidence for making better 00:00:45.250 --> 00:00:47.570 decisions, and kind of cut through the buzzwords 00:00:47.570 --> 00:00:50.369 to see what's genuinely useful. That's the plan. 00:00:50.429 --> 00:00:52.270 Let's get into it. All right. So let's start 00:00:52.270 --> 00:00:54.549 with that common idea, the data -driven educator. 00:00:54.770 --> 00:00:56.890 Sounds good, right? Using research objectively. 00:00:57.259 --> 00:00:59.280 But you're saying the source suggests there's 00:00:59.280 --> 00:01:01.579 a disconnect in practice. What's that look like? 00:01:01.920 --> 00:01:03.520 Yeah, the source describes it pretty clearly. 00:01:04.040 --> 00:01:07.099 These data -driven educators often lean towards 00:01:07.099 --> 00:01:11.680 using data that's just, well, conveniently available. 00:01:12.060 --> 00:01:14.420 Conveniently available? Like what? Almost always. 00:01:14.719 --> 00:01:17.799 It means scores. Standardized test scores, maybe 00:01:17.799 --> 00:01:20.640 those standards -based tests, even the diagnostic 00:01:20.640 --> 00:01:22.799 ones teachers use in the classroom. Whatever's 00:01:22.799 --> 00:01:24.959 easy to get your hands on. OK, I get the appeal. 00:01:25.040 --> 00:01:27.359 It's right there. It's efficient, maybe. But 00:01:27.359 --> 00:01:29.420 what's the catch? What's the problem with just 00:01:29.420 --> 00:01:31.760 using what's easy? Well, here's a big one the 00:01:31.760 --> 00:01:34.299 source highlights, the validity and reliability 00:01:34.299 --> 00:01:36.739 of these tests. They're hardly ever questioned. 00:01:38.359 --> 00:01:40.859 So are they actually measuring what they say 00:01:40.859 --> 00:01:43.459 they're measuring? And are the results consistent, 00:01:43.500 --> 00:01:46.140 that kind of thing? Precisely. Those are fundamental 00:01:46.140 --> 00:01:49.120 questions. Validity, does this test really measure, 00:01:49.120 --> 00:01:51.700 say, critical thinking or just memorization? 00:01:52.299 --> 00:01:54.159 Reliability, if the same student took a similar 00:01:54.159 --> 00:01:56.519 test tomorrow, would the score be roughly the 00:01:56.519 --> 00:01:59.920 same? If you don't ask that, well, your foundation 00:01:59.920 --> 00:02:01.480 is shaken. Makes sense. And another thing, they 00:02:01.480 --> 00:02:03.939 tend to look for interesting and telling trends 00:02:03.939 --> 00:02:08.360 in the scores, spotting patterns. But they rarely 00:02:08.360 --> 00:02:11.020 set out to answer specific questions with that 00:02:11.020 --> 00:02:14.280 data. It's more exploratory, less targeted investigation. 00:02:14.500 --> 00:02:16.879 So pattern spotting, not question answering. 00:02:17.000 --> 00:02:19.939 Right. And critically, they rarely use theory 00:02:19.939 --> 00:02:21.939 to interpret what they see. What do you mean, 00:02:22.020 --> 00:02:24.680 use theory? Like understanding why scores might 00:02:24.680 --> 00:02:28.360 be low. Is it the teaching? Is it student background? 00:02:28.840 --> 00:02:32.479 Curriculum mismatch? Motivation? Theory helps 00:02:32.479 --> 00:02:35.080 frame those potential explanations. Instead, 00:02:35.460 --> 00:02:38.580 the assumption is often just, well, the instruction 00:02:38.580 --> 00:02:40.919 directly caused the scores, and that explains 00:02:40.919 --> 00:02:44.819 all the trends. Full stop. So it sounds, I don't 00:02:44.819 --> 00:02:46.939 know, less like a real investigation and more 00:02:46.939 --> 00:02:48.659 like just looking at the scoreboard after the 00:02:48.659 --> 00:02:50.780 game. That's a good way to put it. More observing 00:02:50.780 --> 00:02:53.159 what happened than really digging into the why 00:02:53.159 --> 00:02:55.659 or what does this actually mean for us. Like 00:02:55.659 --> 00:02:58.699 you said, shaky foundations. So if that's the 00:02:58.699 --> 00:03:00.740 common way, how should we be approaching this? 00:03:00.819 --> 00:03:03.039 How do researchers do it differently? Ah, well 00:03:03.039 --> 00:03:05.139 this is where it gets really interesting. Researchers 00:03:05.139 --> 00:03:08.580 flip that whole process. They start by defining 00:03:08.580 --> 00:03:10.879 the questions they seek to answer. Before they 00:03:10.879 --> 00:03:13.939 even look at data. Absolutely. And they define 00:03:13.939 --> 00:03:17.039 the data methods they will use before gathering 00:03:17.039 --> 00:03:19.319 anything. It's all planned up front. Think of 00:03:19.319 --> 00:03:21.569 it like... Well, you wouldn't build a house by 00:03:21.569 --> 00:03:23.629 just grabbing random lumber, right? No, you'd 00:03:23.629 --> 00:03:26.750 need a blueprint. Exactly. The research question 00:03:26.750 --> 00:03:29.210 and the methods are the blueprint. You know what 00:03:29.210 --> 00:03:31.009 you need to measure and how you're going to measure 00:03:31.009 --> 00:03:34.009 it to answer your specific question. OK, that's 00:03:34.009 --> 00:03:36.849 a massive shift already. Starting with the question, 00:03:37.069 --> 00:03:39.909 not the readily available numbers forces a different 00:03:39.909 --> 00:03:42.469 kind of thinking. It really does. And because 00:03:42.469 --> 00:03:44.889 they have that plan, they gather only the data 00:03:44.889 --> 00:03:48.009 they need. No fishing expeditions for random 00:03:48.009 --> 00:03:50.889 trends. It's focused. More efficient, I guess. 00:03:51.229 --> 00:03:53.270 Definitely. And here's another key difference. 00:03:54.250 --> 00:03:57.330 All data is interpreted in light of theory. Back 00:03:57.330 --> 00:03:59.629 to theory again. So they use existing knowledge 00:03:59.629 --> 00:04:02.610 to make sense of the numbers. Yes. Theory provides 00:04:02.610 --> 00:04:05.759 the framework, the potential explanations. So 00:04:05.759 --> 00:04:08.020 if they see a result, they can connect it to 00:04:08.020 --> 00:04:10.539 established ideas about learning or behavior 00:04:10.539 --> 00:04:13.199 or whatever the topic is. It prevents jumping 00:04:13.199 --> 00:04:16.399 to simplistic conclusions. Plus, researchers 00:04:16.399 --> 00:04:19.040 are constantly challenging themselves and their 00:04:19.040 --> 00:04:22.779 peers to justify all assumptions. Why did you 00:04:22.779 --> 00:04:24.920 choose this measure? Why do you think this factor 00:04:24.920 --> 00:04:26.699 is important? Holding themselves accountable. 00:04:26.939 --> 00:04:29.360 And a big part of that is they have to demonstrate 00:04:29.360 --> 00:04:31.800 the validity and reliability of instruments. 00:04:32.189 --> 00:04:35.649 the tests, the surveys, whatever generates the 00:04:35.649 --> 00:04:38.629 data. Proof is required. Which connects back 00:04:38.629 --> 00:04:41.290 to that earlier point about just accepting convenient 00:04:41.290 --> 00:04:43.350 test scores. Researchers have to prove their 00:04:43.350 --> 00:04:46.069 tools are sound. They do. And the source really 00:04:46.069 --> 00:04:49.110 hammers this home. Researchers challenge themselves 00:04:49.110 --> 00:04:51.569 and peers to demonstrate the quality of their 00:04:51.569 --> 00:04:54.569 data and conclusions. And here's the kicker. 00:04:54.990 --> 00:04:58.029 Conclusions based on invalid or badly collected 00:04:58.029 --> 00:05:01.610 data or unethically collected data must be discarded. 00:05:01.759 --> 00:05:04.439 Wow. Okay, so there's a really strong ethical 00:05:04.439 --> 00:05:06.379 and quality standard there. You can't just use 00:05:06.379 --> 00:05:09.060 bad data. Nope. That rigor, that discipline, 00:05:09.220 --> 00:05:11.480 that critical thinking, that's what elevates 00:05:11.480 --> 00:05:14.680 simple information or data into trustworthy evidence. 00:05:14.819 --> 00:05:16.639 That makes the distinction much clearer. It's 00:05:16.639 --> 00:05:18.800 not just semantics. It's a fundamentally different 00:05:18.800 --> 00:05:21.040 process. World of difference. And you mentioned 00:05:21.040 --> 00:05:23.600 the source connects this research approach to 00:05:23.600 --> 00:05:25.980 real benefits, even for folks like IT managers. 00:05:26.579 --> 00:05:29.399 So for someone listening, maybe not in education 00:05:29.399 --> 00:05:33.060 or research, what's the payoff for thinking more 00:05:33.060 --> 00:05:36.620 like a researcher about data? The benefits are 00:05:36.620 --> 00:05:40.680 very practical. First off, more efficient processes. 00:05:40.779 --> 00:05:43.329 That's so. Because, as we said, Planners use 00:05:43.329 --> 00:05:46.110 theory to focus efforts on relevant factors and 00:05:46.110 --> 00:05:49.050 only relevant factors. You're not wasting time 00:05:49.050 --> 00:05:51.230 and resources collecting or analyzing stuff that 00:05:51.230 --> 00:05:53.670 doesn't actually matter for your question. You 00:05:53.670 --> 00:05:56.509 zero in. Cuts out the noise, okay. Leads to more 00:05:56.509 --> 00:05:58.449 effective decisions. Quite simply because you're 00:05:58.449 --> 00:06:01.250 using multiple reliable and valid data sources, 00:06:01.750 --> 00:06:04.089 you're building a stronger case, a more complete 00:06:04.089 --> 00:06:06.810 picture, not just relying on one potentially 00:06:06.810 --> 00:06:09.589 flawed number. Right, triangulation, kind of, 00:06:09.589 --> 00:06:12.329 using different angles. Exactly. which then leads 00:06:12.329 --> 00:06:14.910 to more effective interventions because your 00:06:14.910 --> 00:06:17.610 actions are focused on locally important factors, 00:06:17.990 --> 00:06:20.050 what actually matters here in this situation, 00:06:20.050 --> 00:06:22.329 and you have a clear rationale for why you're 00:06:22.329 --> 00:06:24.529 doing what you're doing. So it's targeted, not 00:06:24.529 --> 00:06:27.589 just a shot in the dark. Precisely. And finally, 00:06:28.149 --> 00:06:30.810 it means your assessments and evaluations of 00:06:30.810 --> 00:06:33.709 whatever you did become more accurate and more 00:06:33.709 --> 00:06:36.470 informative. The evidence is clear, it's understood, 00:06:36.670 --> 00:06:38.629 so you can actually learn from it and improve 00:06:38.629 --> 00:06:41.240 next time. That's the real learning lube, isn't 00:06:41.240 --> 00:06:43.439 it? That clarity makes continuous improvement 00:06:43.439 --> 00:06:45.459 possible instead of just guessing if something 00:06:45.459 --> 00:06:48.680 worked. You got it. That feels like the aha moment. 00:06:49.040 --> 00:06:52.259 This isn't just ivory tower stuff. This disciplined 00:06:52.259 --> 00:06:55.720 research like approach has direct consequences 00:06:55.720 --> 00:06:59.060 for effectiveness, for efficiency, for actually 00:06:59.060 --> 00:07:01.660 getting things right in the real world. Absolutely. 00:07:02.019 --> 00:07:04.160 It's about moving from just collecting numbers 00:07:04.160 --> 00:07:07.420 to generating real insight. OK. And just to add 00:07:07.420 --> 00:07:09.839 a bit more context, the source briefly touches 00:07:09.839 --> 00:07:12.180 on two main types of research that kind of feed 00:07:12.180 --> 00:07:14.079 into this whole system of building knowledge, 00:07:14.160 --> 00:07:16.850 right? Yeah. It distinguishes between pure research 00:07:16.850 --> 00:07:20.689 and applied research. So pure research. What's 00:07:20.689 --> 00:07:23.110 the goal there? The main goal of pure research 00:07:23.110 --> 00:07:26.910 is to generate and test theory. It's about understanding 00:07:26.910 --> 00:07:29.790 fundamental principles how things work. Think 00:07:29.790 --> 00:07:32.949 discovering gravity or basic principles of learning. 00:07:33.329 --> 00:07:36.149 It often looks for cause and effect relationships, 00:07:36.550 --> 00:07:38.949 maybe using tightly controlled experiments and 00:07:38.949 --> 00:07:41.589 often quantitative data numbers. Foundational 00:07:41.589 --> 00:07:44.670 knowledge building. Got it. And applied research. 00:07:44.860 --> 00:07:47.339 Applied research, sometimes called technology 00:07:47.339 --> 00:07:49.680 development, takes those fundamental discoveries 00:07:49.680 --> 00:07:52.699 from pure research and aims to develop useful 00:07:52.699 --> 00:07:55.560 technologies or practical tools. Ah, so building 00:07:55.560 --> 00:07:57.879 the airplane based on the principles of aerodynamics 00:07:57.879 --> 00:08:00.199 discovered by pure research. That's a perfect 00:08:00.199 --> 00:08:02.720 analogy. People doing applied research want to 00:08:02.720 --> 00:08:05.259 create efficient and effective tools that solve 00:08:05.259 --> 00:08:08.139 real -world problems, leveraging that basic science. 00:08:08.279 --> 00:08:10.699 Okay, so one builds the deep understanding, the 00:08:10.699 --> 00:08:13.339 other builds the practical application. both 00:08:13.339 --> 00:08:15.879 contribute to the evidence base we rely on. Exactly. 00:08:16.000 --> 00:08:18.199 They work together. Understanding both helps 00:08:18.199 --> 00:08:20.100 you see where knowledge comes from, whether it's 00:08:20.100 --> 00:08:21.980 a fundamental principle or a tested solution. 00:08:22.240 --> 00:08:25.240 Right. So when we hear data -driven, we can start 00:08:25.240 --> 00:08:28.639 thinking, is this based on just observing a number? 00:08:29.139 --> 00:08:31.540 Or is it connected to some deeper understanding, 00:08:31.620 --> 00:08:34.419 maybe some pure research, and developed thoughtfully, 00:08:34.480 --> 00:08:37.279 like applied research? It adds layers to how 00:08:37.279 --> 00:08:39.919 we evaluate claims. It gives you more tools for 00:08:39.919 --> 00:08:42.059 critical thinking about the information you receive. 00:08:42.840 --> 00:08:45.080 Wrapping this deep dive up, it really feels like 00:08:45.080 --> 00:08:48.080 the main takeaway is this huge difference between 00:08:48.080 --> 00:08:52.159 just grabbing data and carefully, rigorously 00:08:52.159 --> 00:08:54.220 building evidence. It's the heart of it. The 00:08:54.220 --> 00:08:56.620 real value isn't just having information, it's 00:08:56.620 --> 00:08:58.740 the whole process, how you gather it, how you 00:08:58.740 --> 00:09:01.120 check it, how you interpret it, the theory you 00:09:01.120 --> 00:09:03.960 bring to it. Yeah, and hopefully this discussion 00:09:03.960 --> 00:09:07.200 gives you, the listener, a framework, a way to 00:09:07.200 --> 00:09:09.240 start asking tougher questions when someone presents 00:09:09.240 --> 00:09:12.190 you with data to push for that rigor. Definitely. 00:09:12.370 --> 00:09:14.950 It encourages that critical lens. So the final 00:09:14.950 --> 00:09:17.129 thought to leave you with. Next time you come 00:09:17.129 --> 00:09:19.289 across a claim that's supposedly data -driven, 00:09:19.710 --> 00:09:22.610 what specific things will you ask now? What questions 00:09:22.610 --> 00:09:24.769 will help you figure out if it's genuinely backed 00:09:24.769 --> 00:09:27.049 by solid evidence or if it's maybe just, well, 00:09:27.629 --> 00:09:28.730 a conveniently collected number?