Understanding the Limitations of Data and Research in Education This is an excerpt from a book I’m working on titled, Understanding the Science of Reading: Context Matters, published by Guilford. NAEP data collected by the US Department of Education shows that there is no massive reading crisis. There was a Covid dip starting in 2020, but scores have remained stable for the past two decades and have risen slight from 1971 It Was the Asteroids. NAEP scores are not research, they are data. They give us a sense of things; however, we cannot use them to establish causality. We can’t say for absolute certainty that the Covid crisis was the cause of the dip in reading scores from 2019 to 2022. There is a correlation or co-relation here, but correlation does not infer causation. While a causal relationship seems likely, we cannot be absolutely certain that one thing caused the other thing to occur. It could be that the dip in scores was caused by another coinciding incident. Perhaps it was the two asteroids that flew close to earth in 2019 (2019 OK and 2019 LH5). After all, reading scores were just fine until these two asteroids flew by. Or maybe it was Hurricane Dorian. Or maybe it was a South Korean boy band. Or maybe it was … You can see where this leads us. Assigning causality to things by mere association is not science. It’s pseudoscience and it leads to all sorts of problems. Yes, there’s a natural human tendency to attribute causality to incidents that occur together close in time. When a bad thing happens, we instinctively look back for something else that happened and assign blame to it. We do this to prevent future bad things from happening. On a personal level, it’s natural to ask, “Why is this happening to me?” Or, “What have I done to deserve this?” It’s an instinctive reaction done to make sense of things and to try to control the world around us. When the ancients forgot to offer up a prayer or sacrifice before an unsuccessful hunt, it was logical to assume that one thing caused the other to occur. And when plagues and sickness occurred, any rational person would determine it was caused by the unusual conjunction of planets. And when the cows died after a witch gave them the stink eye, it was just good science to burn the witch. Everybody knows that. Burning witches was an evidence-based practice. But the problem with randomly assigning causality in a universe that has quite a few coinciding incidents going on all the time is that you’re going to be wrong many more times than you’re going to be right. As well, there may be a few witches burned along the way. Oh well. Real science does not randomly assign causes to things. Yet, we see this smorgasbord of causality used to explain all sorts of problems in education. It’s the breakdown of the family unit. It’s the lack of prayer in public schools. It’s an increase in violent video games. It’s inappropriate song lyrics. It’s critical race theory. It’s DEI practices. It’s ideological approaches. It’s federal involvement in education. It’s teachers unions. For problems related to reading, SoR advocates claim that it’s whole language. It’s the three cueing systems. It’s the lack of phonics instruction. It’s balanced literacy instruction. Teachers aren’t getting evidence-based training. Teacher preparation programs are not doing their job. Teacher educators are not teaching the right things. Schools aren’t screening for dyslexia. And so on, and so on, and so on … But what’s truly unique about the SoR movement is that causality has been assigned to something that hasn’t actually happened (that is, a massive drop in reading scores). In other words, the SoR has assigned causality to an effect that never really occurred. Imagine that. But even if there were a massive reading crisis (which there isn’t), you can’t simply run around assigning causes to things based on preconceived ideas, anecdotal evidence, or personal experiences. That’s not how real science works. Stovepiping In 2003 a new word came into my lexicon, ‘stovepiping’. That’s a name for the presentation of data without proper context or analysis (Tetlock & Mellers, 2011). In 2003 such data (raw data) were collected by intelligence agencies (cite). But before it could be properly evaluated and analyzed, data were sent up the “stovepipe” to Vice President Cheney. In the accumulated pile of decontextualized data, confirmation bias was used to find the data he wanted to find. Based on “the data” a case was made for the invasion of Iraq by the US military. Now it was factually accurate to say that the Vice President had data showing that Saddam Husssein had weapons of mass destruction. They even sent General Collin Powel to the United Nations to present this data to the world. In terms of performance and style, this was a wonderful presentation with pictures and charts and other visual aids including a vial of white powder. However, it turns out this raw data were not very accurate or complete. It was pulled out of context and not properly analyzed. Oops. That’s an extreme example of what can happen when data is presented out of context. I only mention this because it’s similar to what has been done to suggest there is a reading crisis (see Chapter 2). Emily Hanford and other journalists continue to use confirmation bias, looking for bits of decontextualized data to create the illusion of a reading crisis. As stated above, there simply is no irrefutable evidence that there is a national crisis in reading (Reinking, Hruby, & Risk, 2023; Thomas, 2024). And yet, here we are. Research Context matters. Research must always be understood in the context of other research studies. As an example, Stephen Kucer (2016) conducted a very interesting study where he had fourth-grade students read complex text aloud followed by a retelling. The number and type of miscues were noted as well as students’ comprehension. He found that students who read with meaning-maintaining miscues were more likely to comprehend information than students who read with no miscues (100% word reading accuracy). This study helps us to further understand the reading process. It doesn’t use controlled experimental design to determine the cause of things. However, we can infer that word reading accuracy may be over-rated and perhaps if students attend overly much to the surface structure of the sentence, they may miss some of the deep structure or meaning when reading complex texts. It doesn’t prove this, it suggests it. But to get a better understanding, this study needs put in the context of Kucer’s other research, (Kucer), as well other eye movement and miscue analysis research conducted by others (cite), as well as other research conducted looking at word reading accuracy and comprehension. Research must always be understood in the context of other research. One study may support a theory, but by itself, it doesn’t prove anything. And by the way, in the social sciences there is not single piece of research the proves anything conclusively once and for all. When dealing with human beings, there are simply far too many variables. Sighting it In The best analogy I can find related to understanding the necessary context of educational research is related to deer hunting: I grew up in a deer hunting culture. Every fall we would sight in our rifles to see if the gun was aligned with the scope. It’s very common for the scope to get bumped during the year. So, you’d go out to the range and shoot at a target 100 yards down range to see where your rifle was shooting and to determine if you needed to adjust the scope. But you didn’t take just one shot. One shot could have been an outlier. You could have been shaking, your platform could have wiggled, or you could have flinched when you pulled the trigger. Instead, you took several shots to see where the grouping was and adjusted your scope accordingly (see Figure 1.4). The more shots you took the more confident you could be in the alignment of your scope. So it is when looking at reading research. You can’t look at a single study to determine anything conclusively. Instead, you look at a variety of research studies, from a variety of fields, using a variety of methodologies to get a sense of things. And the more research you look at, the more confident you can be that you have an accurate sense of things. Proven Results Another thing about research is that it doesn’t prove anything, at least not in the social sciences. There is no single research that conclusively proves anything once and for all about reading instruction. Research may support a hypothesis. It may provide evidence for something, show something, indicate something, or demonstrate something, but in the social sciences, research doesn’t prove things. The results may indicate, implicate, or illustrate, but educational research doesn’t prove things. SoR advocates often claim that there is a “proven science” of reading. But when working with variable human beings interacting in variables social situations there are simply too many variables to say that something proves something else conclusively. Instead, research provides evidence for things. A lot of research provides strong evidence. A little research provides weak evidence. There are evidenced-based practices, but there is no “proven science” of reading. But even saying something is evidence-based says nothing about the quality of the evidence or the validity of the evidence.