This is the Reading Instruction Show. I'm your host as always, Dr. Andy Johnson.
The topic of today's podcast is evidence-based instruction and evidence-based is in quotation
marks. Another name for today's topic is evidence outside of context is Faldarol.
The hoopla in the education world around the term evidence-based. That term is often thrown
around carelessly like a dead fish at a rummage sale and used to end discussion.
It's evidence-based, they declare. We use evidence-based instruction. Their t-shirts
proudly exclaim and on the sides of their shiny packages they print its evidence-based and you
say, of course it is. And of course you do. And of course it is. And pardon me all to hell for
asking the question. I'll try to do better next time. However, there are words and there is
reality. Reality is comprised of words, but words by themselves are not reality. Words,
words, words. Evidence-based is by a hyphenated word or term that has little meaning by itself,
other than to indicate that one has heard this term before and one thinks it appropriate to use
to explain or defend practices. But I would question and I would pause it and pardon me for
being so inquisitive that when you say a practice or strategy is evidence-based,
have you actually looked at the evidence of the evidence-based? Or are you just using the term
because you heard somebody use it? Now let us consider for the sake of argument some evidence-based
practices. Giving electric shocks to mice in a skinner box is evidence-based, but that doesn't
mean we're going to advocate this practice in a classroom. Yes? You could cut out all recesses
and make kids stay inside and fill out worksheets. And I'm sure you could collect evidence that would
indicate that these kids got better at filling out worksheets. It would be an evidence-based
practice, but that doesn't mean we advocate it. You could assign 10 to 15 words for students to
study every week and give them a spelling test on Friday. You could collect evidence that showed
that students got more correct on Friday than they did on Monday, but this doesn't mean they're
better able to spell under real life writing conditions. And it doesn't mean that traditional
spelling instruction is better than other methods or better than no method at all or better than
simply writing a lot with no spelling instruction. Yet, since there is evidence, it must be evidence.
Yes? No? You could give reading instruction to 25 students in Arkansas using a for-profit
reading program that was implemented with fidelity and compare it to reading instruction to 25
students in West Virginia using a meaning-based program where students read a lot of books. If
the average post-treatment scores of the Arkansas students was say 75% and West Virginia students
say 73%, you could indeed say that there was evidence showing that the for-profit program
students scored higher. But is it valid evidence? Is it reliable evidence? And if the evidence
was collected by the for-profit company, would it be creditable evidence? Yet, one could say
that the for-profit program was implemented with fidelity and it was evidence-based. Imagine
that. When someone trots out the tireless refrain of evidence-based, one must always
ask what kind of evidence? For whom? For what? For what purpose? Evidence outside of any
context is falderol. Falderol, by the way, is trivial or nonsensical fuss and that's
exactly what decontextualized evidence is. It's trivial or nonsensical fuss. In any research
study, the type of evidence collected depends upon the question asked and the method of
collection. For example, I could ask the question, will method X improve students' ability
to sound out words? I would give sounding out word pre-treatment measures to a group. Method
X would include sounding out word instruction. The post-treatment sounding out word measure
would most likely show that there was an increase in students' ability to sound out words. One
could thus say that sounding out word instruction is evidence-based. Okay, fair enough. There's
evidence to support its efficacy of sounding out words in students' ability to sound out
words. Yes, indeed. That's like saying teaching students to play tennis enhances their ability
to play tennis. But this does not mean that method X is better than other measures in
recognizing words or identifying words or comprehension text or comprehending text or
helping students learn to read. But yet, method X is said to be evidence-based. Now, let's
take a look at Special Ed World and the Science of Reading Comedy Club. This is the last
thing I'll say about evidence-based. Can it really be evidence-based if it only looks
at a certain type of evidence? And that's the whole problem with the liturgical chance
of evidence and evidence-based made by the Science of Reading Comedy Club to support
a litany of practices taken from their direct instruction holy book. That's the problem
in Special Ed World with their declaration of faith stating that practices used to manipulate
and control behavior are evidence-based. These represent a simplistic understanding of research
in the social sciences. Within Special Ed World and the Science of Reading Comedy Club,
controlled experimental studies are thought to be the only way of establishing causal
relationships that one thing makes the other thing happen. They have determined that only
a single type of research methodology can be used to ask and answer questions in the field
of education. This is ironically labeled as scientifically-based research. However, in
true scientific inquiry, the question determines the method. The method does not determine
the question. And how can anything be said to be truly scientific if it ignores or invalidates
an abundance of data? These groups, Special Ed World and the Science of Reading Comedy
Club, mandate that the same research methods used to study bacteria in a Petri dish be
used to study children in a classroom. But humans are not standardized products and there
are far too many variables in the real world to control or isolate. As well, the results
of large controlled experimental research studies do not always generalize to smaller
populations. In identifying scientifically-based research and evidence-based practices in education,
it's not the type of research methodology that's the problem. I mean, there's nothing
wrong with controlled experimental studies. I'm not advocating that they be eliminated.
It's the exclusive use of one type of research methodology to the exclusion of all others.
This creates a methodological people that results in a very narrow and unrealistic view
of the very educational reality it seeks to examine. The results is that terms like research
and research-based and evidence-based are often used inappropriately to justify a variety
of bad decisions. Saying something that is evidence-based does not end the conversation.
What kind of evidence? For what? For whom? For what purpose? We always must look at the
evidence behind the evidence-based claims.