This is the Reading Instruction Show. I'm your host as always, Dr. Andy Johnson.
Title of today's podcast is called It's Research Based. Now, there's a mistaken
assumption by some that if something is said to be research-based or evidence-based,
it means that it's been proven to work. It will work and it should be used
unequivocably. You just have to see the hyphenated word research-based in front
of or behind a strategy approach or method and you're in. No questions asked. It's
research-based. It proves your method works and all others are wrong. Glory
hallelujah. Your strategies are the right ones to use and anybody who thinks
otherwise is wrong wrong wrong. Oh, the joy of it. And if you say research shows
in front of a declarative statement, you can say to the world with joyful glee
and absolute certainty that the statement is right and all others are wrong
because research proves it. And we should certainly believe your point of
view because it's research-based. And by inference, any other point of view is
unicorn-based or monkey-based or fairy tale-based or pure folly. Now, some believe
that research is used to determine rightness and wrongness to find
certainty in an uncertain world. Research is used to separate sheep from goats,
chaff from wheat, saint from sinners, etc. etc. etc. A holy canon is then created
based on chosen bits of research. The research is chosen to support a
predetermined idea. This is our holy book, they say. The science of reading a holy
book of research-based. It is inerrant, unfailing. Why do I know? Because it's
research-based. We have research to prove that our science of reading a holy
book of research-based is inerrant and unfailing divine word of science. It can
be proven. For in chapter 2 verse 27 it says, quote,
whereily, whereily unto you we speaketh thus. Those who embrace it, the science
of reading a holy book of research-based, are right and all heathens are wrong.
Unquote. Paid consultants then go out and evangelize. In their evangelical zeal,
they demonize those who would dare challenge their thinking. They castigate
those who would speak against the science of reading a holy book of
research-based. Their views are the holy views handed down to them on high from
the holy unseen in a mysterious science. Science is to be loved, feared, and
obeyed. Will be it unto those who turn their back to science? The scientific
commandments are to be followed without question because after all it's science
and the science of reading. And how do I know? Because it's got science in the
title and it says words like research and research-based a lot. Yes, they're
zealots, but they're zealots for science. And by the way, did you know that you can
have two different views on reading both views being research-based? So let's look
at this for a minute, shall we? When you look at the research, if a claim is said
to be research-based, you should be able to look at the research that supports the
claim. That is after all what we do in academia. If a claim is made, the research
behind it is cited so that you can look at the research behind the research-based
claim and see if it's legitimate. And yes, pre-K-12 education is part of the
academy. It's part of academia. And the currency used in academia is peer-reviewed
research published in academic journals. Not I thinkisms or famous people or
radio journalists or anecdotal evidence or personal experience or data pulled
out of context or think tanks. In the academy, we use research and research is
not research unless and until it has been subjected to blind peer review and
published in an academic journal. Now, let's take a look at the importance of
blind peer review. Why is the blind peer review process important? Why can't you
just collect data and call it research? After all, that's what state
legislators and journalists do. But blind peer review is used to make sure that
you don't monkey with the statistics or your measures are valid, your data are
accurate and your conclusions are actually based on the data collected, not
what you want the data to say. Peer review is a subjectivity filter, a bias
filter. It's a monkey business filter. And it's not that we don't trust
researchers. Well, it is actually because we don't trust researchers. There's tenure
and promotion to earn, you see, books to write, appearances to make, consulting
fees to charge and interviews to give. So, it is common that there's
electric desires can get in the way of seeing clearly. What they want to find
can sometimes fog the lens of objectivity just a bit. And that's why we need a jury
of our peers to check our work before it's published. Now, also the blind peer
review process helps the authors of that study find mistake or analysis they
hadn't considered. Now, I've been part of the blind review process and it can be
brutal, but it always leads to a better product. Again, it is one subjectivity
filter to ensure there's a certain amount of objectivity in the research.
Good peer review makes it less likely that the research is not being used.
Peer review makes it less likely that the research won't be used to sell a
product or promote a point of view. Now, in blind peer review research, the
reviewers evaluate the study without knowing who conducted it. Hence the
term blind peer review. Reviewers consider such thing as the clarity of the
research question, the theoretical context in which the research question was
set, the adequacy of the methodology, the analysis of the data, the interpretation
of the data, the validity of the conclusions, and the quality of the
writing. Then the reviewers have four options. They can recommend the article
for publication, recommend for publication with revisions, suggest
specific regions be remade and resubmitted for consideration or recommend
that the article be rejected. Academic journals, yes, they publish studies,
research studies, but they also publish theoretical articles. Blind peer review is
also used for theoretical articles. Now, let's understand that a theory is built
on research. A theory is a dot-to-dot picture used to explain a set of facts
and understand phenomenon. Each dot on this dot-to-dot theory picture is a
research study. It's only by looking at many studies that the theory begins to
emerge and you begin to see the whole, not one or two studies, but many. Now,
theoretical articles are sometimes called secondary research. These are not a
lesser form of scholarly work. Some, however, equate a theoretical article
with opinion. The Educational Research Information Center, Eric, which is part
of the Institute of Educational Science, which is part of the US Department of
Education, they did this for a time. Anything that was not a controlled
experimental study, anything that was not used to test a hypothesis, was called
opinion, an opinion article. Now, here's my opinion on their opinion articles. What
a bunch of clowns. They don't understand. They are ignorant. While an individual
research study looks deeply at a part of a phenomenon, secondary research or
theoretical articles takes a wide look, looking at many studies at that
phenomenon. Both looks are necessary to understand clearly and move the field of
education forward. One is not more important than the other. Original
research is not more important than secondary research. They are both of
value and importance. So let's get back to peer review. Peer review simply
denotes a process, the process of peer review. The quality or rigor of this
process varies. Reviewers and editors of highly prestigious academic journals use
a process that is rigorous and very selective. These journals have low
acceptance rates and tend to have considerable influence on the field. Other
journals have a less rigorous review process and higher acceptance rates.
However, all are still considered peer reviewed journals. And the peer review
process is not without bias or flaws. Peer review does not magically make
research unbiased or pure. It's not possible for human beings to have a
completely objective unbiased view of anything. Peer review is simply one
filter to try to remove some of the impurities related to bias, methodology,
theoretical context, applications, and conclusions. One filter, but it is a
necessary filter. So let's take a look at white papers. Some scholars put out
what's called a white paper. On her letters website, Louisa Mote has a white
paper. It's entitled Literacy Professional Learning, 10 reasons why it's
essential. This white paper is used to prove to people that they should give her
lots of money to get all professionally trained by her letters
professional development. So what's a white paper? A white paper is a paper
that looks like an article you might find in a peer reviewed academic journal. It
has headings and subheadings. It uses objective academic languages, language,
and citations to bolster the declarative statements made. The only difference
between a white paper and an article appearing in a peer reviewed academic
journal is that a white paper has not been peer reviewed and has not been
published in an academic journal. In a white paper, people like Louisa Mote's
can say whatever they want to say and there's no one to counter them or
check their claims. And again, white papers look like articles found in peer
reviewed academic journals. They look like them, but they're not. And people who
don't know the difference reference them as if they were from peer reviewed
academic journals. White papers are published on a lot of websites and put
out by think tanks and political groups. White papers can be interesting. I write
them myself, but they're white papers. They're not published in peer reviewed
journals. And here's the thing. Louisa Mote's white paper was not submitted to
a legitimate academic journal. And when you actually look at the sources she
cites, when you try to be a responsible consumer of educational research, you see
that her claims have nothing whatsoever to do with the research being cited. When
you actually look at her citations and the studies she cites, you see that she
misinterprets, misuses, and falsifies research studies to support her claims.
She also cites poorly conducted research that has little to do with the point
she's trying to make. And you would know this if you didn't critically read and
analyze her cited sources. This is what a responsible consumer of educational
research does whenever confronted with the phrase research base or research. We
never take anything at face value. We have a natural inclination to doubt. So a
little bit more about peer reviewed academic journals. A peer reviewed
academic journal is different from a magazine. Magazines and newspapers, they
just send a reporter out to cover things. And reporters just write stuff. And by
the way, Emily Hanford is a reporter. And she's made quite a name for herself in
the area of reading instruction. But she's a journalist, a radio
documentarian. She's not an expert in reading. She's never taught a reading
class. And I wonder if she has ever read a research article related to
reading. Reporters and radio documentarians are not experts in their fields.
Their work is not peer reviewed. They don't have expertise. They don't know
what questions to ask or what data to look for. They don't know which sources
are valid and which are not. They don't cite their sources. They're not
responsible consumers of educational research. Reporters and radio
documentarians are important. But at the end of the day, they're just people who
write stuff. Unlike magazines, academic journals carry no advertisements. They
don't want to be swayed or give the appearance of being swayed by advertising
revenue. Academic journals are all written by experts in their fields and not
by journalists or radio reporters. As well, all articles are reviewed by a jury
of two or more people who are experts in their field related to the topic of the
article. If accepted, then it's published. That's the difference between a magazine
and an academic journal or a newspaper and an academic journal. Now, this idea
that research shows something's effective. The mistaken assumption based on a
lack of understanding about the research process and educational research is that
if a strategy has been shown to be effective with a certain population for
a certain purpose, for a certain time, and at a certain level, well, then it must
certainly be effective for all students, for all purposes at all times. It's
research-based. They happily declare as they march around the room with high
needs, research-based, research-based. We are so wonderful, research-based. But this
is clownery. When the statement is made that research has shown a practice
method or strategy to be effective, one must always ask effective for who, for
what purpose, how, at what level, for how long. Now, a case in point, John Hattie
wrote a very popular book, Visible Learning. In one chapter, he compares whole
language to direct instruction and he uses a meta-analysis. A meta-analysis is
when you put a whole bunch of studies that look at a similar thing into a
mathematical formula to answer a research question. It's like research that uses
research as data, looks at a whole bunch of studies to answer a question. Now,
after his meta-analysis on direct instruction and whole language, Hattie
comes to the conclusion that direct instruction is superior to whole language.
Imagine that. I guess then that this proves that whole language has been
debunked. And I guess Andy has been wrong for 30 years. And thank goodness for John
Hattie. Maybe that over-caffeinated little bald man will finally shut up. This, by
the way, points to the severe limitations of people outside the field of literacy
trying to do literacy research. John Hattie may be a meta-analysis expert, but
he is far from a literacy expert. In asking his meta-analysis questions, he
has no understanding of the context. He doesn't know what questions to ask, what
data to look for, what measures to use to adequately assess the data or the
research studies. And research, like John Hattie's, based on a false assumption,
always leads to invalid conclusions. Again, research based on false assumptions
always lead to invalid conclusions. Hattie's meta-analysis is based on a
false based on false assumptions. It's based on ignorance. Now, let me explain.
Direct instruction is a pedagogical strategy that has been shown to be
effective for teaching low level skills. Yes, all good teachers understand and use
direct instruction. And yes, there's research to support its use for teaching
some low level skills. It is effective for some purposes with some students when
used correctly. And every teacher should know how and when to use direct
instruction. And this doesn't mean that direct instruction is good for all
students for all purposes, for all times, and that it should become a complete
reading program. It's not direct instruction or whole language. And this is
where ignorance really shows. Because whole language teachers use direct
instruction. It's a pedagogical strategy. Whole language teachers use direct
instruction. So research based on a false assumption always leads to invalid
conclusions. And this shows the extreme limitation of John Hattie's work. So three
big points here to end up to conclude point number one. Teachers and
administrators must become informed and responsible consumers of
educational research. When someone says research shows this and research proves
that, we must always, always ask what research. Give me a citation. Let me look
at it. Effective for what? For who? At what level? For what purpose? How? And then
we must actually go and look at the research study because I know it's hard
to believe, but some people are a little weasley. Point number two. State and
federal legislators. Those who want to impose mandates on teachers telling them
what and how to teach. And I'm going to say this and I mean this in the most
respectful way possible, but pull your heads out of your collective asses. Do
what's right, not what's politically expedient. If you're not going to review
the research related to reading instruction, for God's sakes, at least
call in those who have. Stop making educational decisions based on the
testimony of radio journalists. And point number three. We all want students to
achieve their full literacy potential. Science of reading
ideologists. They want this thing. I know that's your intent and it's a good
intent, but intent does not mitigate impact in five years when the full
impact of what you've done will be realized. And people see just how much
time, money and resources you've wasted and how many teachers you've run out of
the classroom and why we have a shortage of teachers and not enough money to pay
for smaller class sizes. I'll be doing another podcast five years from now and
this one will be called I told you so. This has been the Reading Instruction
Show. I'm your host as always, Dr. Andy Johnson.