This is the Reading Instruction Show. I'm your host as always, Dr. Andy Johnson.
The topic of today's podcast is an important one, well they're all
important, for reading instructors, for teachers, for educators. It's called
collecting data is not research. There's a difference between collecting data and
research. While the research process includes data collection, collecting data
is not research. Let me explain. The basic research process is this. You ask a
question and then use data to answer the question. That's the basic process. What's
the temperature? That's the question. Look at the thermometer. That's the data. 72
degrees. That's the answer. That's the basic process. An example in reading. Do
our eyes fixate on every word as we read? That's the question. The data would be an
analysis of reading using eye tracking systems. That's the data. And the answer
would be no, our eyes do not fixate on every word as we read. The research
process includes data, but collecting data is not research. In the world of
science and academia and education and reading instruction, research is not
research unless and until it has been subjected to blind peer review.
Collecting data is not research. Let me explain. Peer review or blind peer
review refers to the process used to evaluate the quality of research. This
is how it works. Once a study has been conducted, researchers write an article
describing what they did and what they found. This article is sent off to an
academic journal for consideration for publication. The editor of the journal
selects reviewers who are considered to have expertise in the field and
considered to know something about the research or what the research was about.
These reviewers evaluate the study without knowing who conducted it. Hence
the term blind peer reviewer. They review. They take the name off the study, off
the article. Reviewers consider such thing as the clarity of the research
question, the theoretical context in which the research question was set, the
adequacy of the methodology, the analysis of the data, the interpretation of the
data, the validity of the conclusions, and the quality of the writing. They then
have four options. One, recommend it for publication. Two, recommend it for
publication with revisions. Three, suggest specific revisions be made and that it
be resubmitted for consideration. And four, recommend the article be rejected. Peer
review simply denotes a process and the quality or rigor of this process varies.
Reviewers and editors of highly prestigious academic journals use a
process that's rigorous and very selective. These journals have low
acceptance rates and tend to have considerable influence on the field.
Other journals have a less rigorous review process and higher acceptance
rates. However, all are still considered to be peer reviewed journals. Blind peer
review is not a perfect process, but it is a process and this process is
important. We recognize the process is not without bias or flaws. Peer review
does not magically make research unbiased or pure. It's not possible for human
beings to have a completely objective unbiased view of anything. Peer review is
simply another filter to try to remove some of the impurities related to bias,
methodology, theoretical context, analysis, applications, and conclusions. But you,
dear podcast listener, are the ultimate filter. You are the most important peer
reviewer. In this respect, you must always ask, does the strategy or approach work
with the students in front of you? Does it enhance their ability to create
meaning with print? Does it move them forward, unimpeded, in their journal to
achieve their full literacy potential? Does it matter if a strategy or approach
demonstrates significant results with a large sample size if it doesn't work
with your small sample size, your students? So back to data collection. Data is not
the same as research even though some would like you to think so. Let's say I
am the owner of a new type of reading instruction program. We'll call it RIDO.
RIDO is adopted by a school district and I decide to collect data. At the
beginning of the year, the average score on the Walmart standardized reading test
was 85. After three months, the Walmart standardized reading test was again
given. This time, the average score was 95. It raised 10 points. I collected data.
So I then put the following label on my RIDO website. Research shows that RIDO is
effective in raising students' reading achievement. Okay, I had data to back it
up. But it wasn't research. It was just data. We don't know what the variable was
here. What caused the scores to increase? Students' scores could have increased due
to maturation or they were exposed to good books and writing. Or maybe they took
the pre-test on a really bad day and the post-test on a really good day. Maybe
all the good readers were sick on the pre-test and all the struggling readers
were sick on the post-test. Maybe my sample size was only 10 and there are
also no comparison groups. We don't know if RIDO was better than something else or
better than nothing or better than what usually is. Confusing data with research
is a common practice in the for-profit realm. Or let's say I compared RIDO to a
variety of other programs. Posting test scores show that the RIDO average was 95
on the Walmart standardized reading test while the average score of the other
programs was 93. I could say that RIDO outperforms all the other programs.
However, the difference in scores was not statistically significance.
Statistical significance means greater than could occur by chance, which means
there really was no difference. Confusing data with research is also a common
practice with those who have a political or ideological agenda. Without peer
review, we'd have no sense of how these other programs were taught by who, for
how long, or how often. This is done often in the reading instruction world
when they try to say one program or approach is better than another. A
common trick is to have highly trained teachers of program A give extended
specialized programming instruction to a group of students and the measures used
to determine growth essentially replicate what program A is teaching. And
of course program A is going to have higher scores than the other programs.
Maybe these other programs did not have specialized training. Maybe they did not
have specialized instruction. We just don't know without peer review. Confusing
data with research is a common practice outside of education by those outside of
the research or literacy field. Another common example is this. Test scores are
going down. For example, here in Minnesota some data show that reading test scores
in our state have gone down over the last three years. Conclusions by some are
these. Oh, our schools must be doing a horrible job. It's the teachers or the
problem is the teacher education programs. They're not training new teachers how to
teach reading. That's the common wind. We need to buy a product with big fancy
words and fancy charts and graphs or we need to pay a private for-profit person
or group or service to fix the problem. If it's a for-profit and has lobbyists
and PR and marketing people, it has to be better than anything someone from the
public sector could offer. After all, lobbyists and marketing people are always
much more credible than say people with terminal degrees and literacy instruction
who spend years studying research and how to best teach reading, right? This is
the case here in Minnesota. Of all the variables impacting children and their
reading scores, some have decided that it has to be the teacher education programs
that's making the test scores go down. It just has to be that. State
legislators and school administrators who have undoubtedly never read
a research article about reading instruction or teacher education or
anything have decided that since they once went to first grade that they must
have all the answers for teaching reading and preparing teachers. They want to
mandate letters, a for-profit training program, professional development
program for teachers, be mandated for teachers and pre-service teachers and
teacher preparation program. After all, experts say it puts the Y behind the
phonics systems and it is effective and it's research-based. But who is an
expert? Am I an expert? Are you an expert? What does it take to be an expert? And is
the expert an expert in reading instruction or perhaps the expert is an
expert in animal husbandry? It doesn't really say. But apparently an expert has
said it. An expert has said this letters is an important program. After all an
expert said it. You can't argue with an expert can you? Now, reading scores may
have indeed gone down in the last three years but has anything else been going
on? Oh yes, we've had a pandemic and students have been at home and George
Floyd and housing shortages and crowded classrooms. But if you look at long at
the long term at the National Assessment of Educational Progress data by the US
Department of Education and AEP data, Minnesota 4th and 8th grade students are
at about the same place they were in 1998. In 1998 the average reading scores for
4th grade students was 219. It steadily rose to 225 in 2017 and was 222 in 2019.
Same with 8th grade. About the same. We're at about the same place. Now it's
natural for there to be a fluctuation in scores but instead of reporting on the
steady rise in reading scores from 1998 to 2017 or the statistically significant
steadily rise in scores from 1972 to 2019, the focus is on a short-term dip in
scores. Instead of calling for smaller class sizes and better working conditions
for teachers and funding for health care or food assistance, the call is for a
mandatory for-profit letters training program for teachers and for students in
teacher preparation programs because the experts say...