Greetings, I'm Andy Johnson. This presentation
is called Understanding Evidence and Research.
I have a lot of notes and handouts and other
things related to this at my free sub stack.
Plug in the reading instruction show and you
will find it there. It's free, there's no subscription
charge. Also, a lot of people have asked me about
teaching writing and writing instruction is one
of the most under instructed skills in our schools.
Again, another free substack, how to teach writing,
you can find it there. All the stuff there is
based on my book. And of course, like the rest
of 90 % of America, I have a podcast with tens
of listeners. Okay, let's start. Context. matters
to understand things, to understand research,
to understand data, to understand evidence. Context
matters. You can't pull things out of context
and try to understand. So let us look at the
big picture. The big picture is to enable all
children to develop their full literacy potential
so that they can use literacy for real life purposes
in the real world, not to pass a unit test or
to get higher scores on this and that. So to
use literacy to achieve real life goals. And
literacy includes reading and writing, but also
speaking, listening, and thinking. So I ask unto
you, what constitutes evidence of this? Beware
of the number monkeys who try to describe learning
only in terms of numbers. If it can't be quantified,
they tell us it doesn't exist. Reading is not
sounding out words. It's creating meaning with
print. And we read for two purposes. We read
to understand information, and we read to enjoy
books. That is the end state, the goal. All our
objectives should lead to that goal. Everything
we do to teach reading should lead to those two
goals. And the ability to do these things should
define students' reading ability. So what is
evidence of a literacy crisis? We're talking
about evidence. The crisis, a crisis, any crisis,
is a time of intense difficulty, trouble, or
danger. And some insist we have this thing called
a reading crisis. You can read the media and
literacy. Thousands of children functionally
illiterate. Oh, really? Just because you see
it or hear it doesn't mean it's true. Teachers
criticize their colleges of ed for not preparing
them to teach reading. How many teachers? Two
teachers? Four teachers? How old were they? First
year? 20th year? 50th year? How many colleges
of ed? We must look for evidence as we consume
the media, as we consume various forms of media.
It's alarming, they tell us. Children are severely
behind, teachers say. Behind what? Behind who?
What educators are saying what? Just because
is it two educators saying it? Journalists, journalists
are important in a democratic society. I'm not
getting down on them, but journalists write stories.
Stories are not evidence. Stories are things
that people put together. not necessarily for
accuracy, but for impact, to get clicks on their
little websites, to get awards at banquets. That's
what journalists do. There is not legitimate
evidence that there is a literacy crisis. Evidence,
legitimate. Paul Thomas talks about it and has
written about all the various literacy crises
starting in the early 50s. And there'll be another
one coming around in 10 to 15 years when this
one has found to be another boondockle or when
some publisher needs something else to be sold.
There will be another one when they find this
science of reading thing doesn't work. Why? Because
it's much easier to blame schools and teachers
and parents than to address social issues, systemic
poverty, economic inequality, overcrowded schools,
income inequality, lack of school funding. These
things impact learning. But it's easier to say
schools are failing, teachers are doing a bad
job. Everybody who's been to first grade seems
to have all the answers to fix public education.
Teach more phonics, George Bush told us once.
Higher standards, get tough. Get rid of teachers
unions. Reward the good teachers. More tests.
Reward high performing schools. Interesting.
Simple, simple answers. to complex problems.
Blaming teachers takes people's attention from
the real issues and teachers can't fight back.
This has led to a teacher shortage. Teacher turnover
is the leading factor in teacher shortages. They
are leaving. Qualified teachers are leaving teachers
who love their job, good teachers, for reasons
other than retirement. There's no shortage of
teachers. There's a shortage of teachers who
are willing to put up with baloney. As an example,
I am retiring in three months and one of the
reasons I'm retiring is because I would be required
to go through letters training, what I call re
-education camp. Just like in a clockwork orange,
they force that guy's eyes open to watch those
scenes. I would be forced to watch letters or
to take letters training. There is not a reading
crisis, a reading achievement crisis. Now we
are going to look at data from the National Assessment
of Educational Progress. This is part of the
U .S. Department of Education. They take a large
sample, a representative sample. They give tests
to students in grades four, eight, and 12 every
year. It's called the nation's report card. It
gives us a sense of how we are doing. Rikin,
Ruby, and Risco. There's no irrefutable evidence
that we are experiencing a national literacy
crisis. Crisis would look something like that
or something like that. It would not look like
that. These are long -term reading scores in
fourth grade and in eighth grade, starting in
71. You can see there's just a gradual release
of scores, but there's a natural fluctuation,
and that's called normal, normal. This is the
latest one I found a couple of days ago on the
website, and it's hard to find this data. They've
included a whole bunch of stuff that makes it
difficult. No statistically significant difference
from 2013 to 2020. Of course, we had the COVID
dip. didn't we? And before the dip, we were higher
than we were statistically significantly higher
than we were in 92. Same with eighth grade. Here
is the COVID dip and we expected that. But from
12 to 20, no significant dip. Reading scores
fall to new lows, says Sarah Schwartz from Education
Week. And she's a reporter. She just writes a
lot of stuff. She is a reporter. She's not an
academician or a scholarly writer. But here's
the thing, even if she weren't lying, think about
it. This is the dates when all the states have
passed their miracle reading legislation. Mississippi
in 2013. Here's when the miracle started. And
look at all these other things, all these other
miracles starting 2009, 21, 22. Here's Minnesota
right there in 2023. So let the miracles occur.
We should just be sitting back enjoying our cup
of coffee and just watching the miracles unfold
in front of us. Dramatic pause. except they are
not. But even if there were a crisis, which there
is not, the science of reading a fellow's people
have determined, have claimed that they've isolated
the causal variable. They say that they know
both the cause and the solution to a problem
that does not exist. But let me tell you, my
friends, Randomly determining causality is not
science. That's baloney, sometimes called pseudoscience.
These are all the variables that impact student
achievement. And you can pause and look at it
yourself for a good time on a Saturday afternoon.
Again, we must distinguish between scholarly
articles, scholarship, and journalists. Journalists
write down what people tell them. People tell
them things, they write it down, and they create
their stories. And everyone wants to be the next
Emily Hanford. Look how famous she is and how
much money she's getting and how many speaking
gigs. Oh, I want to be just like Emily. publish
stuff. They put numbers to it. They have sensational
things. They always have a picture of a kid who's
crying or looking distraught. And look at this
graph. Oh, that's just horrible. That's horrible.
Pulling things out of context, putting them in
ways that create a sensation. It's a straw man
argument. This is when you create a cartoonish
version of something and argue against the cartoon.
Things like balanced literacy. People describe
it in ways that it is simply not true. Whole
language, the three QE systems. Some people are
just horribly upset about what they call Q -ing,
and they don't even know what it is. They're
describing something horrible. Public schools,
DEI has been made a swearing word. Critical race
theory, same thing by people that have not taken
the time to understand what it is. I had this
fellow on my show. I was interviewing him. He
asked me to, so I did. He had a book which had
a very interesting premise. The book was called
Failure Factor, and in this book, he described
administration. as changing teachers' grades
to show that they were doing a good job. And
in Baltimore, apparently, instead of a superintendent,
they have a CEO, hence the factory motif. But
Brother Chris here has some very simplistic answers
to a very complex situation. And he either misunderstood
or misrepresented information. His answer, of
course, was just to hold students to higher standards.
Oh, okay. Just hold them to higher standards.
Duh. Why didn't I think of that? Based on a lot
of I -thinkisms. Now notice, here's some of the
stuff from his book indicating failure. He says,
well, the budget has stayed consistent, all right,
and the number of enrollment has gone down significantly.
Look at all these things. But none of these things
are indicative of the quality of education occurring
in classrooms. SAT, that's a self -selected test
that people going to college take. That's not
a representation of how schools are doing. AP,
advanced placement exams, again, a representative
sample. Attendance, you can have quality teaching
going on, but that does not impact attendance
or dropout. or absenteeism, or enrollment. Some
of these are societal factors. So either he doesn't
know, he doesn't understand, or he's purposely
misrepresenting to get his little statues and
to get book sales. Again, he said, more discipline.
Hold students to higher standards. Get rid of
teachers unions. I hear that a lot. And in my
book, I will be talking about teachers unions
at some point, not today. Here's the thing I
read in his book that made me want to throw up
in my mouth. When a reporter focuses on one beat,
they become an expert on that beat. Oftentimes
a beach reporter knows the subject matter better
than the industry professional they cover, unquote.
What do you say to something like that? You don't.
You just vomit in your mouth a little bit. They
write stories. This is a story written like a
detective novel. Watch the City has fallen into
a state of decay. And that was certainly my view
one September afternoon in 2017, as my photographer
and I were. You get the idea. This isn't scholarly
writing, not that you need scholarly writing,
but he's creating a story for impact. That's
what they do in television. That's how you get
those little statues at award ceremonies. Everyone
wants a statue. I want a statue. I welcome journalists
into our academic conversations. I welcome you,
but you do not get a free pass on ignorance.
In my conversation with Chris and John Stossel,
I understood that you have to have some background
knowledge to understand and interpret what you
see and hear. I often had to correct the presumption
upon which questions were based before I answered
questions. They had a shallow and disjointed
knowledge base. There's nothing wrong with that.
But you don't realize that. It makes it difficult
to engage in academic discussion. discourse with
these people, these people, with a journalist
sometimes. I don't want to generalize. Reading
instruction is first and formally an academic
topic, and it has been, discussion has been going
on in academic, academia, academic discussions
for decades. And there's always been people with
differing ideas. There isn't a reading war. And
again, You are welcome as anybody is into our
academic discussion. Anybody can publish in an
academic journal. It's subjected to blind peer
review. Anybody could speak at a conference,
but allow for questions, get alternative views.
When you say research, cite the research. You
should not get a pass on ignorance. And that
is happening. As I mentioned, I was interviewed
by John Stossel. The interview took about 60
minutes. He put together a story and my little
part, I'm guessing was two to four minutes at
most. And he created a collage with it to create
the story that he wanted. This is an email I
got from a producer. We're doing a story on the
literacy crisis in America. of which there is
not, I came across your advocacy of whole word
reading instruction. Where did you come up with
that? I don't know. Do you know what whole word
reading instruction is? I've talked about whole
language, but that's different from whole word.
But anyway, he said, we would like to get your
take on the subject. They would like to get my
take. Imagine that. They didn't want my take
on anything. They wanted to create a collage.
Now, the speaking fee for John Stossel, for him
to do a keynote is $75 ,000. I know because I
pretended I was someone else and I contacted
his speaking bureau. Am I a bad man for doing
that? The speaking fee for me to do a keynote
is give me a little flattery and a hamburger
and I will be there. Now, in 1992, somebody determined
that the NAAP reading scores weren't low enough,
so we need to invent categories, advanced, proficient,
and basic. Well, what is basic? We just assume
that it's something. What is proficient? We just
assume that it's something. What's average? This
is what Rinking Ruby and Risco said. The cut
scores used to define these categories were arbitrarily
set by a small group of hand -picked politically
appointed experts against the recommendations
of the psychometricians who have been contracted
to advise the development of this approach. You
hired people? Interesting. Okay, interesting.
Categories and levels and content are arbitrarily
defined. citing some more sources there. Now,
if you go on the website, US Department of Education,
they report in percentile rankings, 10th, 25th,
50th, 75th, and they have so much data and little
data bits, it's designed more to confuse, and
you can pull out little data bits to create a
variety of stories, a variety of impressions.
If you do this, pull it out of context. context
as needed. This is average right here, so let
me explain. 1992, this is average, so they decided,
hmm, I think we'll make basic a little below
average and proficient a little above average.
What would that look like in a normal population?
In a normal population, we have the bell -shaped
curve where 50 % are below the mean and 50 %
are above the mean. And in every school, if it's
a normal population, we're going to have 50 %
below the mean or below average. Well, if they
decide that this is basic right here, by definition,
40 % are gonna be below the basic level. And
60 % are gonna be below the proficient level.
And you can say, oh, we have a crisis. Only 40
% of our students are reading at the proficient
level. You see how the game is played? That's
how they play the game. That's not right. Eighth
grade, same thing. Same thing. Same thing. That's
not right. Walton said, this is from their website.
This is from the Center for Educational Statistics.
Proficiency levels are based on desired expectation.
These levels are aspirational, not mathematical.
We would like this to be basic. We would like
this. there to be used on a trial basis, we need
evidence to determine if they are reasonable,
valid, and informative. Hand -picked political
appointees, arbitrarily decided levels, any time
you hear about NAP levels of basic and proficient,
even on state level tests. These should always
be put in this context and they should always
be questioned. State criterion reference test.
You can't compare state to state because each
state has a different test, but someone establishes
a criterion. We think third graders should know
that and sixth graders should know that. A test
is given every year. And we think this is basic,
and that is, arbitrarily decide who decides the
criteria, who decides the content of these tests,
who decides what's advanced, proficient, and
basic, and how are these decisions made. I had
to do a lot of looking here in Minnesota on their
websites, and I finally found, after a lot of
phone calls and tiny print on their websites,
that originally these MCAs were based on the
Common Core State Standards. They took these
three -dimensional standards and made a two -dimensional
test. Of course, you never actually get to see
what's on the test. And in your real life, how
often have you had to do these things? But yet
these are used to define the literacy potential
of our students. These are, and these standards,
Common Core, were aspirational, not mathematical.
To prepare students, we would like them to be
able to do these things at each level, aspirational.
And this idea, like they have here in Minnesota,
that every student will be reading at grade level,
that's simply absurd. That creates winners and
losers. Grade level. How do you determine grade
level? Criterion based, norm reference based,
aspirational, inspirational, fairy tale based
measures. Now in Minnesota, a couple of years
back, they claimed that test scores were falling,
but no test scores were ever reported. They don't
give you that. They simply give you percentages.
And four of these arbitrarily defined categories.
Those aren't test scores. And you can't use percentages
to make comparison. This is percentages of students
at or above proficiency. That's not a comparison
of scores over time. That's what's called in
the business baloney. And that's the family friendly
term for it. So tips for evaluating scholarly
work. And again, Academic discourse is when you
use research and reason to make a case for one's
argument. Argument here isn't the yelling kind
of argument. It's where you put forth a proposition
and you make a case for that using reason, using
reason and research. And disagreement is a natural
part of the academic conversation. It's how we
evolve our thinking. There should never be a
singular type of thinking. This insulated bubble
that the reading league tries to create for itself,
that creates cognitive inbreeding and deleterious
mutations. We need to hear other ideas to evolve
our thinking. I say the reading league because
when I post things on their various platforms,
they block me. Oh, we can't have our people hearing
that idea. That's horrible. They may believe
the wrong thing. So they block me. Interesting.
Now, it's important that we critique each other's
work. That's what we do in a scholarly, in academia,
in a scholarly context. Now, I have a book coming
out this summer, the one you see here. And I
hope you critique it. I hope you tell me what
I got wrong. Don't tell me I'm a bad person or
I'm a horrible person. I already know that. Don't
tell me I'm not very smart. I already know that.
Tell me what you disagree with and what I got
wrong. And here's the thing. If you tell me what
I get wrong and I'm convinced, I will make another
video. But don't just say I'm a bad person and
I got it wrong. That's not academic. Discourse.
Disagreements based on works published or presented
in an academic context. That's what academic
discourse is. Not on personal attributes, not
on personal emails. This is where we disagree.
We must have disagreement. That's how we evolve
our thinking. Academic discourse using reason
and research. research -based theories. And by
the way, there never was a reading war. That
was a term invented by people outside of an academic
context in the media to sell books or papers,
whatever. There has always been differing ideas
for decades since the National Council of Teachers
of English was formed in 1911. the International
Literacy Association, then the International
Reading Association in 1956. People have differing
ideas based on their theoretical orientations.
Academic discourse is how we hold ourselves accountable.
And I've taken Sally Shaywitz to terms for her
research, for her bad research. but I'm sure
she's a wonderful human being and she has to
be very smart, but I have trouble with all the
flaws and confounding variables in her research.
How does she get away with that, I wonder? The
differences we have in academia are based on
theoretical orientations. Every theory can be
supported by research data and evidence. Theories
are neither right nor wrong. They're weak or
robust. And I'll demonstrate that. Flat earth,
that's a theory. There may be little bits of
data. It's a very, very weak theory. And I think
the simple view of reading is the flat earth
theory of reading. Yes, there's research to support
it. but it's weak research. It's a weak theory.
Theories are based on research. Research study
is a data dot. That's why you can have separate,
you can have different research -based theory
connecting different data dots differently, and
they're both research -based. It's weak or robust.
theory. A robust theory connects a lot of data
dots. A weak theory leaves a lot of data unaccounted
for. The interactive theory of reading, psycholinguistic,
neurocognitive theories, account for a lot of
data dots. The simple view of reading account
for a few. They ignore a whole bunch of stuff.
Now here's one way to transform weak into robust
theories. You can do what the National Reading
Panel did. They ignored or excluded a whole bunch
of research. Oh, all that research over there.
That's not good stuff. Only this kind of research
is good. Okay. Reading instruction is an academic
topic. Academic arguments based on theoretical
differences have been going on for years. It's
a natural part, and as I said, the term reading
wars was invented by the media. All right, let's
take a look at scholars. We are looking at evidence.
Scholars should not get a free pass on obfuscation.
Common tricks, using buzzwords. That's a word
used to elicit an emotional response. rather
than accurately transmit information. And using
them in an academic context, that's an academic
buzzword used to create the illusion of something
important. Now, take a look at these academic
buzzwords. They're used all the time to make
something sound very, very important. Could you
define each one of these? If I went to a reading
league conference and sat down and have each
person write a definition of these, how much
similarity do you think there would be in these?
What is evidence -based instruction? Well, you
know, it's instruction and it's based on evidence.
Okay, buzzword. And we already talked about the
straw man argument. Louisa Moats, oh, Louisa,
Louisa, Louisa. She wrote this for the Thomas
Fordham Institutes, not peer reviewed. It's,
but here's the ultimate straw person talking
about whole language, uses teacher modeling,
not direct instruction. Instead of teaching children
how to read and comprehend, they engage in shared
reading. Students can repeat the language and
read by osmosis or memorization. Just, Louisa,
do you not know better? That is not at all what
whole language is. Do you not know better or
do you know better and you're doing this anyway?
Both are frightening propositions, but this is
a straw man argument. This is a cartoonish version
of something. Do you have no decency? Don't do
that. Baloney, is interpreting research or misrepresenting
decontextualized data. This is why we have to
critically evaluate all the scholarly work. We
can't be snookered by important sounding brain
words. We have to pull them apart. We have to
examine the propositions and the citations. And
a cow pie is when you put a whole bunch of academic
buzzwords together. to create a conveniently
inaccurate perception of something. Here's an
example of a cow pie. Goldenberg wrote this in
reading research quarterly, talking about reading
wars. Now, a proposition is a basic claim or
assertion within a sentence. You can say it's
true or false. The dog is in the house. Three
propositions. There's a dog. The dog is in something
and it is the house in which the dog is in. Three
propositions there. Here's Goldenberg's abstract.
The author's intent is to begin with the knowledge
base of the science of reading, build onto it
research on effective literacy instruction for
ELLs, and then begin articulating what could
become a science of reading instruction for students
who are learning and progressing as readers in
English while learning to speak and understand.
It sounds very good. It sounds so wonderful.
But let's, when we look at that cow pie, we can
find here's just two of the propositions. There
is a knowledge base and the science of reading
has it. The science of reading knowledge base.
Where is that? Where can I find that? Who put
the knowledge in the base? Is there a balanced
literacy knowledge base? Hmm, these are buzzwords
strongly together. I could do that. The presenter's
intent in this presentation is to begin with
the knowledge base of reading research, build
on to it research related to neuroscience, psycholungus.
You get the idea. Anyone can create a cow pie.
It's not that tough. It creates the illusion
of stuff, but it doesn't transmit information.
We have to be wary when we see things like this.
This is a misinterpretation. This is an example
of baloney. In that paper, he referenced two
research studies to make a case for structured
reading instruction. And this is what he said.
He looked at von et al and Eri et al's research
studies. They included explicit and systematic
instruction of targeted knowledge and skills
and daily lessons that include activities focused
on the five core skills of phonological awareness,
phonics, fluency, and comprehension, developing
oral language, integrated oral language. It sounds
very good. Stop using my cow pies, the cow says.
But when we look at that, and I just so happened.
to look at each of these studies, and I've happened
to analyze both of them very carefully. These
are the pretty pretty words that Claude Goldenberg
uses. Oh, they are so pretty, pretty pretty words,
but they are conveniently inaccurate because
in his description none of these words were supported
by the research he is citing. Yes, some of them
were used as descriptors in the studies. Their
description was often inaccurate, incomplete,
or just plain wrong. But none of the pretty words
were variables that were isolated and compared
in these two studies. Why is that important?
And here's more baloney. Effects were moderate
measures on English, phonological awareness,
et cetera, et cetera. OK. Principles of early
reading instruction. Okay. Isolating variables
in controlled experimental research, which is
the science of reading research standards. That's
not mine. It's used to determine if one thing
caused the other thing to occur. If structured
language causes something to occur, but you have
to start with two groups that are similar and
exposed to similar conditions. The treatment
is the variable. That's the only difference that
should be between the two groups. And if there
is a difference on some measure between the two
groups at the end, and if that difference is
statistically significant, we can say with some
amount of confidence that the treatment was the
cause of the variable. That's a very simple overview.
of controlled experimental research. But in both
the studies that Goldenberg is citing so confidently,
they compared students who received individual
tutoring to students who received instruction
in small groups. And they said, oh, it was the
specific type of tutoring that caused the difference.
When in fact, it had nothing to do with the tutoring.
One was individual tutoring with individual tutor
with lots of feedback and a whole bunch of stuff.
And another was in small group using prepackaged
remedial stuff. It was designed to give them
a result that was favorable to the group that
they wanted to. As an example, this is what it
was. My research question was, do blue cages
cause mice to gain weight? And I said, yes, they
do. Because we compared them, and the blue cage
mice were heavier, and the yellow cage mice were
skinnier at the end of this experiment. So if
you want to lose weight, paint your house yellow.
And you see the confounding variable here? These
aren't equal groups. One was fed marshmallows,
the other was fed salaries. In the studies, Goldenberg
cited the type of metaphorical food given to
the metaphorical mice were different. They claimed
it was the color of cages that caused improved
reading scores. Anyone with a sixth grade education
could see that if you read the studies carefully.
So three questions, which A controlled experimental
research will not answer. Did Goldenberg not
read the studies? Did he read the studies and
not see the confounding variable? Or did he read
the studies, see the anomaly, and make the citation
anyway? I would like to know. Understanding research.
And by the way, if state legislators were really
serious about evidence -based practice for reading
and research -based reading instruction, but
they're not. If they were, every school district
would have a subscription to the same kind of
legitimate research database that I do as a university
professor. And not what works Glaringhouse or
Eric. That's not legitimate. That's not legitimate.
How do you expect teachers to make research -based
decisions and evaluate evidence -based practice
without research and evidence, and yet you blame
them and you beat them up. That's not logical.
That's not science. There's no such thing as
the research. The research says. The research
shows. I'm often asked this, show me the research.
What does the research say? They demand of me
the research. By the way, when anyone uses research
in a declarative sentence, they must, in an academic
setting, cite their research. You can't just
say, research says there's no such thing as gravity.
There's no such thing as the research. There's
no objective God. Oh, almighty research, we prayeth
unto you, give us truth and enlightenment. There's
no God who offers this objective view of reality.
There's no such thing as the holy book called
the research, where you can look up and say,
ah, the research says. Maybe they mean Thea research,
a researcher named Thea research. And they're
saying Thea research says. Maybe I should call
my next book the research. So you could say the
research sets. Research involves science. Science
uses research. It's a process of asking a question,
collecting data and using data to answer that
question. The question up front is important.
Louisa Motes tends to just collect data and then
come to a conclusion. That's not research. What's
the weather? Question. Look at the thermometer.
Data. It's hot. Data to answer question. That's
a basic of research. It's a systematic process
used to collect and analyze data. Systematic
means there's a fixed plan. based on a set of
principles that are generally agreed upon by
the field. And the research method is a systematic
process used to gather and analyze that data.
That's important because there is no scientific
method. There is no the method of science. There
are methods of science. Controlled experimental
research, the controlled experimental research
is only one method of science. We must understand
that conducting research in the physical sciences
is a bit different from research in the social
sciences. With human beings, we cannot control
all the variables. We cannot put our children
in giant Petri dishes. and give them similar
conditions. That's not how it works. And research
is not research unless and until it has been
subjected to blind peer review and published
in an academic journal. Here's the basic blind
peer review process. This is why it's important.
You do a study. You write it up. You submit the
manuscript to an editor. An editor submits it
to blind peer review. These are experts in their
field. They don't know who the author of the
article is. The author doesn't know who the reviewers
are. They review that and they make a recommendation
either to reject, accept with revisions, or accept.
There, and it's usually two to five reviewers,
their comments go to the editor, who then makes
a decision based on their comments to accept,
reject, or accept with revisions, and they work
with the author. That's the process, checks and
balances a filter. That's different from the
process a journalist use. People tell them things,
they write a story, and they submit it to the
editor who approves or asks for revision. Much
different. And the process used to put stuff
on the internet. You think of stuff, you write
it, and there you go. It's on the internet. Interesting
article I recently analyzed. It's not research.
These publications don't go through blind peer
review. They just hire reporters to say stop.
American public media. Data is not research.
Published reports are not research. White papers
aren't research. Unpublished theses and dissertations
do not go through blind peer review. Articles
describing research are not research. Famous
researchers aren't research, secretaries of education
aren't research, and anything on Lexia Learning
website is not research. These are often used
as if they were by people outside of academia,
collecting data, citing white papers, think tanks,
not research. Not. research. Non -profit does
not mean non -biased. These are all think tanks
that have things to say about education, but
they all have an agenda. The two kinds of research,
basic and applied, basic science, and applied
science. Basic is research that is done apart
from the setting in which it's used, eye movement,
brain imaging, miscue analysis. Applied research
is done in the setting in which it will be used
in the classroom or learning environment. And
controlled experimental research is used to isolate
a variable to determine a causal relationship.
This thing caused that to occur. Now the science
of reading research standard says only applied
research using control and experimental groups
can be used to make decisions about reading policies,
products, and procedures based on the mistaken
assumption that you can isolate all the variables
to determine universal causality and generalize
that to all of humanity. That's a mistaken assumption.
Research is not research unless it has been subjected
to blind peer review. Research is different from
collecting data. Data is not research. I could
collect data to show that I am the greatest professor
in the world, but that wouldn't be research.
What's a good professor? What's a valid measure
of a good professor? These are all things that
I could collect. One of these, oh, look at my
course evaluation. Oh, look at the number of
YouTube views. That proves I'm a good professor.
I highlight this because at our university, someone
wanted to have metrics, I hate that term, metrics,
to show that our College of Ed was doing a good
job with teachers. So to get metrics, they found
something that could be measured. And they went
to the scores on the state teacher licensure
exams that our students took, and they said,
oh, look at the scores, based on that, we're
doing a good job. That's not a valid measure.
And here's the thing, if we spent the three semesters
we have them, just preparing them to take those
tests, our scores would be out of this world.
But we try to prepare them to do this thing called
teach, teach. Imagine that. A Massachusetts rep
said, show me the data on queuing. The data,
the data. Data is not research. Anybody can collect
data. And here's the thing on queuing. Queuing
is not a method or a approach. How's that? There's
some data. The three queuing is not a method.
It's not something you teach. And we'll talk
about that later. Methods of science. The scientific
method. There's no such thing as the method.
A variety of research methods are used to study.
And there's two groups, quantitative and qualitative.
Let us understand the basics. Quantitative research,
again, seeks to find that causal relationship,
controls or manipulates reality to isolate a
variable. You have a measure. You have two groups
that are similar. You isolate one. The only difference
is this treatment. And then you have a post -test.
That is the basics. The difference must be statistically
significant. That means it's greater than could
occur by chance. Now, you don't need to understand
statistics to understand statistical significance.
statistically significant at the .05 level, that
means in less than 5 % of the cases, can we say
that that difference was attributed to chance?
In 95 % of the cases, we can be confident that
the difference between the two groups was caused
by the variable. .05 is good, .01 is the best.
but 0 .05 is generally regarded as the standard.
Controlled experimental research seeks to quantify
reality. But here's the thing. By controlling
all the variables to study reality, you make
that bit of reality a bit of non -reality. You
mess with the environment. It's not how it really
is if you're controlling all the variables. Qualitative
research seeks to study reality as it is. There's
Jane Goodall studying apes in the wild making
notes. There's a variety of types of qualitative
research. Here you see a professor in there taking
notes that helps us understand, that sees the
people behind the numbers. By using only quantitative
research, we are describing learning and human
beings in numbers. But in 2002, the Education
Science of Reform Act said only quantitative
research could be used. That's the only kind
of research that would be funded. That's the
only kind of knowledge that's valid. So a whole
bunch of qualitative research that helped us
understand. that helped us understand the numbers
was eliminated and slowly went away. It has to
be experimental or quasi experimental research.
No qualitative research, no government funding.
So it's not the use of controlled experimental
research that's problematic. It's the exclusive
use. you disallow us to see the people behind
the numbers by disallowing a whole bunch of good
qualitative research. And this is what the science
of reading standards are. Strategies and practices
that lead to improved outcomes, yes. Strategies
and practices are supported by experimental or
quasi -experimental research, establishes causal
variables and the research has been conducted
in an authentic learning environment. The science
of reading says they use strategies and practices
that have been determined to be effective by
these standards. Those are the standards and
I would be just fine with those standards if
they were consistently applied. Now here's an
example of baloney. In Minnesota and Massachusetts,
they recently passed legislation. They assumed
there was a literacy crisis. They assumed teachers
were teaching the wrong thing. And they assumed
the problem was balanced literacy and the answer
was structured literacy. And the causal relationship
was determined by anecdotal evidence. That's
not science, people. That's not science. If you're
going to be a science of reading, you have to
be consistently scientific. The science of reading
should not be given a free pass on non -scientific
thinking. And these are two examples of things
that are given a pass. Do these. As research
show that they lead to improved reading achievement
or improved teacher ability to teach reading
effectively? And have they been compared with
something else? No single study provides irrefutable
proof of anything. Research is a lot like sighting
in your deer rifle. I grew up in a small town
in a deer hunting culture. and every fall we
would sight in our rifles to see if the rifle
was shooting straight. The scope often got bumped
over the winter and we wouldn't take a single
shot. We would take a lot of shots to give us
a sense of where the scope was shooting because
we could have wiggled or there could have been
something that happened with the table that gave
us an extreme shot. By taking many shots, we
get a sense of where the scope is shooting, and
then we adjust our scope. But even then, if we're
pretty certain of where it's shooting, even though
we start walking in the woods, it's the person
behind the scope that determines where the bullet
lands and a bunch of other variables. No single
study. Research is like that. We can't find one
study and say, ah, that proves that. That's not
how research works. Research does not prove or
debunk. And by the way, whole language has not
been debunked. Yes, there's been people that
say it's been debunked. In the name of almighty
research, I debunked you, B debunked you. It
hasn't been debunked just because people say
that. Okay? Well, I could debunk things. I could
debunk the theory of gravity. There it is. It's
been debunked. I debunked it. And here's my proof.
You can't just go around debunking things. Research
answers a specific research question. Research
proves or disproves a hypothesis. And we have
to be wary of research that doesn't have a specific
research question. or purpose, and that is true
of Louisa Moats's stuff. Can't use the word proven
with science. Observations are never perfect.
Interpretations are never perfect. Truth needs
a context and information is often complete.
Scientists rarely use that word. Here's an example.
Phonics instruction is good for increasing phonics
skills. Absolutely. You teach a bunch of phonics
and students are going to score higher on phonics
measures. Absolutely. Absolutely. Phonics instruction
has shown to be more effective in raising scores
on measures of phonics than no phonics instruction.
Absolutely. prove that all students need lots
of phonics instruction? Something can be technically
true and conveniently inaccurate at the same
time. This idea that phonics should be the center
of reading instruction, that is based on a lot
of I thinkisms. This does not mean all students
need heavy doses of phonics. Or students would
be better readers if they get more phonics. Or
more phonics is better than less phonics. Or
phonics is the key variable in learning to read.
Anything other than phonics instruction should
be excluded. And phonics should be taught at
all ages and levels. Phonic skills transfer to
authentic reading situations. These are all assumptions.
And these research is based on the assumption
that reading is sounding out words. Research
based on sounding out words focuses on sounding
out words and includes sounding out word measures.
If you ask sounder out questions, you use sounder
out measures, you collect sounder out data, you
are going to make sounder out conclusions. That's
what science of reading research is. Research
is not collecting a bunch of data and saying
stuff. Research must be based on a specific question
or hypothesis. When you read a research study
after the abstract, the first thing you should
look for is that research question. And research
always... focuses on a specific group at a specific
location doing a specific task. That's why we
can't use one study. For example, what works
with a first grade student in Blackwater, Arizona
struggling with word recognition may not work
with a fifth grader in Edina, Minnesota struggling
with comprehension. Human beings are not standardized
products. Readers are not standardized products.
Students are not standardized products. Again,
when you read the research, skip the literature,
look for the question. and these are the big
three. All right, I have seen that this is 60
minutes. I am going to stop my presentation here.
I will host this presentation on my sub stack,
the reading instruction show, and I'll post the
rest of this. Those who are still here, I thank
you for listening. I am Andy Johnson.