1
00:00:00,000 --> 00:00:04,800
This is the Reading Instruction Show. I'm your host as always, Dr. Andy Johnson.

2
00:00:04,800 --> 00:00:13,040
Title of today's podcast is called It's Research Based. Now, there's a mistaken

3
00:00:13,040 --> 00:00:20,240
assumption by some that if something is said to be research-based or evidence-based,

4
00:00:20,240 --> 00:00:26,240
it means that it's been proven to work. It will work and it should be used

5
00:00:26,240 --> 00:00:33,920
unequivocably. You just have to see the hyphenated word research-based in front

6
00:00:33,920 --> 00:00:39,760
of or behind a strategy approach or method and you're in. No questions asked. It's

7
00:00:39,760 --> 00:00:47,480
research-based. It proves your method works and all others are wrong. Glory

8
00:00:47,480 --> 00:00:53,240
hallelujah. Your strategies are the right ones to use and anybody who thinks

9
00:00:53,240 --> 00:01:01,480
otherwise is wrong wrong wrong. Oh, the joy of it. And if you say research shows

10
00:01:01,480 --> 00:01:07,640
in front of a declarative statement, you can say to the world with joyful glee

11
00:01:07,640 --> 00:01:14,080
and absolute certainty that the statement is right and all others are wrong

12
00:01:14,080 --> 00:01:22,000
because research proves it. And we should certainly believe your point of

13
00:01:22,000 --> 00:01:28,720
view because it's research-based. And by inference, any other point of view is

14
00:01:28,720 --> 00:01:36,400
unicorn-based or monkey-based or fairy tale-based or pure folly. Now, some believe

15
00:01:36,400 --> 00:01:43,440
that research is used to determine rightness and wrongness to find

16
00:01:43,440 --> 00:01:49,440
certainty in an uncertain world. Research is used to separate sheep from goats,

17
00:01:49,440 --> 00:01:57,120
chaff from wheat, saint from sinners, etc. etc. etc. A holy canon is then created

18
00:01:57,120 --> 00:02:03,480
based on chosen bits of research. The research is chosen to support a

19
00:02:03,480 --> 00:02:10,240
predetermined idea. This is our holy book, they say. The science of reading a holy

20
00:02:10,240 --> 00:02:17,720
book of research-based. It is inerrant, unfailing. Why do I know? Because it's

21
00:02:17,720 --> 00:02:23,360
research-based. We have research to prove that our science of reading a holy

22
00:02:23,360 --> 00:02:31,040
book of research-based is inerrant and unfailing divine word of science. It can

23
00:02:31,040 --> 00:02:37,840
be proven. For in chapter 2 verse 27 it says, quote,

24
00:02:37,840 --> 00:02:44,920
whereily, whereily unto you we speaketh thus. Those who embrace it, the science

25
00:02:44,920 --> 00:02:50,080
of reading a holy book of research-based, are right and all heathens are wrong.

26
00:02:50,080 --> 00:02:59,400
Unquote. Paid consultants then go out and evangelize. In their evangelical zeal,

27
00:02:59,400 --> 00:03:05,880
they demonize those who would dare challenge their thinking. They castigate

28
00:03:05,880 --> 00:03:09,880
those who would speak against the science of reading a holy book of

29
00:03:09,880 --> 00:03:17,400
research-based. Their views are the holy views handed down to them on high from

30
00:03:17,400 --> 00:03:25,200
the holy unseen in a mysterious science. Science is to be loved, feared, and

31
00:03:25,200 --> 00:03:31,920
obeyed. Will be it unto those who turn their back to science? The scientific

32
00:03:31,920 --> 00:03:38,320
commandments are to be followed without question because after all it's science

33
00:03:38,320 --> 00:03:45,400
and the science of reading. And how do I know? Because it's got science in the

34
00:03:45,400 --> 00:03:52,320
title and it says words like research and research-based a lot. Yes, they're

35
00:03:52,320 --> 00:03:59,560
zealots, but they're zealots for science. And by the way, did you know that you can

36
00:03:59,560 --> 00:04:07,200
have two different views on reading both views being research-based? So let's look

37
00:04:07,200 --> 00:04:13,920
at this for a minute, shall we? When you look at the research, if a claim is said

38
00:04:13,920 --> 00:04:20,560
to be research-based, you should be able to look at the research that supports the

39
00:04:20,560 --> 00:04:29,680
claim. That is after all what we do in academia. If a claim is made, the research

40
00:04:29,680 --> 00:04:35,680
behind it is cited so that you can look at the research behind the research-based

41
00:04:35,680 --> 00:04:43,680
claim and see if it's legitimate. And yes, pre-K-12 education is part of the

42
00:04:43,680 --> 00:04:52,800
academy. It's part of academia. And the currency used in academia is peer-reviewed

43
00:04:52,800 --> 00:05:01,160
research published in academic journals. Not I thinkisms or famous people or

44
00:05:01,160 --> 00:05:06,800
radio journalists or anecdotal evidence or personal experience or data pulled

45
00:05:06,800 --> 00:05:16,320
out of context or think tanks. In the academy, we use research and research is

46
00:05:16,320 --> 00:05:23,360
not research unless and until it has been subjected to blind peer review and

47
00:05:23,360 --> 00:05:29,400
published in an academic journal. Now, let's take a look at the importance of

48
00:05:29,400 --> 00:05:35,640
blind peer review. Why is the blind peer review process important? Why can't you

49
00:05:35,640 --> 00:05:41,680
just collect data and call it research? After all, that's what state

50
00:05:41,680 --> 00:05:49,560
legislators and journalists do. But blind peer review is used to make sure that

51
00:05:49,560 --> 00:05:54,640
you don't monkey with the statistics or your measures are valid, your data are

52
00:05:54,640 --> 00:06:00,240
accurate and your conclusions are actually based on the data collected, not

53
00:06:00,240 --> 00:06:06,720
what you want the data to say. Peer review is a subjectivity filter, a bias

54
00:06:06,720 --> 00:06:12,280
filter. It's a monkey business filter. And it's not that we don't trust

55
00:06:12,280 --> 00:06:19,760
researchers. Well, it is actually because we don't trust researchers. There's tenure

56
00:06:19,760 --> 00:06:23,640
and promotion to earn, you see, books to write, appearances to make, consulting

57
00:06:23,640 --> 00:06:29,040
fees to charge and interviews to give. So, it is common that there's

58
00:06:29,040 --> 00:06:35,480
electric desires can get in the way of seeing clearly. What they want to find

59
00:06:35,480 --> 00:06:44,280
can sometimes fog the lens of objectivity just a bit. And that's why we need a jury

60
00:06:44,280 --> 00:06:50,000
of our peers to check our work before it's published. Now, also the blind peer

61
00:06:50,000 --> 00:06:55,600
review process helps the authors of that study find mistake or analysis they

62
00:06:55,600 --> 00:07:01,080
hadn't considered. Now, I've been part of the blind review process and it can be

63
00:07:01,080 --> 00:07:08,360
brutal, but it always leads to a better product. Again, it is one subjectivity

64
00:07:08,360 --> 00:07:14,680
filter to ensure there's a certain amount of objectivity in the research.

65
00:07:14,680 --> 00:07:23,200
Good peer review makes it less likely that the research is not being used.

66
00:07:23,200 --> 00:07:28,400
Peer review makes it less likely that the research won't be used to sell a

67
00:07:28,400 --> 00:07:35,200
product or promote a point of view. Now, in blind peer review research, the

68
00:07:35,200 --> 00:07:40,240
reviewers evaluate the study without knowing who conducted it. Hence the

69
00:07:40,240 --> 00:07:47,920
term blind peer review. Reviewers consider such thing as the clarity of the

70
00:07:47,920 --> 00:07:53,360
research question, the theoretical context in which the research question was

71
00:07:53,360 --> 00:08:01,000
set, the adequacy of the methodology, the analysis of the data, the interpretation

72
00:08:01,000 --> 00:08:06,200
of the data, the validity of the conclusions, and the quality of the

73
00:08:06,200 --> 00:08:12,200
writing. Then the reviewers have four options. They can recommend the article

74
00:08:12,200 --> 00:08:17,520
for publication, recommend for publication with revisions, suggest

75
00:08:17,520 --> 00:08:23,760
specific regions be remade and resubmitted for consideration or recommend

76
00:08:23,760 --> 00:08:31,160
that the article be rejected. Academic journals, yes, they publish studies,

77
00:08:31,160 --> 00:08:37,920
research studies, but they also publish theoretical articles. Blind peer review is

78
00:08:37,920 --> 00:08:45,640
also used for theoretical articles. Now, let's understand that a theory is built

79
00:08:45,640 --> 00:08:53,440
on research. A theory is a dot-to-dot picture used to explain a set of facts

80
00:08:53,440 --> 00:09:01,760
and understand phenomenon. Each dot on this dot-to-dot theory picture is a

81
00:09:01,760 --> 00:09:10,400
research study. It's only by looking at many studies that the theory begins to

82
00:09:10,400 --> 00:09:16,240
emerge and you begin to see the whole, not one or two studies, but many. Now,

83
00:09:16,240 --> 00:09:22,520
theoretical articles are sometimes called secondary research. These are not a

84
00:09:22,520 --> 00:09:30,040
lesser form of scholarly work. Some, however, equate a theoretical article

85
00:09:30,040 --> 00:09:36,640
with opinion. The Educational Research Information Center, Eric, which is part

86
00:09:36,640 --> 00:09:40,680
of the Institute of Educational Science, which is part of the US Department of

87
00:09:40,680 --> 00:09:48,080
Education, they did this for a time. Anything that was not a controlled

88
00:09:48,080 --> 00:09:55,160
experimental study, anything that was not used to test a hypothesis, was called

89
00:09:55,160 --> 00:10:02,360
opinion, an opinion article. Now, here's my opinion on their opinion articles. What

90
00:10:02,360 --> 00:10:11,040
a bunch of clowns. They don't understand. They are ignorant. While an individual

91
00:10:11,040 --> 00:10:17,800
research study looks deeply at a part of a phenomenon, secondary research or

92
00:10:17,800 --> 00:10:22,720
theoretical articles takes a wide look, looking at many studies at that

93
00:10:22,720 --> 00:10:30,520
phenomenon. Both looks are necessary to understand clearly and move the field of

94
00:10:30,520 --> 00:10:38,280
education forward. One is not more important than the other. Original

95
00:10:38,280 --> 00:10:43,200
research is not more important than secondary research. They are both of

96
00:10:43,200 --> 00:10:50,440
value and importance. So let's get back to peer review. Peer review simply

97
00:10:50,440 --> 00:10:57,600
denotes a process, the process of peer review. The quality or rigor of this

98
00:10:57,600 --> 00:11:05,880
process varies. Reviewers and editors of highly prestigious academic journals use

99
00:11:05,880 --> 00:11:12,320
a process that is rigorous and very selective. These journals have low

100
00:11:12,320 --> 00:11:19,080
acceptance rates and tend to have considerable influence on the field. Other

101
00:11:19,080 --> 00:11:24,680
journals have a less rigorous review process and higher acceptance rates.

102
00:11:24,680 --> 00:11:32,600
However, all are still considered peer reviewed journals. And the peer review

103
00:11:32,600 --> 00:11:39,920
process is not without bias or flaws. Peer review does not magically make

104
00:11:39,920 --> 00:11:47,000
research unbiased or pure. It's not possible for human beings to have a

105
00:11:47,000 --> 00:11:55,440
completely objective unbiased view of anything. Peer review is simply one

106
00:11:55,440 --> 00:12:01,880
filter to try to remove some of the impurities related to bias, methodology,

107
00:12:01,880 --> 00:12:08,360
theoretical context, applications, and conclusions. One filter, but it is a

108
00:12:08,360 --> 00:12:16,280
necessary filter. So let's take a look at white papers. Some scholars put out

109
00:12:16,280 --> 00:12:23,080
what's called a white paper. On her letters website, Louisa Mote has a white

110
00:12:23,080 --> 00:12:27,480
paper. It's entitled Literacy Professional Learning, 10 reasons why it's

111
00:12:27,480 --> 00:12:35,080
essential. This white paper is used to prove to people that they should give her

112
00:12:35,080 --> 00:12:39,400
lots of money to get all professionally trained by her letters

113
00:12:39,400 --> 00:12:47,160
professional development. So what's a white paper? A white paper is a paper

114
00:12:47,160 --> 00:12:52,960
that looks like an article you might find in a peer reviewed academic journal. It

115
00:12:52,960 --> 00:12:59,320
has headings and subheadings. It uses objective academic languages, language,

116
00:12:59,320 --> 00:13:05,360
and citations to bolster the declarative statements made. The only difference

117
00:13:05,360 --> 00:13:10,480
between a white paper and an article appearing in a peer reviewed academic

118
00:13:10,480 --> 00:13:15,880
journal is that a white paper has not been peer reviewed and has not been

119
00:13:15,880 --> 00:13:22,560
published in an academic journal. In a white paper, people like Louisa Mote's

120
00:13:22,560 --> 00:13:27,800
can say whatever they want to say and there's no one to counter them or

121
00:13:27,800 --> 00:13:34,960
check their claims. And again, white papers look like articles found in peer

122
00:13:34,960 --> 00:13:41,960
reviewed academic journals. They look like them, but they're not. And people who

123
00:13:41,960 --> 00:13:48,840
don't know the difference reference them as if they were from peer reviewed

124
00:13:48,840 --> 00:13:54,800
academic journals. White papers are published on a lot of websites and put

125
00:13:54,800 --> 00:14:00,240
out by think tanks and political groups. White papers can be interesting. I write

126
00:14:00,240 --> 00:14:06,760
them myself, but they're white papers. They're not published in peer reviewed

127
00:14:06,760 --> 00:14:14,440
journals. And here's the thing. Louisa Mote's white paper was not submitted to

128
00:14:14,440 --> 00:14:20,680
a legitimate academic journal. And when you actually look at the sources she

129
00:14:20,680 --> 00:14:27,800
cites, when you try to be a responsible consumer of educational research, you see

130
00:14:27,800 --> 00:14:34,600
that her claims have nothing whatsoever to do with the research being cited. When

131
00:14:34,600 --> 00:14:40,600
you actually look at her citations and the studies she cites, you see that she

132
00:14:40,600 --> 00:14:47,480
misinterprets, misuses, and falsifies research studies to support her claims.

133
00:14:47,480 --> 00:14:52,360
She also cites poorly conducted research that has little to do with the point

134
00:14:52,360 --> 00:14:58,280
she's trying to make. And you would know this if you didn't critically read and

135
00:14:58,280 --> 00:15:05,360
analyze her cited sources. This is what a responsible consumer of educational

136
00:15:05,360 --> 00:15:12,320
research does whenever confronted with the phrase research base or research. We

137
00:15:12,320 --> 00:15:21,200
never take anything at face value. We have a natural inclination to doubt. So a

138
00:15:21,200 --> 00:15:26,000
little bit more about peer reviewed academic journals. A peer reviewed

139
00:15:26,000 --> 00:15:33,080
academic journal is different from a magazine. Magazines and newspapers, they

140
00:15:33,080 --> 00:15:40,520
just send a reporter out to cover things. And reporters just write stuff. And by

141
00:15:40,520 --> 00:15:46,840
the way, Emily Hanford is a reporter. And she's made quite a name for herself in

142
00:15:46,840 --> 00:15:53,480
the area of reading instruction. But she's a journalist, a radio

143
00:15:53,480 --> 00:15:59,480
documentarian. She's not an expert in reading. She's never taught a reading

144
00:15:59,480 --> 00:16:06,360
class. And I wonder if she has ever read a research article related to

145
00:16:06,360 --> 00:16:13,960
reading. Reporters and radio documentarians are not experts in their fields.

146
00:16:13,960 --> 00:16:20,640
Their work is not peer reviewed. They don't have expertise. They don't know

147
00:16:20,640 --> 00:16:26,840
what questions to ask or what data to look for. They don't know which sources

148
00:16:26,840 --> 00:16:30,600
are valid and which are not. They don't cite their sources. They're not

149
00:16:30,600 --> 00:16:37,200
responsible consumers of educational research. Reporters and radio

150
00:16:37,200 --> 00:16:42,000
documentarians are important. But at the end of the day, they're just people who

151
00:16:42,000 --> 00:16:50,400
write stuff. Unlike magazines, academic journals carry no advertisements. They

152
00:16:50,400 --> 00:16:55,400
don't want to be swayed or give the appearance of being swayed by advertising

153
00:16:55,400 --> 00:17:03,480
revenue. Academic journals are all written by experts in their fields and not

154
00:17:03,480 --> 00:17:10,240
by journalists or radio reporters. As well, all articles are reviewed by a jury

155
00:17:10,240 --> 00:17:16,760
of two or more people who are experts in their field related to the topic of the

156
00:17:16,760 --> 00:17:23,400
article. If accepted, then it's published. That's the difference between a magazine

157
00:17:23,400 --> 00:17:32,240
and an academic journal or a newspaper and an academic journal. Now, this idea

158
00:17:32,240 --> 00:17:38,000
that research shows something's effective. The mistaken assumption based on a

159
00:17:38,000 --> 00:17:44,200
lack of understanding about the research process and educational research is that

160
00:17:44,200 --> 00:17:49,920
if a strategy has been shown to be effective with a certain population for

161
00:17:49,920 --> 00:17:55,240
a certain purpose, for a certain time, and at a certain level, well, then it must

162
00:17:55,240 --> 00:18:01,040
certainly be effective for all students, for all purposes at all times. It's

163
00:18:01,040 --> 00:18:05,240
research-based. They happily declare as they march around the room with high

164
00:18:05,240 --> 00:18:11,480
needs, research-based, research-based. We are so wonderful, research-based. But this

165
00:18:11,480 --> 00:18:18,000
is clownery. When the statement is made that research has shown a practice

166
00:18:18,000 --> 00:18:25,280
method or strategy to be effective, one must always ask effective for who, for

167
00:18:25,280 --> 00:18:31,960
what purpose, how, at what level, for how long. Now, a case in point, John Hattie

168
00:18:31,960 --> 00:18:38,360
wrote a very popular book, Visible Learning. In one chapter, he compares whole

169
00:18:38,360 --> 00:18:45,080
language to direct instruction and he uses a meta-analysis. A meta-analysis is

170
00:18:45,080 --> 00:18:50,000
when you put a whole bunch of studies that look at a similar thing into a

171
00:18:50,000 --> 00:18:56,640
mathematical formula to answer a research question. It's like research that uses

172
00:18:56,640 --> 00:19:02,680
research as data, looks at a whole bunch of studies to answer a question. Now,

173
00:19:02,680 --> 00:19:10,240
after his meta-analysis on direct instruction and whole language, Hattie

174
00:19:10,240 --> 00:19:16,440
comes to the conclusion that direct instruction is superior to whole language.

175
00:19:16,440 --> 00:19:22,440
Imagine that. I guess then that this proves that whole language has been

176
00:19:22,440 --> 00:19:27,920
debunked. And I guess Andy has been wrong for 30 years. And thank goodness for John

177
00:19:27,920 --> 00:19:34,360
Hattie. Maybe that over-caffeinated little bald man will finally shut up. This, by

178
00:19:34,360 --> 00:19:41,080
the way, points to the severe limitations of people outside the field of literacy

179
00:19:41,080 --> 00:19:49,600
trying to do literacy research. John Hattie may be a meta-analysis expert, but

180
00:19:49,600 --> 00:19:57,600
he is far from a literacy expert. In asking his meta-analysis questions, he

181
00:19:57,600 --> 00:20:02,880
has no understanding of the context. He doesn't know what questions to ask, what

182
00:20:02,880 --> 00:20:08,480
data to look for, what measures to use to adequately assess the data or the

183
00:20:08,480 --> 00:20:15,520
research studies. And research, like John Hattie's, based on a false assumption,

184
00:20:15,520 --> 00:20:22,360
always leads to invalid conclusions. Again, research based on false assumptions

185
00:20:22,360 --> 00:20:29,040
always lead to invalid conclusions. Hattie's meta-analysis is based on a

186
00:20:29,040 --> 00:20:34,400
false based on false assumptions. It's based on ignorance. Now, let me explain.

187
00:20:34,400 --> 00:20:40,120
Direct instruction is a pedagogical strategy that has been shown to be

188
00:20:40,120 --> 00:20:47,480
effective for teaching low level skills. Yes, all good teachers understand and use

189
00:20:47,480 --> 00:20:54,000
direct instruction. And yes, there's research to support its use for teaching

190
00:20:54,000 --> 00:21:01,120
some low level skills. It is effective for some purposes with some students when

191
00:21:01,120 --> 00:21:08,080
used correctly. And every teacher should know how and when to use direct

192
00:21:08,080 --> 00:21:14,160
instruction. And this doesn't mean that direct instruction is good for all

193
00:21:14,160 --> 00:21:19,480
students for all purposes, for all times, and that it should become a complete

194
00:21:19,480 --> 00:21:26,560
reading program. It's not direct instruction or whole language. And this is

195
00:21:26,560 --> 00:21:32,800
where ignorance really shows. Because whole language teachers use direct

196
00:21:32,800 --> 00:21:41,400
instruction. It's a pedagogical strategy. Whole language teachers use direct

197
00:21:41,400 --> 00:21:48,400
instruction. So research based on a false assumption always leads to invalid

198
00:21:48,400 --> 00:21:54,400
conclusions. And this shows the extreme limitation of John Hattie's work. So three

199
00:21:54,400 --> 00:21:59,560
big points here to end up to conclude point number one. Teachers and

200
00:21:59,560 --> 00:22:05,080
administrators must become informed and responsible consumers of

201
00:22:05,080 --> 00:22:11,000
educational research. When someone says research shows this and research proves

202
00:22:11,000 --> 00:22:18,000
that, we must always, always ask what research. Give me a citation. Let me look

203
00:22:18,000 --> 00:22:23,120
at it. Effective for what? For who? At what level? For what purpose? How? And then

204
00:22:23,120 --> 00:22:27,240
we must actually go and look at the research study because I know it's hard

205
00:22:27,240 --> 00:22:33,880
to believe, but some people are a little weasley. Point number two. State and

206
00:22:33,880 --> 00:22:39,240
federal legislators. Those who want to impose mandates on teachers telling them

207
00:22:39,240 --> 00:22:44,720
what and how to teach. And I'm going to say this and I mean this in the most

208
00:22:44,720 --> 00:22:50,680
respectful way possible, but pull your heads out of your collective asses. Do

209
00:22:50,680 --> 00:22:57,520
what's right, not what's politically expedient. If you're not going to review

210
00:22:57,520 --> 00:23:01,960
the research related to reading instruction, for God's sakes, at least

211
00:23:01,960 --> 00:23:09,440
call in those who have. Stop making educational decisions based on the

212
00:23:09,440 --> 00:23:18,360
testimony of radio journalists. And point number three. We all want students to

213
00:23:18,360 --> 00:23:24,720
achieve their full literacy potential. Science of reading

214
00:23:24,720 --> 00:23:29,760
ideologists. They want this thing. I know that's your intent and it's a good

215
00:23:29,760 --> 00:23:39,440
intent, but intent does not mitigate impact in five years when the full

216
00:23:39,440 --> 00:23:45,600
impact of what you've done will be realized. And people see just how much

217
00:23:45,600 --> 00:23:52,440
time, money and resources you've wasted and how many teachers you've run out of

218
00:23:52,440 --> 00:23:58,400
the classroom and why we have a shortage of teachers and not enough money to pay

219
00:23:58,400 --> 00:24:04,200
for smaller class sizes. I'll be doing another podcast five years from now and

220
00:24:04,200 --> 00:24:10,400
this one will be called I told you so. This has been the Reading Instruction

221
00:24:10,400 --> 00:24:30,400
Show. I'm your host as always, Dr. Andy Johnson.