WEBVTT 00:00:02.060 --> 00:00:04.960 Greetings, I'm Andy Johnson. This presentation 00:00:04.960 --> 00:00:11.140 is called Understanding Evidence and Research. 00:00:11.480 --> 00:00:14.119 I have a lot of notes and handouts and other 00:00:14.119 --> 00:00:18.100 things related to this at my free sub stack. 00:00:18.460 --> 00:00:20.760 Plug in the reading instruction show and you 00:00:20.760 --> 00:00:23.679 will find it there. It's free, there's no subscription 00:00:23.679 --> 00:00:27.260 charge. Also, a lot of people have asked me about 00:00:27.260 --> 00:00:30.300 teaching writing and writing instruction is one 00:00:30.300 --> 00:00:33.810 of the most under instructed skills in our schools. 00:00:34.049 --> 00:00:37.250 Again, another free substack, how to teach writing, 00:00:37.609 --> 00:00:40.549 you can find it there. All the stuff there is 00:00:40.549 --> 00:00:43.390 based on my book. And of course, like the rest 00:00:43.390 --> 00:00:48.329 of 90 % of America, I have a podcast with tens 00:00:48.329 --> 00:00:54.859 of listeners. Okay, let's start. Context. matters 00:00:54.859 --> 00:00:57.340 to understand things, to understand research, 00:00:57.520 --> 00:01:02.479 to understand data, to understand evidence. Context 00:01:02.479 --> 00:01:05.439 matters. You can't pull things out of context 00:01:05.439 --> 00:01:09.439 and try to understand. So let us look at the 00:01:09.439 --> 00:01:12.780 big picture. The big picture is to enable all 00:01:12.780 --> 00:01:16.569 children to develop their full literacy potential 00:01:16.569 --> 00:01:21.590 so that they can use literacy for real life purposes 00:01:21.590 --> 00:01:26.209 in the real world, not to pass a unit test or 00:01:26.209 --> 00:01:30.730 to get higher scores on this and that. So to 00:01:30.730 --> 00:01:35.489 use literacy to achieve real life goals. And 00:01:35.489 --> 00:01:38.010 literacy includes reading and writing, but also 00:01:38.010 --> 00:01:42.829 speaking, listening, and thinking. So I ask unto 00:01:42.829 --> 00:01:48.310 you, what constitutes evidence of this? Beware 00:01:48.310 --> 00:01:51.590 of the number monkeys who try to describe learning 00:01:51.590 --> 00:01:56.450 only in terms of numbers. If it can't be quantified, 00:01:56.609 --> 00:02:01.049 they tell us it doesn't exist. Reading is not 00:02:01.049 --> 00:02:03.829 sounding out words. It's creating meaning with 00:02:03.829 --> 00:02:08.039 print. And we read for two purposes. We read 00:02:08.039 --> 00:02:12.360 to understand information, and we read to enjoy 00:02:12.360 --> 00:02:18.580 books. That is the end state, the goal. All our 00:02:18.580 --> 00:02:22.840 objectives should lead to that goal. Everything 00:02:22.840 --> 00:02:26.719 we do to teach reading should lead to those two 00:02:26.719 --> 00:02:31.240 goals. And the ability to do these things should 00:02:31.240 --> 00:02:37.289 define students' reading ability. So what is 00:02:37.289 --> 00:02:40.050 evidence of a literacy crisis? We're talking 00:02:40.050 --> 00:02:44.550 about evidence. The crisis, a crisis, any crisis, 00:02:44.870 --> 00:02:47.770 is a time of intense difficulty, trouble, or 00:02:47.770 --> 00:02:51.969 danger. And some insist we have this thing called 00:02:51.969 --> 00:02:56.289 a reading crisis. You can read the media and 00:02:56.289 --> 00:02:59.569 literacy. Thousands of children functionally 00:02:59.569 --> 00:03:03.150 illiterate. Oh, really? Just because you see 00:03:03.150 --> 00:03:06.639 it or hear it doesn't mean it's true. Teachers 00:03:06.639 --> 00:03:09.580 criticize their colleges of ed for not preparing 00:03:09.580 --> 00:03:13.400 them to teach reading. How many teachers? Two 00:03:13.400 --> 00:03:16.900 teachers? Four teachers? How old were they? First 00:03:16.900 --> 00:03:20.740 year? 20th year? 50th year? How many colleges 00:03:20.740 --> 00:03:26.139 of ed? We must look for evidence as we consume 00:03:26.139 --> 00:03:30.400 the media, as we consume various forms of media. 00:03:31.240 --> 00:03:35.949 It's alarming, they tell us. Children are severely 00:03:35.949 --> 00:03:39.889 behind, teachers say. Behind what? Behind who? 00:03:40.490 --> 00:03:43.530 What educators are saying what? Just because 00:03:43.530 --> 00:03:50.590 is it two educators saying it? Journalists, journalists 00:03:50.590 --> 00:03:52.770 are important in a democratic society. I'm not 00:03:52.770 --> 00:03:56.669 getting down on them, but journalists write stories. 00:03:57.590 --> 00:04:02.110 Stories are not evidence. Stories are things 00:04:02.110 --> 00:04:06.479 that people put together. not necessarily for 00:04:06.479 --> 00:04:11.479 accuracy, but for impact, to get clicks on their 00:04:11.479 --> 00:04:15.319 little websites, to get awards at banquets. That's 00:04:15.319 --> 00:04:19.199 what journalists do. There is not legitimate 00:04:19.199 --> 00:04:24.480 evidence that there is a literacy crisis. Evidence, 00:04:24.740 --> 00:04:28.519 legitimate. Paul Thomas talks about it and has 00:04:28.519 --> 00:04:32.509 written about all the various literacy crises 00:04:32.509 --> 00:04:36.209 starting in the early 50s. And there'll be another 00:04:36.209 --> 00:04:39.670 one coming around in 10 to 15 years when this 00:04:39.670 --> 00:04:42.829 one has found to be another boondockle or when 00:04:42.829 --> 00:04:47.089 some publisher needs something else to be sold. 00:04:47.769 --> 00:04:51.350 There will be another one when they find this 00:04:51.350 --> 00:04:56.779 science of reading thing doesn't work. Why? Because 00:04:56.779 --> 00:05:00.740 it's much easier to blame schools and teachers 00:05:00.740 --> 00:05:06.100 and parents than to address social issues, systemic 00:05:06.100 --> 00:05:11.680 poverty, economic inequality, overcrowded schools, 00:05:13.120 --> 00:05:16.839 income inequality, lack of school funding. These 00:05:16.839 --> 00:05:21.740 things impact learning. But it's easier to say 00:05:21.740 --> 00:05:24.360 schools are failing, teachers are doing a bad 00:05:24.360 --> 00:05:30.149 job. Everybody who's been to first grade seems 00:05:30.149 --> 00:05:35.569 to have all the answers to fix public education. 00:05:35.970 --> 00:05:39.389 Teach more phonics, George Bush told us once. 00:05:40.050 --> 00:05:43.410 Higher standards, get tough. Get rid of teachers 00:05:43.410 --> 00:05:48.930 unions. Reward the good teachers. More tests. 00:05:49.170 --> 00:05:52.230 Reward high performing schools. Interesting. 00:05:52.970 --> 00:05:57.629 Simple, simple answers. to complex problems. 00:05:58.509 --> 00:06:02.509 Blaming teachers takes people's attention from 00:06:02.509 --> 00:06:07.350 the real issues and teachers can't fight back. 00:06:09.389 --> 00:06:14.029 This has led to a teacher shortage. Teacher turnover 00:06:14.029 --> 00:06:19.110 is the leading factor in teacher shortages. They 00:06:19.110 --> 00:06:23.300 are leaving. Qualified teachers are leaving teachers 00:06:23.300 --> 00:06:26.920 who love their job, good teachers, for reasons 00:06:26.920 --> 00:06:32.139 other than retirement. There's no shortage of 00:06:32.139 --> 00:06:35.199 teachers. There's a shortage of teachers who 00:06:35.199 --> 00:06:42.240 are willing to put up with baloney. As an example, 00:06:42.699 --> 00:06:46.540 I am retiring in three months and one of the 00:06:46.540 --> 00:06:50.079 reasons I'm retiring is because I would be required 00:06:50.240 --> 00:06:54.639 to go through letters training, what I call re 00:06:54.639 --> 00:06:58.800 -education camp. Just like in a clockwork orange, 00:06:58.980 --> 00:07:01.899 they force that guy's eyes open to watch those 00:07:01.899 --> 00:07:05.639 scenes. I would be forced to watch letters or 00:07:05.639 --> 00:07:09.959 to take letters training. There is not a reading 00:07:09.959 --> 00:07:14.180 crisis, a reading achievement crisis. Now we 00:07:14.180 --> 00:07:18.730 are going to look at data from the National Assessment 00:07:18.730 --> 00:07:21.589 of Educational Progress. This is part of the 00:07:21.589 --> 00:07:24.850 U .S. Department of Education. They take a large 00:07:24.850 --> 00:07:28.250 sample, a representative sample. They give tests 00:07:28.250 --> 00:07:30.870 to students in grades four, eight, and 12 every 00:07:30.870 --> 00:07:34.790 year. It's called the nation's report card. It 00:07:34.790 --> 00:07:40.129 gives us a sense of how we are doing. Rikin, 00:07:40.209 --> 00:07:44.100 Ruby, and Risco. There's no irrefutable evidence 00:07:44.100 --> 00:07:47.259 that we are experiencing a national literacy 00:07:47.259 --> 00:07:52.480 crisis. Crisis would look something like that 00:07:52.480 --> 00:07:58.480 or something like that. It would not look like 00:07:58.480 --> 00:08:05.660 that. These are long -term reading scores in 00:08:05.660 --> 00:08:09.139 fourth grade and in eighth grade, starting in 00:08:09.139 --> 00:08:12.629 71. You can see there's just a gradual release 00:08:12.629 --> 00:08:15.689 of scores, but there's a natural fluctuation, 00:08:15.689 --> 00:08:21.649 and that's called normal, normal. This is the 00:08:21.649 --> 00:08:24.329 latest one I found a couple of days ago on the 00:08:24.329 --> 00:08:27.810 website, and it's hard to find this data. They've 00:08:27.810 --> 00:08:30.149 included a whole bunch of stuff that makes it 00:08:30.149 --> 00:08:33.649 difficult. No statistically significant difference 00:08:33.649 --> 00:08:39.570 from 2013 to 2020. Of course, we had the COVID 00:08:39.570 --> 00:08:44.090 dip. didn't we? And before the dip, we were higher 00:08:44.090 --> 00:08:48.710 than we were statistically significantly higher 00:08:48.710 --> 00:08:57.309 than we were in 92. Same with eighth grade. Here 00:08:57.309 --> 00:09:01.889 is the COVID dip and we expected that. But from 00:09:01.889 --> 00:09:07.519 12 to 20, no significant dip. Reading scores 00:09:07.519 --> 00:09:12.100 fall to new lows, says Sarah Schwartz from Education 00:09:12.100 --> 00:09:15.659 Week. And she's a reporter. She just writes a 00:09:15.659 --> 00:09:19.840 lot of stuff. She is a reporter. She's not an 00:09:19.840 --> 00:09:23.620 academician or a scholarly writer. But here's 00:09:23.620 --> 00:09:26.960 the thing, even if she weren't lying, think about 00:09:26.960 --> 00:09:30.740 it. This is the dates when all the states have 00:09:30.740 --> 00:09:36.039 passed their miracle reading legislation. Mississippi 00:09:36.039 --> 00:09:39.779 in 2013. Here's when the miracle started. And 00:09:39.779 --> 00:09:43.159 look at all these other things, all these other 00:09:43.159 --> 00:09:47.740 miracles starting 2009, 21, 22. Here's Minnesota 00:09:47.740 --> 00:09:52.960 right there in 2023. So let the miracles occur. 00:09:53.179 --> 00:09:56.679 We should just be sitting back enjoying our cup 00:09:56.679 --> 00:09:59.820 of coffee and just watching the miracles unfold 00:09:59.820 --> 00:10:06.240 in front of us. Dramatic pause. except they are 00:10:06.240 --> 00:10:10.200 not. But even if there were a crisis, which there 00:10:10.200 --> 00:10:16.639 is not, the science of reading a fellow's people 00:10:16.639 --> 00:10:21.580 have determined, have claimed that they've isolated 00:10:21.580 --> 00:10:25.379 the causal variable. They say that they know 00:10:25.379 --> 00:10:28.700 both the cause and the solution to a problem 00:10:28.700 --> 00:10:32.600 that does not exist. But let me tell you, my 00:10:32.600 --> 00:10:38.429 friends, Randomly determining causality is not 00:10:38.429 --> 00:10:44.090 science. That's baloney, sometimes called pseudoscience. 00:10:44.490 --> 00:10:49.309 These are all the variables that impact student 00:10:49.309 --> 00:10:52.870 achievement. And you can pause and look at it 00:10:52.870 --> 00:10:56.169 yourself for a good time on a Saturday afternoon. 00:10:57.120 --> 00:11:00.720 Again, we must distinguish between scholarly 00:11:00.720 --> 00:11:06.220 articles, scholarship, and journalists. Journalists 00:11:06.220 --> 00:11:09.799 write down what people tell them. People tell 00:11:09.799 --> 00:11:12.679 them things, they write it down, and they create 00:11:12.679 --> 00:11:16.360 their stories. And everyone wants to be the next 00:11:16.360 --> 00:11:19.220 Emily Hanford. Look how famous she is and how 00:11:19.220 --> 00:11:22.039 much money she's getting and how many speaking 00:11:22.039 --> 00:11:26.830 gigs. Oh, I want to be just like Emily. publish 00:11:26.830 --> 00:11:30.990 stuff. They put numbers to it. They have sensational 00:11:30.990 --> 00:11:35.230 things. They always have a picture of a kid who's 00:11:35.230 --> 00:11:38.090 crying or looking distraught. And look at this 00:11:38.090 --> 00:11:41.169 graph. Oh, that's just horrible. That's horrible. 00:11:41.710 --> 00:11:45.730 Pulling things out of context, putting them in 00:11:45.730 --> 00:11:50.149 ways that create a sensation. It's a straw man 00:11:50.149 --> 00:11:53.789 argument. This is when you create a cartoonish 00:11:53.789 --> 00:11:57.610 version of something and argue against the cartoon. 00:11:58.230 --> 00:12:01.629 Things like balanced literacy. People describe 00:12:01.629 --> 00:12:05.309 it in ways that it is simply not true. Whole 00:12:05.309 --> 00:12:08.710 language, the three QE systems. Some people are 00:12:08.710 --> 00:12:12.549 just horribly upset about what they call Q -ing, 00:12:13.110 --> 00:12:14.990 and they don't even know what it is. They're 00:12:14.990 --> 00:12:18.279 describing something horrible. Public schools, 00:12:18.799 --> 00:12:23.159 DEI has been made a swearing word. Critical race 00:12:23.159 --> 00:12:26.220 theory, same thing by people that have not taken 00:12:26.220 --> 00:12:31.559 the time to understand what it is. I had this 00:12:31.559 --> 00:12:34.659 fellow on my show. I was interviewing him. He 00:12:34.659 --> 00:12:38.539 asked me to, so I did. He had a book which had 00:12:38.539 --> 00:12:41.340 a very interesting premise. The book was called 00:12:41.340 --> 00:12:44.559 Failure Factor, and in this book, he described 00:12:44.559 --> 00:12:49.210 administration. as changing teachers' grades 00:12:49.210 --> 00:12:53.169 to show that they were doing a good job. And 00:12:53.169 --> 00:12:55.950 in Baltimore, apparently, instead of a superintendent, 00:12:56.049 --> 00:13:00.590 they have a CEO, hence the factory motif. But 00:13:00.590 --> 00:13:04.370 Brother Chris here has some very simplistic answers 00:13:04.370 --> 00:13:08.649 to a very complex situation. And he either misunderstood 00:13:08.649 --> 00:13:14.360 or misrepresented information. His answer, of 00:13:14.360 --> 00:13:17.440 course, was just to hold students to higher standards. 00:13:17.539 --> 00:13:20.220 Oh, okay. Just hold them to higher standards. 00:13:20.360 --> 00:13:23.679 Duh. Why didn't I think of that? Based on a lot 00:13:23.679 --> 00:13:27.460 of I -thinkisms. Now notice, here's some of the 00:13:27.460 --> 00:13:30.940 stuff from his book indicating failure. He says, 00:13:31.059 --> 00:13:35.539 well, the budget has stayed consistent, all right, 00:13:35.759 --> 00:13:39.000 and the number of enrollment has gone down significantly. 00:13:39.419 --> 00:13:43.019 Look at all these things. But none of these things 00:13:43.019 --> 00:13:46.240 are indicative of the quality of education occurring 00:13:46.240 --> 00:13:51.419 in classrooms. SAT, that's a self -selected test 00:13:51.419 --> 00:13:54.919 that people going to college take. That's not 00:13:54.919 --> 00:13:58.539 a representation of how schools are doing. AP, 00:13:58.779 --> 00:14:01.700 advanced placement exams, again, a representative 00:14:01.700 --> 00:14:05.320 sample. Attendance, you can have quality teaching 00:14:05.320 --> 00:14:09.139 going on, but that does not impact attendance 00:14:09.139 --> 00:14:14.659 or dropout. or absenteeism, or enrollment. Some 00:14:14.659 --> 00:14:19.679 of these are societal factors. So either he doesn't 00:14:19.679 --> 00:14:22.659 know, he doesn't understand, or he's purposely 00:14:22.659 --> 00:14:25.500 misrepresenting to get his little statues and 00:14:25.500 --> 00:14:30.019 to get book sales. Again, he said, more discipline. 00:14:30.580 --> 00:14:33.679 Hold students to higher standards. Get rid of 00:14:33.679 --> 00:14:37.179 teachers unions. I hear that a lot. And in my 00:14:37.179 --> 00:14:39.940 book, I will be talking about teachers unions 00:14:39.940 --> 00:14:43.980 at some point, not today. Here's the thing I 00:14:43.980 --> 00:14:46.700 read in his book that made me want to throw up 00:14:46.700 --> 00:14:50.620 in my mouth. When a reporter focuses on one beat, 00:14:50.759 --> 00:14:54.220 they become an expert on that beat. Oftentimes 00:14:54.220 --> 00:14:57.559 a beach reporter knows the subject matter better 00:14:57.559 --> 00:15:01.039 than the industry professional they cover, unquote. 00:15:03.759 --> 00:15:06.600 What do you say to something like that? You don't. 00:15:06.620 --> 00:15:09.899 You just vomit in your mouth a little bit. They 00:15:09.899 --> 00:15:13.940 write stories. This is a story written like a 00:15:13.940 --> 00:15:17.799 detective novel. Watch the City has fallen into 00:15:17.799 --> 00:15:20.779 a state of decay. And that was certainly my view 00:15:20.779 --> 00:15:25.419 one September afternoon in 2017, as my photographer 00:15:25.419 --> 00:15:29.559 and I were. You get the idea. This isn't scholarly 00:15:29.559 --> 00:15:32.500 writing, not that you need scholarly writing, 00:15:32.759 --> 00:15:36.750 but he's creating a story for impact. That's 00:15:36.750 --> 00:15:39.210 what they do in television. That's how you get 00:15:39.210 --> 00:15:42.909 those little statues at award ceremonies. Everyone 00:15:42.909 --> 00:15:47.830 wants a statue. I want a statue. I welcome journalists 00:15:47.830 --> 00:15:51.549 into our academic conversations. I welcome you, 00:15:51.669 --> 00:15:54.929 but you do not get a free pass on ignorance. 00:15:55.970 --> 00:16:00.230 In my conversation with Chris and John Stossel, 00:16:00.429 --> 00:16:03.690 I understood that you have to have some background 00:16:03.690 --> 00:16:06.629 knowledge to understand and interpret what you 00:16:06.629 --> 00:16:12.070 see and hear. I often had to correct the presumption 00:16:12.070 --> 00:16:15.450 upon which questions were based before I answered 00:16:15.450 --> 00:16:19.769 questions. They had a shallow and disjointed 00:16:19.769 --> 00:16:22.610 knowledge base. There's nothing wrong with that. 00:16:23.389 --> 00:16:27.350 But you don't realize that. It makes it difficult 00:16:27.350 --> 00:16:30.320 to engage in academic discussion. discourse with 00:16:30.320 --> 00:16:33.480 these people, these people, with a journalist 00:16:33.480 --> 00:16:36.559 sometimes. I don't want to generalize. Reading 00:16:36.559 --> 00:16:40.600 instruction is first and formally an academic 00:16:40.600 --> 00:16:44.080 topic, and it has been, discussion has been going 00:16:44.080 --> 00:16:47.779 on in academic, academia, academic discussions 00:16:47.779 --> 00:16:52.340 for decades. And there's always been people with 00:16:52.340 --> 00:16:55.980 differing ideas. There isn't a reading war. And 00:16:55.980 --> 00:17:00.590 again, You are welcome as anybody is into our 00:17:00.590 --> 00:17:04.589 academic discussion. Anybody can publish in an 00:17:04.589 --> 00:17:07.470 academic journal. It's subjected to blind peer 00:17:07.470 --> 00:17:11.170 review. Anybody could speak at a conference, 00:17:11.730 --> 00:17:14.869 but allow for questions, get alternative views. 00:17:14.990 --> 00:17:19.509 When you say research, cite the research. You 00:17:19.509 --> 00:17:22.769 should not get a pass on ignorance. And that 00:17:22.769 --> 00:17:26.390 is happening. As I mentioned, I was interviewed 00:17:26.390 --> 00:17:29.210 by John Stossel. The interview took about 60 00:17:29.210 --> 00:17:33.789 minutes. He put together a story and my little 00:17:33.789 --> 00:17:36.410 part, I'm guessing was two to four minutes at 00:17:36.410 --> 00:17:39.549 most. And he created a collage with it to create 00:17:39.549 --> 00:17:43.369 the story that he wanted. This is an email I 00:17:43.369 --> 00:17:46.750 got from a producer. We're doing a story on the 00:17:46.750 --> 00:17:49.769 literacy crisis in America. of which there is 00:17:49.769 --> 00:17:53.309 not, I came across your advocacy of whole word 00:17:53.309 --> 00:17:56.750 reading instruction. Where did you come up with 00:17:56.750 --> 00:18:00.650 that? I don't know. Do you know what whole word 00:18:00.650 --> 00:18:03.710 reading instruction is? I've talked about whole 00:18:03.710 --> 00:18:06.630 language, but that's different from whole word. 00:18:07.190 --> 00:18:09.910 But anyway, he said, we would like to get your 00:18:09.910 --> 00:18:13.410 take on the subject. They would like to get my 00:18:13.410 --> 00:18:17.490 take. Imagine that. They didn't want my take 00:18:17.490 --> 00:18:20.940 on anything. They wanted to create a collage. 00:18:21.500 --> 00:18:24.700 Now, the speaking fee for John Stossel, for him 00:18:24.700 --> 00:18:29.400 to do a keynote is $75 ,000. I know because I 00:18:29.400 --> 00:18:31.599 pretended I was someone else and I contacted 00:18:31.599 --> 00:18:34.700 his speaking bureau. Am I a bad man for doing 00:18:34.700 --> 00:18:38.900 that? The speaking fee for me to do a keynote 00:18:38.900 --> 00:18:41.359 is give me a little flattery and a hamburger 00:18:41.359 --> 00:18:46.890 and I will be there. Now, in 1992, somebody determined 00:18:46.890 --> 00:18:49.569 that the NAAP reading scores weren't low enough, 00:18:49.569 --> 00:18:53.549 so we need to invent categories, advanced, proficient, 00:18:53.730 --> 00:18:58.289 and basic. Well, what is basic? We just assume 00:18:58.289 --> 00:19:01.569 that it's something. What is proficient? We just 00:19:01.569 --> 00:19:06.769 assume that it's something. What's average? This 00:19:06.769 --> 00:19:10.170 is what Rinking Ruby and Risco said. The cut 00:19:10.170 --> 00:19:14.799 scores used to define these categories were arbitrarily 00:19:14.799 --> 00:19:20.019 set by a small group of hand -picked politically 00:19:20.019 --> 00:19:24.160 appointed experts against the recommendations 00:19:24.160 --> 00:19:27.619 of the psychometricians who have been contracted 00:19:27.619 --> 00:19:32.420 to advise the development of this approach. You 00:19:32.420 --> 00:19:37.359 hired people? Interesting. Okay, interesting. 00:19:38.599 --> 00:19:42.140 Categories and levels and content are arbitrarily 00:19:42.140 --> 00:19:46.480 defined. citing some more sources there. Now, 00:19:46.480 --> 00:19:50.279 if you go on the website, US Department of Education, 00:19:50.380 --> 00:19:54.319 they report in percentile rankings, 10th, 25th, 00:19:54.359 --> 00:19:58.960 50th, 75th, and they have so much data and little 00:19:58.960 --> 00:20:02.519 data bits, it's designed more to confuse, and 00:20:02.519 --> 00:20:05.700 you can pull out little data bits to create a 00:20:05.700 --> 00:20:08.579 variety of stories, a variety of impressions. 00:20:09.420 --> 00:20:13.609 If you do this, pull it out of context. context 00:20:13.609 --> 00:20:17.549 as needed. This is average right here, so let 00:20:17.549 --> 00:20:23.930 me explain. 1992, this is average, so they decided, 00:20:23.950 --> 00:20:28.490 hmm, I think we'll make basic a little below 00:20:28.490 --> 00:20:33.529 average and proficient a little above average. 00:20:34.029 --> 00:20:37.009 What would that look like in a normal population? 00:20:37.329 --> 00:20:40.190 In a normal population, we have the bell -shaped 00:20:40.190 --> 00:20:45.089 curve where 50 % are below the mean and 50 % 00:20:45.089 --> 00:20:49.009 are above the mean. And in every school, if it's 00:20:49.009 --> 00:20:52.549 a normal population, we're going to have 50 % 00:20:52.549 --> 00:20:57.170 below the mean or below average. Well, if they 00:20:57.170 --> 00:21:02.069 decide that this is basic right here, by definition, 00:21:02.879 --> 00:21:08.579 40 % are gonna be below the basic level. And 00:21:08.579 --> 00:21:13.039 60 % are gonna be below the proficient level. 00:21:13.339 --> 00:21:17.579 And you can say, oh, we have a crisis. Only 40 00:21:17.579 --> 00:21:20.599 % of our students are reading at the proficient 00:21:20.599 --> 00:21:24.559 level. You see how the game is played? That's 00:21:24.559 --> 00:21:29.420 how they play the game. That's not right. Eighth 00:21:29.420 --> 00:21:35.920 grade, same thing. Same thing. Same thing. That's 00:21:35.920 --> 00:21:42.579 not right. Walton said, this is from their website. 00:21:43.160 --> 00:21:45.819 This is from the Center for Educational Statistics. 00:21:46.359 --> 00:21:49.359 Proficiency levels are based on desired expectation. 00:21:49.819 --> 00:21:54.240 These levels are aspirational, not mathematical. 00:21:54.900 --> 00:21:57.859 We would like this to be basic. We would like 00:21:57.859 --> 00:22:02.329 this. there to be used on a trial basis, we need 00:22:02.329 --> 00:22:05.750 evidence to determine if they are reasonable, 00:22:06.230 --> 00:22:13.250 valid, and informative. Hand -picked political 00:22:13.250 --> 00:22:17.950 appointees, arbitrarily decided levels, any time 00:22:17.950 --> 00:22:21.970 you hear about NAP levels of basic and proficient, 00:22:22.289 --> 00:22:26.519 even on state level tests. These should always 00:22:26.519 --> 00:22:30.079 be put in this context and they should always 00:22:30.079 --> 00:22:36.440 be questioned. State criterion reference test. 00:22:37.380 --> 00:22:40.140 You can't compare state to state because each 00:22:40.140 --> 00:22:44.359 state has a different test, but someone establishes 00:22:44.359 --> 00:22:47.160 a criterion. We think third graders should know 00:22:47.160 --> 00:22:50.720 that and sixth graders should know that. A test 00:22:50.720 --> 00:22:54.299 is given every year. And we think this is basic, 00:22:54.559 --> 00:22:59.720 and that is, arbitrarily decide who decides the 00:22:59.720 --> 00:23:03.019 criteria, who decides the content of these tests, 00:23:03.380 --> 00:23:07.119 who decides what's advanced, proficient, and 00:23:07.119 --> 00:23:12.059 basic, and how are these decisions made. I had 00:23:12.059 --> 00:23:15.380 to do a lot of looking here in Minnesota on their 00:23:15.380 --> 00:23:18.559 websites, and I finally found, after a lot of 00:23:18.559 --> 00:23:21.019 phone calls and tiny print on their websites, 00:23:22.919 --> 00:23:26.740 that originally these MCAs were based on the 00:23:26.740 --> 00:23:30.000 Common Core State Standards. They took these 00:23:30.000 --> 00:23:33.160 three -dimensional standards and made a two -dimensional 00:23:33.160 --> 00:23:35.519 test. Of course, you never actually get to see 00:23:35.519 --> 00:23:39.759 what's on the test. And in your real life, how 00:23:39.759 --> 00:23:42.960 often have you had to do these things? But yet 00:23:42.960 --> 00:23:47.220 these are used to define the literacy potential 00:23:47.220 --> 00:23:51.400 of our students. These are, and these standards, 00:23:51.640 --> 00:23:55.079 Common Core, were aspirational, not mathematical. 00:23:55.779 --> 00:23:58.200 To prepare students, we would like them to be 00:23:58.200 --> 00:24:02.099 able to do these things at each level, aspirational. 00:24:02.799 --> 00:24:05.779 And this idea, like they have here in Minnesota, 00:24:06.180 --> 00:24:08.859 that every student will be reading at grade level, 00:24:09.460 --> 00:24:14.519 that's simply absurd. That creates winners and 00:24:14.519 --> 00:24:18.740 losers. Grade level. How do you determine grade 00:24:18.740 --> 00:24:21.960 level? Criterion based, norm reference based, 00:24:22.339 --> 00:24:25.460 aspirational, inspirational, fairy tale based 00:24:25.460 --> 00:24:29.859 measures. Now in Minnesota, a couple of years 00:24:29.859 --> 00:24:32.579 back, they claimed that test scores were falling, 00:24:32.819 --> 00:24:35.980 but no test scores were ever reported. They don't 00:24:35.980 --> 00:24:39.220 give you that. They simply give you percentages. 00:24:39.660 --> 00:24:44.259 And four of these arbitrarily defined categories. 00:24:44.839 --> 00:24:50.180 Those aren't test scores. And you can't use percentages 00:24:50.180 --> 00:24:54.700 to make comparison. This is percentages of students 00:24:54.700 --> 00:24:58.900 at or above proficiency. That's not a comparison 00:24:58.900 --> 00:25:02.519 of scores over time. That's what's called in 00:25:02.519 --> 00:25:06.119 the business baloney. And that's the family friendly 00:25:06.119 --> 00:25:10.299 term for it. So tips for evaluating scholarly 00:25:10.299 --> 00:25:14.130 work. And again, Academic discourse is when you 00:25:14.130 --> 00:25:17.509 use research and reason to make a case for one's 00:25:17.509 --> 00:25:20.430 argument. Argument here isn't the yelling kind 00:25:20.430 --> 00:25:23.170 of argument. It's where you put forth a proposition 00:25:23.170 --> 00:25:27.150 and you make a case for that using reason, using 00:25:27.150 --> 00:25:33.789 reason and research. And disagreement is a natural 00:25:33.789 --> 00:25:37.289 part of the academic conversation. It's how we 00:25:37.289 --> 00:25:41.390 evolve our thinking. There should never be a 00:25:41.390 --> 00:25:44.710 singular type of thinking. This insulated bubble 00:25:44.710 --> 00:25:47.670 that the reading league tries to create for itself, 00:25:48.390 --> 00:25:52.450 that creates cognitive inbreeding and deleterious 00:25:52.450 --> 00:25:57.349 mutations. We need to hear other ideas to evolve 00:25:57.349 --> 00:26:00.349 our thinking. I say the reading league because 00:26:00.349 --> 00:26:03.609 when I post things on their various platforms, 00:26:03.730 --> 00:26:06.980 they block me. Oh, we can't have our people hearing 00:26:06.980 --> 00:26:09.940 that idea. That's horrible. They may believe 00:26:09.940 --> 00:26:14.160 the wrong thing. So they block me. Interesting. 00:26:15.019 --> 00:26:19.359 Now, it's important that we critique each other's 00:26:19.359 --> 00:26:23.700 work. That's what we do in a scholarly, in academia, 00:26:24.240 --> 00:26:28.000 in a scholarly context. Now, I have a book coming 00:26:28.000 --> 00:26:30.920 out this summer, the one you see here. And I 00:26:30.920 --> 00:26:34.119 hope you critique it. I hope you tell me what 00:26:34.119 --> 00:26:37.059 I got wrong. Don't tell me I'm a bad person or 00:26:37.059 --> 00:26:40.500 I'm a horrible person. I already know that. Don't 00:26:40.500 --> 00:26:43.500 tell me I'm not very smart. I already know that. 00:26:43.680 --> 00:26:46.119 Tell me what you disagree with and what I got 00:26:46.119 --> 00:26:49.200 wrong. And here's the thing. If you tell me what 00:26:49.200 --> 00:26:53.420 I get wrong and I'm convinced, I will make another 00:26:53.420 --> 00:26:56.619 video. But don't just say I'm a bad person and 00:26:56.619 --> 00:27:00.839 I got it wrong. That's not academic. Discourse. 00:27:01.240 --> 00:27:04.559 Disagreements based on works published or presented 00:27:04.559 --> 00:27:09.400 in an academic context. That's what academic 00:27:09.400 --> 00:27:13.319 discourse is. Not on personal attributes, not 00:27:13.319 --> 00:27:18.039 on personal emails. This is where we disagree. 00:27:18.259 --> 00:27:21.599 We must have disagreement. That's how we evolve 00:27:21.599 --> 00:27:26.759 our thinking. Academic discourse using reason 00:27:26.759 --> 00:27:29.400 and research. research -based theories. And by 00:27:29.400 --> 00:27:33.559 the way, there never was a reading war. That 00:27:33.559 --> 00:27:39.099 was a term invented by people outside of an academic 00:27:39.099 --> 00:27:43.160 context in the media to sell books or papers, 00:27:43.180 --> 00:27:46.940 whatever. There has always been differing ideas 00:27:46.940 --> 00:27:50.740 for decades since the National Council of Teachers 00:27:50.740 --> 00:27:55.259 of English was formed in 1911. the International 00:27:55.259 --> 00:27:57.880 Literacy Association, then the International 00:27:57.880 --> 00:28:02.539 Reading Association in 1956. People have differing 00:28:02.539 --> 00:28:06.279 ideas based on their theoretical orientations. 00:28:06.839 --> 00:28:10.500 Academic discourse is how we hold ourselves accountable. 00:28:11.539 --> 00:28:17.140 And I've taken Sally Shaywitz to terms for her 00:28:17.140 --> 00:28:20.210 research, for her bad research. but I'm sure 00:28:20.210 --> 00:28:22.609 she's a wonderful human being and she has to 00:28:22.609 --> 00:28:27.450 be very smart, but I have trouble with all the 00:28:27.450 --> 00:28:32.910 flaws and confounding variables in her research. 00:28:33.269 --> 00:28:37.059 How does she get away with that, I wonder? The 00:28:37.059 --> 00:28:39.759 differences we have in academia are based on 00:28:39.759 --> 00:28:42.960 theoretical orientations. Every theory can be 00:28:42.960 --> 00:28:46.500 supported by research data and evidence. Theories 00:28:46.500 --> 00:28:49.640 are neither right nor wrong. They're weak or 00:28:49.640 --> 00:28:52.240 robust. And I'll demonstrate that. Flat earth, 00:28:52.380 --> 00:28:55.960 that's a theory. There may be little bits of 00:28:55.960 --> 00:28:59.000 data. It's a very, very weak theory. And I think 00:28:59.000 --> 00:29:02.200 the simple view of reading is the flat earth 00:29:02.200 --> 00:29:05.200 theory of reading. Yes, there's research to support 00:29:05.200 --> 00:29:09.519 it. but it's weak research. It's a weak theory. 00:29:10.099 --> 00:29:14.980 Theories are based on research. Research study 00:29:14.980 --> 00:29:18.700 is a data dot. That's why you can have separate, 00:29:18.940 --> 00:29:21.940 you can have different research -based theory 00:29:21.940 --> 00:29:24.660 connecting different data dots differently, and 00:29:24.660 --> 00:29:30.819 they're both research -based. It's weak or robust. 00:29:31.019 --> 00:29:34.200 theory. A robust theory connects a lot of data 00:29:34.200 --> 00:29:37.799 dots. A weak theory leaves a lot of data unaccounted 00:29:37.799 --> 00:29:43.000 for. The interactive theory of reading, psycholinguistic, 00:29:43.099 --> 00:29:47.059 neurocognitive theories, account for a lot of 00:29:47.059 --> 00:29:50.920 data dots. The simple view of reading account 00:29:50.920 --> 00:29:53.759 for a few. They ignore a whole bunch of stuff. 00:29:55.180 --> 00:29:59.079 Now here's one way to transform weak into robust 00:29:59.079 --> 00:30:01.759 theories. You can do what the National Reading 00:30:01.759 --> 00:30:06.319 Panel did. They ignored or excluded a whole bunch 00:30:06.319 --> 00:30:08.839 of research. Oh, all that research over there. 00:30:09.079 --> 00:30:13.019 That's not good stuff. Only this kind of research 00:30:13.019 --> 00:30:19.720 is good. Okay. Reading instruction is an academic 00:30:19.720 --> 00:30:24.500 topic. Academic arguments based on theoretical 00:30:24.500 --> 00:30:27.980 differences have been going on for years. It's 00:30:27.980 --> 00:30:32.099 a natural part, and as I said, the term reading 00:30:32.099 --> 00:30:38.779 wars was invented by the media. All right, let's 00:30:38.779 --> 00:30:41.319 take a look at scholars. We are looking at evidence. 00:30:41.740 --> 00:30:46.619 Scholars should not get a free pass on obfuscation. 00:30:49.400 --> 00:30:52.859 Common tricks, using buzzwords. That's a word 00:30:52.859 --> 00:30:57.130 used to elicit an emotional response. rather 00:30:57.130 --> 00:31:02.529 than accurately transmit information. And using 00:31:02.529 --> 00:31:05.710 them in an academic context, that's an academic 00:31:05.710 --> 00:31:08.730 buzzword used to create the illusion of something 00:31:08.730 --> 00:31:13.769 important. Now, take a look at these academic 00:31:13.769 --> 00:31:17.009 buzzwords. They're used all the time to make 00:31:17.009 --> 00:31:20.869 something sound very, very important. Could you 00:31:20.869 --> 00:31:26.319 define each one of these? If I went to a reading 00:31:26.319 --> 00:31:30.019 league conference and sat down and have each 00:31:30.019 --> 00:31:34.059 person write a definition of these, how much 00:31:34.059 --> 00:31:38.119 similarity do you think there would be in these? 00:31:39.119 --> 00:31:43.380 What is evidence -based instruction? Well, you 00:31:43.380 --> 00:31:46.579 know, it's instruction and it's based on evidence. 00:31:46.859 --> 00:31:51.859 Okay, buzzword. And we already talked about the 00:31:51.859 --> 00:31:58.559 straw man argument. Louisa Moats, oh, Louisa, 00:31:58.619 --> 00:32:03.059 Louisa, Louisa. She wrote this for the Thomas 00:32:03.059 --> 00:32:06.640 Fordham Institutes, not peer reviewed. It's, 00:32:06.859 --> 00:32:10.720 but here's the ultimate straw person talking 00:32:10.720 --> 00:32:13.839 about whole language, uses teacher modeling, 00:32:14.000 --> 00:32:19.539 not direct instruction. Instead of teaching children 00:32:19.539 --> 00:32:22.140 how to read and comprehend, they engage in shared 00:32:22.140 --> 00:32:26.019 reading. Students can repeat the language and 00:32:26.019 --> 00:32:30.160 read by osmosis or memorization. Just, Louisa, 00:32:30.460 --> 00:32:33.700 do you not know better? That is not at all what 00:32:33.700 --> 00:32:36.279 whole language is. Do you not know better or 00:32:36.279 --> 00:32:38.819 do you know better and you're doing this anyway? 00:32:39.319 --> 00:32:42.819 Both are frightening propositions, but this is 00:32:42.819 --> 00:32:46.180 a straw man argument. This is a cartoonish version 00:32:46.180 --> 00:32:50.960 of something. Do you have no decency? Don't do 00:32:50.960 --> 00:32:59.089 that. Baloney, is interpreting research or misrepresenting 00:32:59.089 --> 00:33:04.549 decontextualized data. This is why we have to 00:33:04.549 --> 00:33:08.190 critically evaluate all the scholarly work. We 00:33:08.190 --> 00:33:11.109 can't be snookered by important sounding brain 00:33:11.109 --> 00:33:14.309 words. We have to pull them apart. We have to 00:33:14.309 --> 00:33:18.490 examine the propositions and the citations. And 00:33:18.490 --> 00:33:21.670 a cow pie is when you put a whole bunch of academic 00:33:21.670 --> 00:33:25.099 buzzwords together. to create a conveniently 00:33:25.099 --> 00:33:28.880 inaccurate perception of something. Here's an 00:33:28.880 --> 00:33:33.039 example of a cow pie. Goldenberg wrote this in 00:33:33.039 --> 00:33:36.960 reading research quarterly, talking about reading 00:33:36.960 --> 00:33:41.299 wars. Now, a proposition is a basic claim or 00:33:41.299 --> 00:33:43.359 assertion within a sentence. You can say it's 00:33:43.359 --> 00:33:47.059 true or false. The dog is in the house. Three 00:33:47.059 --> 00:33:50.309 propositions. There's a dog. The dog is in something 00:33:50.309 --> 00:33:54.430 and it is the house in which the dog is in. Three 00:33:54.430 --> 00:34:00.029 propositions there. Here's Goldenberg's abstract. 00:34:00.569 --> 00:34:03.890 The author's intent is to begin with the knowledge 00:34:03.890 --> 00:34:07.089 base of the science of reading, build onto it 00:34:07.089 --> 00:34:09.670 research on effective literacy instruction for 00:34:09.670 --> 00:34:12.269 ELLs, and then begin articulating what could 00:34:12.269 --> 00:34:15.269 become a science of reading instruction for students 00:34:15.269 --> 00:34:17.550 who are learning and progressing as readers in 00:34:17.550 --> 00:34:20.130 English while learning to speak and understand. 00:34:20.210 --> 00:34:25.050 It sounds very good. It sounds so wonderful. 00:34:26.050 --> 00:34:30.719 But let's, when we look at that cow pie, we can 00:34:30.719 --> 00:34:34.039 find here's just two of the propositions. There 00:34:34.039 --> 00:34:37.239 is a knowledge base and the science of reading 00:34:37.239 --> 00:34:40.019 has it. The science of reading knowledge base. 00:34:40.320 --> 00:34:44.019 Where is that? Where can I find that? Who put 00:34:44.019 --> 00:34:47.159 the knowledge in the base? Is there a balanced 00:34:47.159 --> 00:34:50.900 literacy knowledge base? Hmm, these are buzzwords 00:34:50.900 --> 00:34:55.110 strongly together. I could do that. The presenter's 00:34:55.110 --> 00:34:57.610 intent in this presentation is to begin with 00:34:57.610 --> 00:35:00.630 the knowledge base of reading research, build 00:35:00.630 --> 00:35:03.409 on to it research related to neuroscience, psycholungus. 00:35:03.570 --> 00:35:07.550 You get the idea. Anyone can create a cow pie. 00:35:07.889 --> 00:35:11.570 It's not that tough. It creates the illusion 00:35:11.570 --> 00:35:16.130 of stuff, but it doesn't transmit information. 00:35:16.590 --> 00:35:19.730 We have to be wary when we see things like this. 00:35:20.570 --> 00:35:23.269 This is a misinterpretation. This is an example 00:35:23.269 --> 00:35:26.610 of baloney. In that paper, he referenced two 00:35:26.610 --> 00:35:30.210 research studies to make a case for structured 00:35:30.210 --> 00:35:32.670 reading instruction. And this is what he said. 00:35:33.070 --> 00:35:38.289 He looked at von et al and Eri et al's research 00:35:38.289 --> 00:35:41.829 studies. They included explicit and systematic 00:35:41.829 --> 00:35:44.269 instruction of targeted knowledge and skills 00:35:44.269 --> 00:35:47.170 and daily lessons that include activities focused 00:35:47.170 --> 00:35:50.250 on the five core skills of phonological awareness, 00:35:50.530 --> 00:35:53.809 phonics, fluency, and comprehension, developing 00:35:53.809 --> 00:35:56.610 oral language, integrated oral language. It sounds 00:35:56.610 --> 00:36:01.090 very good. Stop using my cow pies, the cow says. 00:36:01.989 --> 00:36:04.710 But when we look at that, and I just so happened. 00:36:06.139 --> 00:36:08.699 to look at each of these studies, and I've happened 00:36:08.699 --> 00:36:11.619 to analyze both of them very carefully. These 00:36:11.619 --> 00:36:14.900 are the pretty pretty words that Claude Goldenberg 00:36:14.900 --> 00:36:18.699 uses. Oh, they are so pretty, pretty pretty words, 00:36:18.760 --> 00:36:23.619 but they are conveniently inaccurate because 00:36:23.619 --> 00:36:27.679 in his description none of these words were supported 00:36:27.679 --> 00:36:33.199 by the research he is citing. Yes, some of them 00:36:33.340 --> 00:36:36.420 were used as descriptors in the studies. Their 00:36:36.420 --> 00:36:39.320 description was often inaccurate, incomplete, 00:36:39.460 --> 00:36:44.139 or just plain wrong. But none of the pretty words 00:36:44.139 --> 00:36:47.599 were variables that were isolated and compared 00:36:47.599 --> 00:36:51.420 in these two studies. Why is that important? 00:36:52.480 --> 00:36:55.699 And here's more baloney. Effects were moderate 00:36:55.699 --> 00:36:59.139 measures on English, phonological awareness, 00:36:59.420 --> 00:37:02.849 et cetera, et cetera. OK. Principles of early 00:37:02.849 --> 00:37:06.610 reading instruction. Okay. Isolating variables 00:37:06.610 --> 00:37:10.210 in controlled experimental research, which is 00:37:10.210 --> 00:37:12.570 the science of reading research standards. That's 00:37:12.570 --> 00:37:17.429 not mine. It's used to determine if one thing 00:37:17.429 --> 00:37:22.449 caused the other thing to occur. If structured 00:37:22.449 --> 00:37:25.289 language causes something to occur, but you have 00:37:25.289 --> 00:37:28.489 to start with two groups that are similar and 00:37:28.489 --> 00:37:33.000 exposed to similar conditions. The treatment 00:37:33.000 --> 00:37:35.840 is the variable. That's the only difference that 00:37:35.840 --> 00:37:39.539 should be between the two groups. And if there 00:37:39.539 --> 00:37:42.880 is a difference on some measure between the two 00:37:42.880 --> 00:37:46.039 groups at the end, and if that difference is 00:37:46.039 --> 00:37:49.480 statistically significant, we can say with some 00:37:49.480 --> 00:37:53.380 amount of confidence that the treatment was the 00:37:53.380 --> 00:37:57.219 cause of the variable. That's a very simple overview. 00:37:57.480 --> 00:38:00.900 of controlled experimental research. But in both 00:38:00.900 --> 00:38:04.539 the studies that Goldenberg is citing so confidently, 00:38:05.420 --> 00:38:09.119 they compared students who received individual 00:38:09.119 --> 00:38:13.280 tutoring to students who received instruction 00:38:13.280 --> 00:38:16.659 in small groups. And they said, oh, it was the 00:38:16.659 --> 00:38:20.460 specific type of tutoring that caused the difference. 00:38:21.400 --> 00:38:24.619 When in fact, it had nothing to do with the tutoring. 00:38:25.130 --> 00:38:29.449 One was individual tutoring with individual tutor 00:38:29.449 --> 00:38:32.909 with lots of feedback and a whole bunch of stuff. 00:38:33.690 --> 00:38:36.570 And another was in small group using prepackaged 00:38:36.570 --> 00:38:40.710 remedial stuff. It was designed to give them 00:38:40.710 --> 00:38:44.650 a result that was favorable to the group that 00:38:44.650 --> 00:38:49.409 they wanted to. As an example, this is what it 00:38:49.409 --> 00:38:53.670 was. My research question was, do blue cages 00:38:53.670 --> 00:38:56.489 cause mice to gain weight? And I said, yes, they 00:38:56.489 --> 00:39:00.909 do. Because we compared them, and the blue cage 00:39:00.909 --> 00:39:05.170 mice were heavier, and the yellow cage mice were 00:39:05.170 --> 00:39:08.250 skinnier at the end of this experiment. So if 00:39:08.250 --> 00:39:11.449 you want to lose weight, paint your house yellow. 00:39:12.409 --> 00:39:16.469 And you see the confounding variable here? These 00:39:16.469 --> 00:39:20.420 aren't equal groups. One was fed marshmallows, 00:39:20.440 --> 00:39:24.860 the other was fed salaries. In the studies, Goldenberg 00:39:24.860 --> 00:39:29.199 cited the type of metaphorical food given to 00:39:29.199 --> 00:39:33.699 the metaphorical mice were different. They claimed 00:39:33.699 --> 00:39:36.840 it was the color of cages that caused improved 00:39:36.840 --> 00:39:40.179 reading scores. Anyone with a sixth grade education 00:39:40.179 --> 00:39:43.039 could see that if you read the studies carefully. 00:39:44.119 --> 00:39:48.059 So three questions, which A controlled experimental 00:39:48.059 --> 00:39:51.440 research will not answer. Did Goldenberg not 00:39:51.440 --> 00:39:54.760 read the studies? Did he read the studies and 00:39:54.760 --> 00:39:58.420 not see the confounding variable? Or did he read 00:39:58.420 --> 00:40:02.440 the studies, see the anomaly, and make the citation 00:40:02.440 --> 00:40:08.460 anyway? I would like to know. Understanding research. 00:40:08.699 --> 00:40:12.260 And by the way, if state legislators were really 00:40:12.260 --> 00:40:17.030 serious about evidence -based practice for reading 00:40:17.030 --> 00:40:19.730 and research -based reading instruction, but 00:40:19.730 --> 00:40:23.110 they're not. If they were, every school district 00:40:23.110 --> 00:40:26.309 would have a subscription to the same kind of 00:40:26.309 --> 00:40:30.510 legitimate research database that I do as a university 00:40:30.510 --> 00:40:34.030 professor. And not what works Glaringhouse or 00:40:34.030 --> 00:40:37.969 Eric. That's not legitimate. That's not legitimate. 00:40:38.590 --> 00:40:42.210 How do you expect teachers to make research -based 00:40:42.210 --> 00:40:45.969 decisions and evaluate evidence -based practice 00:40:45.969 --> 00:40:50.130 without research and evidence, and yet you blame 00:40:50.130 --> 00:40:54.050 them and you beat them up. That's not logical. 00:40:54.090 --> 00:40:58.110 That's not science. There's no such thing as 00:40:58.110 --> 00:41:02.190 the research. The research says. The research 00:41:02.190 --> 00:41:06.590 shows. I'm often asked this, show me the research. 00:41:06.809 --> 00:41:10.940 What does the research say? They demand of me 00:41:10.940 --> 00:41:15.079 the research. By the way, when anyone uses research 00:41:15.079 --> 00:41:18.639 in a declarative sentence, they must, in an academic 00:41:18.639 --> 00:41:21.820 setting, cite their research. You can't just 00:41:21.820 --> 00:41:24.780 say, research says there's no such thing as gravity. 00:41:25.900 --> 00:41:29.800 There's no such thing as the research. There's 00:41:29.800 --> 00:41:34.260 no objective God. Oh, almighty research, we prayeth 00:41:34.260 --> 00:41:38.570 unto you, give us truth and enlightenment. There's 00:41:38.570 --> 00:41:42.809 no God who offers this objective view of reality. 00:41:44.929 --> 00:41:48.670 There's no such thing as the holy book called 00:41:48.670 --> 00:41:51.489 the research, where you can look up and say, 00:41:52.070 --> 00:41:58.289 ah, the research says. Maybe they mean Thea research, 00:41:58.590 --> 00:42:00.909 a researcher named Thea research. And they're 00:42:00.909 --> 00:42:06.079 saying Thea research says. Maybe I should call 00:42:06.079 --> 00:42:09.860 my next book the research. So you could say the 00:42:09.860 --> 00:42:17.059 research sets. Research involves science. Science 00:42:17.059 --> 00:42:21.159 uses research. It's a process of asking a question, 00:42:21.340 --> 00:42:25.079 collecting data and using data to answer that 00:42:25.079 --> 00:42:28.300 question. The question up front is important. 00:42:28.590 --> 00:42:31.630 Louisa Motes tends to just collect data and then 00:42:31.630 --> 00:42:35.670 come to a conclusion. That's not research. What's 00:42:35.670 --> 00:42:38.610 the weather? Question. Look at the thermometer. 00:42:38.989 --> 00:42:43.550 Data. It's hot. Data to answer question. That's 00:42:43.550 --> 00:42:47.550 a basic of research. It's a systematic process 00:42:47.550 --> 00:42:50.909 used to collect and analyze data. Systematic 00:42:50.909 --> 00:42:55.289 means there's a fixed plan. based on a set of 00:42:55.289 --> 00:42:57.869 principles that are generally agreed upon by 00:42:57.869 --> 00:43:02.190 the field. And the research method is a systematic 00:43:02.190 --> 00:43:06.590 process used to gather and analyze that data. 00:43:07.849 --> 00:43:10.809 That's important because there is no scientific 00:43:10.809 --> 00:43:13.989 method. There is no the method of science. There 00:43:13.989 --> 00:43:21.210 are methods of science. Controlled experimental 00:43:21.210 --> 00:43:23.550 research, the controlled experimental research 00:43:23.550 --> 00:43:31.829 is only one method of science. We must understand 00:43:31.829 --> 00:43:35.650 that conducting research in the physical sciences 00:43:35.650 --> 00:43:39.530 is a bit different from research in the social 00:43:39.530 --> 00:43:43.090 sciences. With human beings, we cannot control 00:43:43.090 --> 00:43:45.889 all the variables. We cannot put our children 00:43:45.889 --> 00:43:50.420 in giant Petri dishes. and give them similar 00:43:50.420 --> 00:43:54.739 conditions. That's not how it works. And research 00:43:54.739 --> 00:43:59.059 is not research unless and until it has been 00:43:59.059 --> 00:44:02.340 subjected to blind peer review and published 00:44:02.340 --> 00:44:10.980 in an academic journal. Here's the basic blind 00:44:10.980 --> 00:44:14.980 peer review process. This is why it's important. 00:44:16.360 --> 00:44:20.500 You do a study. You write it up. You submit the 00:44:20.500 --> 00:44:24.860 manuscript to an editor. An editor submits it 00:44:24.860 --> 00:44:28.000 to blind peer review. These are experts in their 00:44:28.000 --> 00:44:30.900 field. They don't know who the author of the 00:44:30.900 --> 00:44:34.219 article is. The author doesn't know who the reviewers 00:44:34.219 --> 00:44:38.260 are. They review that and they make a recommendation 00:44:38.260 --> 00:44:43.820 either to reject, accept with revisions, or accept. 00:44:44.519 --> 00:44:47.420 There, and it's usually two to five reviewers, 00:44:48.420 --> 00:44:51.559 their comments go to the editor, who then makes 00:44:51.559 --> 00:44:57.460 a decision based on their comments to accept, 00:44:58.079 --> 00:45:00.820 reject, or accept with revisions, and they work 00:45:00.820 --> 00:45:04.500 with the author. That's the process, checks and 00:45:04.500 --> 00:45:08.420 balances a filter. That's different from the 00:45:08.420 --> 00:45:11.980 process a journalist use. People tell them things, 00:45:12.239 --> 00:45:14.940 they write a story, and they submit it to the 00:45:14.940 --> 00:45:19.119 editor who approves or asks for revision. Much 00:45:19.119 --> 00:45:23.519 different. And the process used to put stuff 00:45:23.519 --> 00:45:26.519 on the internet. You think of stuff, you write 00:45:26.519 --> 00:45:30.420 it, and there you go. It's on the internet. Interesting 00:45:30.420 --> 00:45:35.300 article I recently analyzed. It's not research. 00:45:35.719 --> 00:45:38.360 These publications don't go through blind peer 00:45:38.360 --> 00:45:41.960 review. They just hire reporters to say stop. 00:45:42.460 --> 00:45:47.460 American public media. Data is not research. 00:45:47.840 --> 00:45:51.800 Published reports are not research. White papers 00:45:51.800 --> 00:45:56.320 aren't research. Unpublished theses and dissertations 00:45:56.320 --> 00:46:01.820 do not go through blind peer review. Articles 00:46:01.820 --> 00:46:06.030 describing research are not research. Famous 00:46:06.030 --> 00:46:10.190 researchers aren't research, secretaries of education 00:46:10.190 --> 00:46:14.469 aren't research, and anything on Lexia Learning 00:46:14.469 --> 00:46:19.510 website is not research. These are often used 00:46:19.510 --> 00:46:22.969 as if they were by people outside of academia, 00:46:24.030 --> 00:46:28.369 collecting data, citing white papers, think tanks, 00:46:29.170 --> 00:46:39.139 not research. Not. research. Non -profit does 00:46:39.139 --> 00:46:43.519 not mean non -biased. These are all think tanks 00:46:43.519 --> 00:46:46.940 that have things to say about education, but 00:46:46.940 --> 00:46:53.619 they all have an agenda. The two kinds of research, 00:46:54.159 --> 00:46:58.130 basic and applied, basic science, and applied 00:46:58.130 --> 00:47:01.650 science. Basic is research that is done apart 00:47:01.650 --> 00:47:04.769 from the setting in which it's used, eye movement, 00:47:04.889 --> 00:47:08.070 brain imaging, miscue analysis. Applied research 00:47:08.070 --> 00:47:10.769 is done in the setting in which it will be used 00:47:10.769 --> 00:47:16.269 in the classroom or learning environment. And 00:47:16.269 --> 00:47:19.849 controlled experimental research is used to isolate 00:47:19.849 --> 00:47:23.570 a variable to determine a causal relationship. 00:47:24.030 --> 00:47:31.849 This thing caused that to occur. Now the science 00:47:31.849 --> 00:47:37.550 of reading research standard says only applied 00:47:37.550 --> 00:47:41.309 research using control and experimental groups 00:47:41.309 --> 00:47:44.849 can be used to make decisions about reading policies, 00:47:45.389 --> 00:47:49.090 products, and procedures based on the mistaken 00:47:49.090 --> 00:47:52.369 assumption that you can isolate all the variables 00:47:52.369 --> 00:47:58.409 to determine universal causality and generalize 00:47:58.409 --> 00:48:01.769 that to all of humanity. That's a mistaken assumption. 00:48:04.329 --> 00:48:07.289 Research is not research unless it has been subjected 00:48:07.289 --> 00:48:10.989 to blind peer review. Research is different from 00:48:10.989 --> 00:48:19.829 collecting data. Data is not research. I could 00:48:19.829 --> 00:48:24.079 collect data to show that I am the greatest professor 00:48:24.079 --> 00:48:27.579 in the world, but that wouldn't be research. 00:48:28.300 --> 00:48:31.119 What's a good professor? What's a valid measure 00:48:31.119 --> 00:48:35.320 of a good professor? These are all things that 00:48:35.320 --> 00:48:38.780 I could collect. One of these, oh, look at my 00:48:38.780 --> 00:48:41.760 course evaluation. Oh, look at the number of 00:48:41.760 --> 00:48:46.099 YouTube views. That proves I'm a good professor. 00:48:47.239 --> 00:48:51.130 I highlight this because at our university, someone 00:48:51.130 --> 00:48:55.449 wanted to have metrics, I hate that term, metrics, 00:48:56.030 --> 00:48:59.510 to show that our College of Ed was doing a good 00:48:59.510 --> 00:49:03.909 job with teachers. So to get metrics, they found 00:49:03.909 --> 00:49:07.750 something that could be measured. And they went 00:49:07.750 --> 00:49:11.369 to the scores on the state teacher licensure 00:49:11.369 --> 00:49:14.050 exams that our students took, and they said, 00:49:14.050 --> 00:49:16.389 oh, look at the scores, based on that, we're 00:49:16.389 --> 00:49:20.530 doing a good job. That's not a valid measure. 00:49:21.309 --> 00:49:24.070 And here's the thing, if we spent the three semesters 00:49:24.070 --> 00:49:27.650 we have them, just preparing them to take those 00:49:27.650 --> 00:49:31.110 tests, our scores would be out of this world. 00:49:31.809 --> 00:49:34.349 But we try to prepare them to do this thing called 00:49:34.349 --> 00:49:41.489 teach, teach. Imagine that. A Massachusetts rep 00:49:41.489 --> 00:49:45.030 said, show me the data on queuing. The data, 00:49:45.230 --> 00:49:50.389 the data. Data is not research. Anybody can collect 00:49:50.389 --> 00:49:55.250 data. And here's the thing on queuing. Queuing 00:49:55.250 --> 00:50:00.010 is not a method or a approach. How's that? There's 00:50:00.010 --> 00:50:03.070 some data. The three queuing is not a method. 00:50:03.250 --> 00:50:06.150 It's not something you teach. And we'll talk 00:50:06.150 --> 00:50:12.250 about that later. Methods of science. The scientific 00:50:12.250 --> 00:50:19.809 method. There's no such thing as the method. 00:50:20.389 --> 00:50:23.710 A variety of research methods are used to study. 00:50:23.989 --> 00:50:27.909 And there's two groups, quantitative and qualitative. 00:50:28.449 --> 00:50:32.389 Let us understand the basics. Quantitative research, 00:50:32.409 --> 00:50:35.349 again, seeks to find that causal relationship, 00:50:36.130 --> 00:50:40.210 controls or manipulates reality to isolate a 00:50:40.210 --> 00:50:44.079 variable. You have a measure. You have two groups 00:50:44.079 --> 00:50:47.460 that are similar. You isolate one. The only difference 00:50:47.460 --> 00:50:50.280 is this treatment. And then you have a post -test. 00:50:50.579 --> 00:50:55.579 That is the basics. The difference must be statistically 00:50:55.579 --> 00:50:59.699 significant. That means it's greater than could 00:50:59.699 --> 00:51:03.260 occur by chance. Now, you don't need to understand 00:51:03.260 --> 00:51:06.860 statistics to understand statistical significance. 00:51:09.990 --> 00:51:14.909 statistically significant at the .05 level, that 00:51:14.909 --> 00:51:19.269 means in less than 5 % of the cases, can we say 00:51:19.269 --> 00:51:23.849 that that difference was attributed to chance? 00:51:24.630 --> 00:51:28.650 In 95 % of the cases, we can be confident that 00:51:28.650 --> 00:51:31.150 the difference between the two groups was caused 00:51:31.150 --> 00:51:36.869 by the variable. .05 is good, .01 is the best. 00:51:37.050 --> 00:51:41.110 but 0 .05 is generally regarded as the standard. 00:51:42.809 --> 00:51:46.929 Controlled experimental research seeks to quantify 00:51:46.929 --> 00:51:50.949 reality. But here's the thing. By controlling 00:51:50.949 --> 00:51:54.929 all the variables to study reality, you make 00:51:54.929 --> 00:51:59.789 that bit of reality a bit of non -reality. You 00:51:59.789 --> 00:52:03.530 mess with the environment. It's not how it really 00:52:03.530 --> 00:52:07.650 is if you're controlling all the variables. Qualitative 00:52:07.650 --> 00:52:13.110 research seeks to study reality as it is. There's 00:52:13.110 --> 00:52:16.849 Jane Goodall studying apes in the wild making 00:52:16.849 --> 00:52:23.309 notes. There's a variety of types of qualitative 00:52:23.309 --> 00:52:28.949 research. Here you see a professor in there taking 00:52:28.949 --> 00:52:32.309 notes that helps us understand, that sees the 00:52:32.309 --> 00:52:37.500 people behind the numbers. By using only quantitative 00:52:37.500 --> 00:52:40.900 research, we are describing learning and human 00:52:40.900 --> 00:52:45.539 beings in numbers. But in 2002, the Education 00:52:45.539 --> 00:52:50.179 Science of Reform Act said only quantitative 00:52:50.179 --> 00:52:52.860 research could be used. That's the only kind 00:52:52.860 --> 00:52:55.760 of research that would be funded. That's the 00:52:55.760 --> 00:52:59.739 only kind of knowledge that's valid. So a whole 00:52:59.739 --> 00:53:02.199 bunch of qualitative research that helped us 00:53:02.199 --> 00:53:05.340 understand. that helped us understand the numbers 00:53:05.340 --> 00:53:10.000 was eliminated and slowly went away. It has to 00:53:10.000 --> 00:53:13.639 be experimental or quasi experimental research. 00:53:14.059 --> 00:53:17.840 No qualitative research, no government funding. 00:53:19.840 --> 00:53:23.300 So it's not the use of controlled experimental 00:53:23.300 --> 00:53:26.719 research that's problematic. It's the exclusive 00:53:26.719 --> 00:53:30.980 use. you disallow us to see the people behind 00:53:30.980 --> 00:53:35.239 the numbers by disallowing a whole bunch of good 00:53:35.239 --> 00:53:39.719 qualitative research. And this is what the science 00:53:39.719 --> 00:53:44.539 of reading standards are. Strategies and practices 00:53:44.539 --> 00:53:49.639 that lead to improved outcomes, yes. Strategies 00:53:49.639 --> 00:53:52.360 and practices are supported by experimental or 00:53:52.360 --> 00:53:56.679 quasi -experimental research, establishes causal 00:53:56.679 --> 00:54:00.340 variables and the research has been conducted 00:54:00.340 --> 00:54:03.780 in an authentic learning environment. The science 00:54:03.780 --> 00:54:06.980 of reading says they use strategies and practices 00:54:06.980 --> 00:54:10.440 that have been determined to be effective by 00:54:10.440 --> 00:54:14.280 these standards. Those are the standards and 00:54:14.280 --> 00:54:19.400 I would be just fine with those standards if 00:54:19.400 --> 00:54:23.679 they were consistently applied. Now here's an 00:54:23.679 --> 00:54:28.420 example of baloney. In Minnesota and Massachusetts, 00:54:29.000 --> 00:54:32.800 they recently passed legislation. They assumed 00:54:32.800 --> 00:54:36.860 there was a literacy crisis. They assumed teachers 00:54:36.860 --> 00:54:39.639 were teaching the wrong thing. And they assumed 00:54:39.639 --> 00:54:42.420 the problem was balanced literacy and the answer 00:54:42.420 --> 00:54:46.420 was structured literacy. And the causal relationship 00:54:46.420 --> 00:54:50.039 was determined by anecdotal evidence. That's 00:54:50.039 --> 00:54:54.309 not science, people. That's not science. If you're 00:54:54.309 --> 00:54:56.510 going to be a science of reading, you have to 00:54:56.510 --> 00:55:00.989 be consistently scientific. The science of reading 00:55:00.989 --> 00:55:04.670 should not be given a free pass on non -scientific 00:55:04.670 --> 00:55:08.869 thinking. And these are two examples of things 00:55:08.869 --> 00:55:12.829 that are given a pass. Do these. As research 00:55:12.829 --> 00:55:16.139 show that they lead to improved reading achievement 00:55:16.139 --> 00:55:20.059 or improved teacher ability to teach reading 00:55:20.059 --> 00:55:22.480 effectively? And have they been compared with 00:55:22.480 --> 00:55:29.380 something else? No single study provides irrefutable 00:55:29.380 --> 00:55:36.260 proof of anything. Research is a lot like sighting 00:55:36.260 --> 00:55:39.760 in your deer rifle. I grew up in a small town 00:55:39.760 --> 00:55:43.230 in a deer hunting culture. and every fall we 00:55:43.230 --> 00:55:46.429 would sight in our rifles to see if the rifle 00:55:46.429 --> 00:55:48.869 was shooting straight. The scope often got bumped 00:55:48.869 --> 00:55:51.710 over the winter and we wouldn't take a single 00:55:51.710 --> 00:55:54.570 shot. We would take a lot of shots to give us 00:55:54.570 --> 00:55:58.130 a sense of where the scope was shooting because 00:55:58.130 --> 00:56:00.590 we could have wiggled or there could have been 00:56:00.590 --> 00:56:03.210 something that happened with the table that gave 00:56:03.210 --> 00:56:07.460 us an extreme shot. By taking many shots, we 00:56:07.460 --> 00:56:10.000 get a sense of where the scope is shooting, and 00:56:10.000 --> 00:56:14.420 then we adjust our scope. But even then, if we're 00:56:14.420 --> 00:56:16.760 pretty certain of where it's shooting, even though 00:56:16.760 --> 00:56:19.019 we start walking in the woods, it's the person 00:56:19.019 --> 00:56:22.000 behind the scope that determines where the bullet 00:56:22.000 --> 00:56:27.500 lands and a bunch of other variables. No single 00:56:27.500 --> 00:56:30.679 study. Research is like that. We can't find one 00:56:30.679 --> 00:56:34.570 study and say, ah, that proves that. That's not 00:56:34.570 --> 00:56:39.489 how research works. Research does not prove or 00:56:39.489 --> 00:56:44.090 debunk. And by the way, whole language has not 00:56:44.090 --> 00:56:47.489 been debunked. Yes, there's been people that 00:56:47.489 --> 00:56:50.849 say it's been debunked. In the name of almighty 00:56:50.849 --> 00:56:55.690 research, I debunked you, B debunked you. It 00:56:55.690 --> 00:56:59.369 hasn't been debunked just because people say 00:56:59.369 --> 00:57:03.110 that. Okay? Well, I could debunk things. I could 00:57:03.110 --> 00:57:05.989 debunk the theory of gravity. There it is. It's 00:57:05.989 --> 00:57:09.010 been debunked. I debunked it. And here's my proof. 00:57:09.989 --> 00:57:13.829 You can't just go around debunking things. Research 00:57:13.829 --> 00:57:19.730 answers a specific research question. Research 00:57:19.730 --> 00:57:24.750 proves or disproves a hypothesis. And we have 00:57:24.750 --> 00:57:28.110 to be wary of research that doesn't have a specific 00:57:28.110 --> 00:57:32.440 research question. or purpose, and that is true 00:57:32.440 --> 00:57:37.340 of Louisa Moats's stuff. Can't use the word proven 00:57:37.340 --> 00:57:42.559 with science. Observations are never perfect. 00:57:43.500 --> 00:57:48.440 Interpretations are never perfect. Truth needs 00:57:48.440 --> 00:57:51.840 a context and information is often complete. 00:57:52.360 --> 00:57:55.960 Scientists rarely use that word. Here's an example. 00:57:56.489 --> 00:58:00.210 Phonics instruction is good for increasing phonics 00:58:00.210 --> 00:58:04.610 skills. Absolutely. You teach a bunch of phonics 00:58:04.610 --> 00:58:08.289 and students are going to score higher on phonics 00:58:08.289 --> 00:58:13.070 measures. Absolutely. Absolutely. Phonics instruction 00:58:13.070 --> 00:58:16.090 has shown to be more effective in raising scores 00:58:16.090 --> 00:58:19.250 on measures of phonics than no phonics instruction. 00:58:19.929 --> 00:58:25.599 Absolutely. prove that all students need lots 00:58:25.599 --> 00:58:30.099 of phonics instruction? Something can be technically 00:58:30.099 --> 00:58:33.780 true and conveniently inaccurate at the same 00:58:33.780 --> 00:58:37.440 time. This idea that phonics should be the center 00:58:37.440 --> 00:58:40.519 of reading instruction, that is based on a lot 00:58:40.519 --> 00:58:44.219 of I thinkisms. This does not mean all students 00:58:44.219 --> 00:58:48.340 need heavy doses of phonics. Or students would 00:58:48.340 --> 00:58:52.670 be better readers if they get more phonics. Or 00:58:52.670 --> 00:58:56.570 more phonics is better than less phonics. Or 00:58:56.570 --> 00:58:59.530 phonics is the key variable in learning to read. 00:59:00.909 --> 00:59:03.210 Anything other than phonics instruction should 00:59:03.210 --> 00:59:07.570 be excluded. And phonics should be taught at 00:59:07.570 --> 00:59:12.530 all ages and levels. Phonic skills transfer to 00:59:12.530 --> 00:59:17.090 authentic reading situations. These are all assumptions. 00:59:17.599 --> 00:59:21.019 And these research is based on the assumption 00:59:21.019 --> 00:59:25.019 that reading is sounding out words. Research 00:59:25.019 --> 00:59:29.360 based on sounding out words focuses on sounding 00:59:29.360 --> 00:59:32.639 out words and includes sounding out word measures. 00:59:34.019 --> 00:59:37.420 If you ask sounder out questions, you use sounder 00:59:37.420 --> 00:59:40.420 out measures, you collect sounder out data, you 00:59:40.420 --> 00:59:44.710 are going to make sounder out conclusions. That's 00:59:44.710 --> 00:59:50.110 what science of reading research is. Research 00:59:50.110 --> 00:59:52.969 is not collecting a bunch of data and saying 00:59:52.969 --> 00:59:56.590 stuff. Research must be based on a specific question 00:59:56.590 --> 00:59:59.949 or hypothesis. When you read a research study 00:59:59.949 --> 01:00:02.389 after the abstract, the first thing you should 01:00:02.389 --> 01:00:07.570 look for is that research question. And research 01:00:07.570 --> 01:00:12.010 always... focuses on a specific group at a specific 01:00:12.010 --> 01:00:15.469 location doing a specific task. That's why we 01:00:15.469 --> 01:00:22.349 can't use one study. For example, what works 01:00:22.349 --> 01:00:25.630 with a first grade student in Blackwater, Arizona 01:00:25.630 --> 01:00:29.929 struggling with word recognition may not work 01:00:29.929 --> 01:00:32.909 with a fifth grader in Edina, Minnesota struggling 01:00:32.909 --> 01:00:38.320 with comprehension. Human beings are not standardized 01:00:38.320 --> 01:00:41.420 products. Readers are not standardized products. 01:00:41.679 --> 01:00:46.519 Students are not standardized products. Again, 01:00:46.539 --> 01:00:50.960 when you read the research, skip the literature, 01:00:52.039 --> 01:00:58.570 look for the question. and these are the big 01:00:58.570 --> 01:01:02.050 three. All right, I have seen that this is 60 01:01:02.050 --> 01:01:04.909 minutes. I am going to stop my presentation here. 01:01:05.190 --> 01:01:08.929 I will host this presentation on my sub stack, 01:01:09.030 --> 01:01:12.210 the reading instruction show, and I'll post the 01:01:12.210 --> 01:01:17.949 rest of this. Those who are still here, I thank 01:01:17.949 --> 01:01:20.530 you for listening. I am Andy Johnson.