1
00:00:00,000 --> 00:00:02,600
This is the Reading Instruction Show.

2
00:00:02,600 --> 00:00:04,280
I'm your host, Dr. Andy Johnson.

3
00:00:04,280 --> 00:00:13,000
Today is the first in a two-part series of podcasts looking at documenting growth and

4
00:00:13,000 --> 00:00:18,000
monitoring progress for writing or assessing writing.

5
00:00:18,000 --> 00:00:21,880
Now, I want to start out with this idea.

6
00:00:21,880 --> 00:00:24,960
A pig doesn't get heavier by weighing it.

7
00:00:24,960 --> 00:00:29,240
A pig doesn't get heavier by weighing it.

8
00:00:29,240 --> 00:00:34,960
readers don't become better readers and writers by assessing them.

9
00:00:34,960 --> 00:00:40,200
There's this idea that we can assess our way to reading and writing proficiency.

10
00:00:40,200 --> 00:00:47,560
And that's a kooky, wacky, nutty, zany idea that we must divest ourselves out.

11
00:00:47,560 --> 00:00:51,040
So let's start with the why of assessment.

12
00:00:51,040 --> 00:00:55,600
The why questions don't get asked often enough.

13
00:00:55,600 --> 00:00:58,560
Why do we assess?

14
00:00:58,560 --> 00:01:00,280
For what purpose?

15
00:01:00,280 --> 00:01:03,880
Now in terms of writing, I hope the answer is this.

16
00:01:03,880 --> 00:01:12,680
I hope the answer is we assess so that we can help students become better writers.

17
00:01:12,680 --> 00:01:15,880
If not, we're wasting our time.

18
00:01:15,880 --> 00:01:23,480
We assess to determine strengths and to build upon weaknesses.

19
00:01:23,480 --> 00:01:29,920
We assess to document growth, but ultimately we assess to find out what we can do to help

20
00:01:29,920 --> 00:01:32,280
students become better writers.

21
00:01:32,280 --> 00:01:43,440
Now, to offman, assessment is used to, quote, hold teachers accountable, unquote, ug.

22
00:01:43,440 --> 00:01:50,360
Now why is it that some groups continue this war against teachers and insist on the deep

23
00:01:50,360 --> 00:01:54,400
professionalization of education?

24
00:01:54,400 --> 00:01:58,540
Hold teachers accountable.

25
00:01:58,540 --> 00:02:04,560
We want intelligent creative teachers to enter the field, but then we treat them like young

26
00:02:04,560 --> 00:02:07,120
wayward adolescents.

27
00:02:07,120 --> 00:02:08,920
We don't trust them.

28
00:02:08,920 --> 00:02:12,480
We think we need to hold them accountable.

29
00:02:12,480 --> 00:02:17,020
We tell them what they must teach and how they must teach it.

30
00:02:17,020 --> 00:02:21,880
We assign standards and measures that must be used.

31
00:02:21,880 --> 00:02:24,840
Maybe we should give them a curfew as well.

32
00:02:24,840 --> 00:02:25,840
That's it.

33
00:02:25,840 --> 00:02:31,920
Let's give all teachers a curfew and a mandatory bedtime and restrict their TV watching and

34
00:02:31,920 --> 00:02:36,840
phone access if test scores fall below a certain percentage.

35
00:02:36,840 --> 00:02:39,000
Yeah, that's the ticket.

36
00:02:39,000 --> 00:02:41,800
Hold them accountable.

37
00:02:41,800 --> 00:02:51,400
However, what happens when groups continually try to hold teachers accountable?

38
00:02:51,400 --> 00:02:54,640
Teachers teach to the accountability measure.

39
00:02:54,640 --> 00:02:57,720
In other words, they teach to the test.

40
00:02:57,720 --> 00:03:00,240
I mean, who wouldn't?

41
00:03:00,240 --> 00:03:06,720
The result of this accountability nonsense is that instead of testing what is taught,

42
00:03:06,720 --> 00:03:09,360
teachers teach what is tested.

43
00:03:09,360 --> 00:03:16,000
And this is called ask backwards in some circles, other clownery in other circles.

44
00:03:16,000 --> 00:03:23,360
With such ask backward clownery, the curriculum becomes narrow, focusing only on that which

45
00:03:23,360 --> 00:03:29,440
can be measured and quantified by an external entity.

46
00:03:29,440 --> 00:03:32,560
So what doesn't work?

47
00:03:32,560 --> 00:03:34,960
What about standardized tests for writing?

48
00:03:34,960 --> 00:03:39,320
That's the topic of this two-part series assessing writing.

49
00:03:39,320 --> 00:03:44,360
Standardized tests are relatively easy to administer and score.

50
00:03:44,360 --> 00:03:47,560
Takes an hour at most, maybe two.

51
00:03:47,560 --> 00:03:51,520
Whole bunch of students can sit down all at the same time.

52
00:03:51,520 --> 00:03:56,560
And within a short amount of time, the test is complete.

53
00:03:56,560 --> 00:04:03,680
And a norm reference test enables schools to compare each student to a national norm.

54
00:04:03,680 --> 00:04:05,520
This is a wonderful idea.

55
00:04:05,520 --> 00:04:06,520
Yes.

56
00:04:06,520 --> 00:04:09,280
No.

57
00:04:09,280 --> 00:04:15,560
Standardized tests can measure spelling, punctuation, and grammar using multiple choice

58
00:04:15,560 --> 00:04:21,320
or true false questions, but that's not writing.

59
00:04:21,320 --> 00:04:23,880
So let's look at writing tests.

60
00:04:23,880 --> 00:04:28,000
These writing tests have been around for a while.

61
00:04:28,000 --> 00:04:32,920
Here generally students are given a prompt and a piece of paper to do some pre-writing

62
00:04:32,920 --> 00:04:35,720
thinking before writing.

63
00:04:35,720 --> 00:04:42,200
And then they have a time limit to write anywhere from 15 to 30 minutes usually.

64
00:04:42,200 --> 00:04:49,760
And then these are either computer scored or sent off to be rated by scored, trained,

65
00:04:49,760 --> 00:04:53,440
rated and scored by trained raters.

66
00:04:53,440 --> 00:04:56,800
Now Richie Rootman said this about trained raters.

67
00:04:56,800 --> 00:05:02,800
She said, in general, scores are poorly paid, have minimal training, are given very little

68
00:05:02,800 --> 00:05:09,040
time to read each sample, and are expected to read student papers for many hours at a

69
00:05:09,040 --> 00:05:10,040
stretch.

70
00:05:10,040 --> 00:05:16,920
And these people are determining your success as teachers of writing.

71
00:05:16,920 --> 00:05:22,720
Now these are better than standardized tests, the writing tests, but still not good because

72
00:05:22,720 --> 00:05:27,160
it's an artificial writing situation.

73
00:05:27,160 --> 00:05:31,000
It's not what real writers do when they write, hopefully.

74
00:05:31,000 --> 00:05:36,920
Thus, with the writing test, it's a validity question.

75
00:05:36,920 --> 00:05:44,120
Now it would be very easy to teach students to perform like trained seals at a circus

76
00:05:44,120 --> 00:05:48,800
and make writing scores go up on these artificial writing measures.

77
00:05:48,800 --> 00:05:51,920
Absolutely, it would be easy.

78
00:05:51,920 --> 00:05:57,880
You simply break this artificial writing down into steps and teach each step with guided

79
00:05:57,880 --> 00:06:00,200
and independent practice.

80
00:06:00,200 --> 00:06:05,680
Then you'd create a lot of artificial writing experiences with artificial writing prompts

81
00:06:05,680 --> 00:06:10,840
and artificial time limits, and you'd have artificial writing practice, and you'd spend

82
00:06:10,840 --> 00:06:15,920
time talking about your artificial writing and how to get better at artificial writing.

83
00:06:15,920 --> 00:06:23,240
And maybe we could teach students to balance the ball on their nose at the same time.

84
00:06:23,240 --> 00:06:25,680
But the question is this.

85
00:06:25,680 --> 00:06:33,600
Do you really want students to perform like trained seals on some cockamamie writing assessment?

86
00:06:33,600 --> 00:06:39,520
Or do you want students to be able to write and think?

87
00:06:39,520 --> 00:06:46,760
Now the way some schools approach assessment of writing, I sometimes wonder.

88
00:06:46,760 --> 00:06:50,240
Seems like they want the trained seals.

89
00:06:50,240 --> 00:06:52,600
Now let's look at traits of writing.

90
00:06:52,600 --> 00:07:00,320
What if we define the traits of good writing and then taught and assessed each of the traits?

91
00:07:00,320 --> 00:07:02,440
Now that's been done.

92
00:07:02,440 --> 00:07:07,560
In her book Six Plus One, Traits of Writing, Ruth Cullum recommends teaching writing by

93
00:07:07,560 --> 00:07:13,960
focusing on the six plus one or seven traits.

94
00:07:13,960 --> 00:07:19,680
Her emphasis is on teaching these traits and using rubrics to assess.

95
00:07:19,680 --> 00:07:21,400
Now here are the traits.

96
00:07:21,400 --> 00:07:25,360
Number one, ideas, meeting and development of the message.

97
00:07:25,360 --> 00:07:29,000
Number two, organization, the internal structure of the piece.

98
00:07:29,000 --> 00:07:34,120
Number three, voice, the tone of the piece and the personal stamp.

99
00:07:34,120 --> 00:07:38,200
Word choice, the specific vocabulary the writer uses.

100
00:07:38,200 --> 00:07:41,560
Sentence fluency, the way the words and phrases flow.

101
00:07:41,560 --> 00:07:47,240
Conventions, that's the mechanical correctness and presentation, the overall appearance.

102
00:07:47,240 --> 00:07:53,360
These are the seven traits that Cullum says are so very important.

103
00:07:53,360 --> 00:07:55,400
And she creates rubrics.

104
00:07:55,400 --> 00:07:57,680
This approach is based on rubrics.

105
00:07:57,680 --> 00:08:05,040
Each rubric has seven traits, the seven traits I just said and descriptions of five levels

106
00:08:05,040 --> 00:08:08,200
of each trait.

107
00:08:08,200 --> 00:08:14,640
You simply look at students writing and assign them a number for each trace based on the

108
00:08:14,640 --> 00:08:17,480
rubric description.

109
00:08:17,480 --> 00:08:20,520
Better writing through rubrics, yes?

110
00:08:20,520 --> 00:08:22,320
No.

111
00:08:22,320 --> 00:08:29,400
The problem with the six plus one traits is that the focus is on the product or the traits

112
00:08:29,400 --> 00:08:33,240
and not on the process.

113
00:08:33,240 --> 00:08:36,720
The traits then become the writing program.

114
00:08:36,720 --> 00:08:40,560
You're teaching traits and not process.

115
00:08:40,560 --> 00:08:47,800
But who was it that decided that these were the traits that were important and why?

116
00:08:47,800 --> 00:08:53,480
Do they reflect the real world writing that real world writers write?

117
00:08:53,480 --> 00:08:56,320
Now, I'm a real writer.

118
00:08:56,320 --> 00:08:57,320
I really am.

119
00:08:57,320 --> 00:08:59,120
I'm real and I'm a writer.

120
00:08:59,120 --> 00:09:01,360
Thus, I'm a real writer.

121
00:09:01,360 --> 00:09:07,520
And as I'm writing this podcast, I'm looking back at Cullum's six plus one traits.

122
00:09:07,520 --> 00:09:12,160
And you know what?

123
00:09:12,160 --> 00:09:18,240
In my professional life, writing letters and memos and emails and articles and reports

124
00:09:18,240 --> 00:09:27,040
and books, they just don't fit into this other worldly list of traits.

125
00:09:27,040 --> 00:09:28,040
Voice and tone.

126
00:09:28,040 --> 00:09:34,200
Now, in most professional writing, the goal is objective, academic or professional voice,

127
00:09:34,200 --> 00:09:35,200
word choice.

128
00:09:35,200 --> 00:09:36,200
What does that mean?

129
00:09:36,200 --> 00:09:39,720
They simply were looking for things they could measure.

130
00:09:39,720 --> 00:09:43,400
And they said, aha, we can measure these traits.

131
00:09:43,400 --> 00:09:49,440
Let's get them up on a rubric and we can teach and we can numberize our kids.

132
00:09:49,440 --> 00:09:52,880
We've been over rubricized.

133
00:09:52,880 --> 00:09:57,080
The basis of a rubric is this.

134
00:09:57,080 --> 00:10:01,960
You look at the product or performance you're trying to teach.

135
00:10:01,960 --> 00:10:07,320
Then you define three to six traits of a good product or performance.

136
00:10:07,320 --> 00:10:12,080
And then you describe three to five levels of each trait.

137
00:10:12,080 --> 00:10:16,680
For example, given this trait, this is what a rating four would look like, a rating three,

138
00:10:16,680 --> 00:10:19,520
a rating two, a rating one and zero.

139
00:10:19,520 --> 00:10:24,840
And then you compare the product or performance to each trait and find the number level that

140
00:10:24,840 --> 00:10:26,720
seems to match.

141
00:10:26,720 --> 00:10:29,160
Simple as that.

142
00:10:29,160 --> 00:10:39,040
Now, there's nothing wrong with using a rubric here and there, but we've become over rubricized.

143
00:10:39,040 --> 00:10:45,680
We've got ourselves a bad case of rubritis and the stinking rubrics do little to move

144
00:10:45,680 --> 00:10:54,280
us forward in our teaching and little to move students forward in their writing.

145
00:10:54,280 --> 00:11:02,840
But since the stinking rubrics generate stinking numbers, the stinking rubrics continue to exist.

146
00:11:02,840 --> 00:11:10,640
Now I want you to know you youngsters out there, there was educational life before rubrics

147
00:11:10,640 --> 00:11:14,720
and it was a wonderful life.

148
00:11:14,720 --> 00:11:18,160
We taught stuff and children learn stuff.

149
00:11:18,160 --> 00:11:22,800
But then the number monkeys began to creep into education.

150
00:11:22,800 --> 00:11:28,840
Nothing exists, they said, if it can't be quantifying.

151
00:11:28,840 --> 00:11:35,000
So the number monkeys threw down their bananas and invented the rubric.

152
00:11:35,000 --> 00:11:40,320
Rubrics enable them to put numbers to things and enumerate our students and quantify their

153
00:11:40,320 --> 00:11:42,400
learning.

154
00:11:42,400 --> 00:11:49,000
They could point their little monkey fingers and say, that's an 18 and that's a 16 and

155
00:11:49,000 --> 00:11:53,720
18 is better than 16, so that one is better than that one.

156
00:11:53,720 --> 00:11:58,760
And then they'd screech and shake their monkey arms above their head.

157
00:11:58,760 --> 00:12:04,960
And if the average writing rubric scores in class A was 21.3 and the average writing rubric

158
00:12:04,960 --> 00:12:11,680
score in class B was 18.7, the number monkeys would conclude that teacher A is doing a better

159
00:12:11,680 --> 00:12:15,800
job of teaching than teacher B.

160
00:12:15,800 --> 00:12:19,680
We've got the whole teacher B accountable, see?

161
00:12:19,680 --> 00:12:27,640
We've got to get those test scores, those 18s up to 21s, everyone has to be a 21.

162
00:12:27,640 --> 00:12:30,960
So what's wrong with the stinkin' rubrics?

163
00:12:30,960 --> 00:12:34,160
Every form of assessment has strengths and limitations.

164
00:12:34,160 --> 00:12:37,760
And here are some of the limitations with using rubrics.

165
00:12:37,760 --> 00:12:43,480
Number one, they are too complex and cumbersome to be of any use.

166
00:12:43,480 --> 00:12:52,120
It takes time to figure out what exactly constitutes each level of performance with each trait.

167
00:12:52,120 --> 00:12:57,520
Then evaluating and deciding if it's one level or another, a two or three, a three or four,

168
00:12:57,520 --> 00:12:59,960
takes additional time.

169
00:12:59,960 --> 00:13:06,800
They take time to create and time to use and the cost reward here seems high in terms of

170
00:13:06,800 --> 00:13:08,880
use and time.

171
00:13:08,880 --> 00:13:13,560
Number two, they give the illusion of objectivity.

172
00:13:13,560 --> 00:13:19,560
We think because it's described, we can just say, there it is, it's definitely a two or

173
00:13:19,560 --> 00:13:21,400
definitely a three.

174
00:13:21,400 --> 00:13:25,480
Just match the paper to the trait, right?

175
00:13:25,480 --> 00:13:30,880
But often a paper will meet some of level three and some of level two and then do you

176
00:13:30,880 --> 00:13:32,960
give it a 2.5?

177
00:13:32,960 --> 00:13:37,040
Who defines the traits and the levels of each trait?

178
00:13:37,040 --> 00:13:44,240
The rubric subjectivity is wrapped in an illusion of objectivity.

179
00:13:44,240 --> 00:13:50,860
The subjectivity of the rubric is wrapped in the illusion of objectivity.

180
00:13:50,860 --> 00:13:56,680
Number three, the focus is on the traits or writing product and not on the writing process.

181
00:13:56,680 --> 00:14:02,360
Number four, a rubric often creates a distorted version of the performance they're trying

182
00:14:02,360 --> 00:14:04,000
to capture.

183
00:14:04,000 --> 00:14:11,040
In writing, rubrics never capture the full range of what writers do.

184
00:14:11,040 --> 00:14:18,660
And number five, rubrics get students focusing on the micro instead of the macro.

185
00:14:18,660 --> 00:14:23,800
It's assumed that by putting all the little micro rubric traits together that they'll

186
00:14:23,800 --> 00:14:28,260
have an excellent macro product or performance.

187
00:14:28,260 --> 00:14:37,160
This is called part to whole learning, but this is not how we learn complex things most

188
00:14:37,160 --> 00:14:40,000
effectively.

189
00:14:40,000 --> 00:14:45,760
Whole to part learning has been shown to be much more effective for learning complex skills

190
00:14:45,760 --> 00:14:50,600
such as writing than part to whole.

191
00:14:50,600 --> 00:14:53,160
All right, this has been the reading instruction show.

192
00:14:53,160 --> 00:14:59,600
We've been looking at some assessment, some ways to document students growth and progress

193
00:14:59,600 --> 00:15:01,480
as writers.

194
00:15:01,480 --> 00:15:08,280
In the next podcast, we'll look at some better ways, some alternatives that make sense that

195
00:15:08,280 --> 00:15:13,560
are more valid and reliable and help students growth as writers.

196
00:15:13,560 --> 00:15:23,560
Imagine that.