1
00:00:00,000 --> 00:00:04,000
Welcome to the Improving Development Evaluation Podcast.

2
00:00:04,000 --> 00:00:07,360
I'm your host David Wand and in this trailer

3
00:00:07,360 --> 00:00:11,040
we're going to give you some more detail about what the podcast is actually

4
00:00:11,040 --> 00:00:13,840
trying to do. We're going to talk about three things.

5
00:00:13,840 --> 00:00:17,600
One of the content of each episode including who the guests will be on the

6
00:00:17,600 --> 00:00:20,640
podcast. Secondly we're going to talk about

7
00:00:20,640 --> 00:00:24,240
season one about what we plan to cover in season one

8
00:00:24,240 --> 00:00:28,880
which includes 12 episodes and then finally we're going to talk about some

9
00:00:28,880 --> 00:00:32,160
ideas about how you the listener may want to

10
00:00:32,160 --> 00:00:35,440
get more involved with the podcast in addition to just

11
00:00:35,440 --> 00:00:38,720
hopefully listening to it. So let's get right into it.

12
00:00:38,720 --> 00:00:42,240
So the content of each episode we're going to have two parts.

13
00:00:42,240 --> 00:00:46,080
In part one the host is going to introduce

14
00:00:46,080 --> 00:00:50,400
a new international development organization for each episode

15
00:00:50,400 --> 00:00:55,280
and in this part one I'm going to talk about the project

16
00:00:55,280 --> 00:00:58,320
that this new international development organization

17
00:00:58,320 --> 00:01:02,800
has been funded to deliver in a developing country by the government of

18
00:01:02,800 --> 00:01:04,960
Canada. So I'm going to briefly describe the

19
00:01:04,960 --> 00:01:08,560
services the project delivers, the cost of the project to the Canadian

20
00:01:08,560 --> 00:01:13,040
taxpayer and most provocatively and most importantly

21
00:01:13,040 --> 00:01:17,280
I'm going to conclude at the end of part one that we've got a problem.

22
00:01:17,280 --> 00:01:22,800
That is the project has failed to achieve its outcomes and you may be

23
00:01:22,800 --> 00:01:26,960
wondering how I'm going to get there. First of all what do we mean by

24
00:01:26,960 --> 00:01:30,640
outcomes? So for example a project trains people on how to

25
00:01:30,640 --> 00:01:36,000
grow peanuts and how to sell peanuts. The outcomes you would expect from that

26
00:01:36,000 --> 00:01:39,360
training would be that those people would learn how to

27
00:01:39,360 --> 00:01:42,080
grow peanuts by having increased skill and knowledge

28
00:01:42,080 --> 00:01:46,480
levels as well as increased skill and knowledge levels on how

29
00:01:46,480 --> 00:01:49,680
to sell those peanuts.

30
00:01:49,920 --> 00:01:52,960
And I've come to the conclusion that they have

31
00:01:52,960 --> 00:01:58,080
not been able to in the project claim that those outcomes have been achieved.

32
00:01:58,080 --> 00:02:05,120
Why? Well it's because I've acquired through a access to information request

33
00:02:05,120 --> 00:02:09,040
a PMF, a performance measurement framework.

34
00:02:09,040 --> 00:02:12,400
These are not available to the public. You have to first make

35
00:02:12,400 --> 00:02:16,800
an access to information request and I have acquired 12 of them.

36
00:02:16,800 --> 00:02:21,360
So in part one what I'm going to do is briefly describe

37
00:02:21,360 --> 00:02:25,040
how the PMF or performance measurement framework fails

38
00:02:25,040 --> 00:02:32,240
to adequately measure these outcomes. So they fall into five problem areas

39
00:02:32,240 --> 00:02:35,840
and the first problem is we have to look at the outcome indicators that are

40
00:02:35,840 --> 00:02:39,040
trying to measure the outcomes like I've just given an example.

41
00:02:39,040 --> 00:02:42,880
So in the example and the first problem would be

42
00:02:42,880 --> 00:02:46,080
people show up for the training on how to grow peanuts and how to sell those

43
00:02:46,080 --> 00:02:48,480
peanuts but they're never measured on their

44
00:02:48,480 --> 00:02:52,240
skill and knowledge levels. Instead they just take attendance and

45
00:02:52,240 --> 00:02:55,520
everybody goes home. So that's the first problem. The second

46
00:02:55,520 --> 00:02:59,600
problem they could have is they show up they measure their skill

47
00:02:59,600 --> 00:03:03,280
levels but they don't measure them directly.

48
00:03:03,280 --> 00:03:10,560
Instead they ask them how do they feel about their ability to grow peanuts

49
00:03:10,560 --> 00:03:17,280
and their ability to sell those peanuts after getting the training on how to

50
00:03:17,280 --> 00:03:22,400
grow peanuts and how to sell peanuts. They're asking them how they feel

51
00:03:22,400 --> 00:03:25,920
about their ability. They're not being measured objectively like through a test

52
00:03:25,920 --> 00:03:30,480
or a certification process or whatever. So that's the second problem. The third

53
00:03:30,480 --> 00:03:35,520
problem is sometimes the outcome indicator doesn't measure the outcome at

54
00:03:35,520 --> 00:03:37,760
all. So let's say expected outcome from the

55
00:03:37,760 --> 00:03:40,640
training on how to grow and sell peanuts is

56
00:03:40,640 --> 00:03:43,920
an increase in income from selling those peanuts.

57
00:03:43,920 --> 00:03:51,120
Sounds logical. Problem is they don't say show me the money. Let's see your bank

58
00:03:51,120 --> 00:03:53,280
account. Let's see your bank balance from selling

59
00:03:53,280 --> 00:03:58,480
those peanuts. Instead what they do is they say show me a savings plan

60
00:03:58,480 --> 00:04:01,520
that you have. Well that's not really related.

61
00:04:01,520 --> 00:04:06,800
They've got to show direct measurement of the income. So that's the third problem.

62
00:04:06,800 --> 00:04:12,720
The fourth problem is maybe they show up they properly measure them objectively

63
00:04:12,720 --> 00:04:16,560
on their skill and knowledge levels on how to sell peanuts and how to

64
00:04:16,560 --> 00:04:24,480
grow peanuts. But they only do it once before the training and they do it once

65
00:04:24,480 --> 00:04:28,480
after the training. So this is a problem also

66
00:04:28,480 --> 00:04:32,080
because let's say they measure them as a group

67
00:04:32,080 --> 00:04:34,880
on their skill and knowledge levels on how to grow peanuts and how to sell

68
00:04:34,880 --> 00:04:41,280
peanuts before the training starts. And then they get their test scores back

69
00:04:41,280 --> 00:04:46,080
and some of them say oh jeez I'm really bad. My knowledge levels

70
00:04:46,080 --> 00:04:49,920
are really low on how to grow peanuts and how to sell

71
00:04:49,920 --> 00:04:52,080
peanuts. I think I'm going to go down the road

72
00:04:52,080 --> 00:04:56,960
and have a chat with a very successful peanut farmer and ask him or her

73
00:04:56,960 --> 00:05:01,280
how do you do it. Or maybe they'll go online

74
00:05:01,280 --> 00:05:04,960
and figure it out if they have access to the internet through a YouTube

75
00:05:04,960 --> 00:05:09,280
video on how to grow and sell peanuts. So then they after the training

76
00:05:09,280 --> 00:05:13,280
they measure them as a group once and guess what?

77
00:05:13,280 --> 00:05:18,160
Their test scores go up. The group has improved

78
00:05:18,160 --> 00:05:22,400
in their knowledge and skill levels which you would expect.

79
00:05:22,400 --> 00:05:26,160
Problem is we don't know why. Could be because of the peanut farmers they were

80
00:05:26,160 --> 00:05:29,120
talking to down the road before they got the last

81
00:05:29,120 --> 00:05:31,840
test. Or they could be figuring it out on the

82
00:05:31,840 --> 00:05:36,320
internet all by themselves. So that's problem number four. We just

83
00:05:36,320 --> 00:05:41,200
don't know if the training was responsible for the increases

84
00:05:41,200 --> 00:05:45,120
in knowledge and skill levels that they actually observed and measured.

85
00:05:45,120 --> 00:05:50,240
And then finally problem number five which is let's say they do it

86
00:05:50,240 --> 00:05:53,920
perfectly. They measure them before during and

87
00:05:53,920 --> 00:05:57,040
after the training. So you can see this nice

88
00:05:57,040 --> 00:06:00,480
trend of increases in skill levels and

89
00:06:00,480 --> 00:06:04,880
knowledge level. They also watch their incomes. They

90
00:06:04,880 --> 00:06:08,880
look at their bank accounts and they notice oh this is great. Their

91
00:06:08,880 --> 00:06:12,960
incomes from selling peanuts has gone up due to the training

92
00:06:12,960 --> 00:06:17,680
before during and after. It's gradually gone up which is pretty good.

93
00:06:17,680 --> 00:06:24,720
But problem is there's no comparison group of people

94
00:06:24,720 --> 00:06:28,720
who also grow peanuts but did not attend the training.

95
00:06:28,720 --> 00:06:32,240
So there could be another group of people in some other part of the country

96
00:06:32,240 --> 00:06:35,920
that are doing just fine teaching themselves how to grow peanuts

97
00:06:35,920 --> 00:06:39,360
and how to sell those peanuts by talking to other peanut farmers

98
00:06:39,360 --> 00:06:42,560
or whatever. So the problem there is we don't know

99
00:06:42,560 --> 00:06:47,360
if the project training that you paid for as a Canadian taxpayer

100
00:06:47,360 --> 00:06:52,320
was actually responsible for the increase in skill and knowledge

101
00:06:52,320 --> 00:06:55,280
levels of how to grow and how to sell peanuts.

102
00:06:55,280 --> 00:07:00,320
And also the increase in incomes from selling those peanuts. We just don't know.

103
00:07:00,320 --> 00:07:04,480
It could be equal or it could be even lower even though we're seeing increases

104
00:07:04,480 --> 00:07:08,560
in incomes and knowledge and skill levels. It could be lower than the group

105
00:07:08,560 --> 00:07:12,880
that never attended the project training. So that's problem number five.

106
00:07:12,880 --> 00:07:16,400
So the podcast is going to focus in part one

107
00:07:16,400 --> 00:07:21,440
on a summary of those problems for the indicators for the performance

108
00:07:21,440 --> 00:07:23,680
measurement framework and where they fall.

109
00:07:23,680 --> 00:07:28,800
Now there are so many outcome indicators in these projects.

110
00:07:28,800 --> 00:07:34,000
We will not have time to go through all of them but I will provide a summary

111
00:07:34,000 --> 00:07:38,000
of all of those indicators as to which ones

112
00:07:38,000 --> 00:07:40,880
fall into one of those five problem areas.

113
00:07:40,880 --> 00:07:44,160
So at the end of part one I will summarize

114
00:07:44,160 --> 00:07:48,000
the PMF or performance measurement framework and I will just cut to the

115
00:07:48,000 --> 00:07:52,160
chase here. I will be concluding that the project

116
00:07:52,160 --> 00:07:56,400
in its current form with its current performance measurement framework

117
00:07:56,400 --> 00:08:02,880
is flawed and cannot support the international development organization

118
00:08:02,880 --> 00:08:08,800
making the claim that their project services are achieving their project

119
00:08:08,800 --> 00:08:12,000
outcomes. That's why we have this podcast to

120
00:08:12,000 --> 00:08:16,480
bring this out and provide solutions on how to improve

121
00:08:16,480 --> 00:08:21,200
development evaluation. So at the very end of part one I'm also going to invite

122
00:08:21,200 --> 00:08:25,200
the international development organization to attend part two

123
00:08:25,200 --> 00:08:29,520
of the podcast where they will respond to this critique.

124
00:08:29,520 --> 00:08:32,800
But this is only a starting point for me as the host.

125
00:08:32,800 --> 00:08:40,400
I also will be inviting in part two on the podcast at least one and

126
00:08:40,400 --> 00:08:45,280
possibly two evaluation experts to also respond to my critique

127
00:08:45,280 --> 00:08:52,000
and provide alternatives to how to improve the evaluation of the project.

128
00:08:52,000 --> 00:08:56,080
In part two it will take about 40 minutes so we won't be able to go through

129
00:08:56,080 --> 00:09:00,560
all of the outcome indicators but we hopefully will be able to select a few

130
00:09:00,560 --> 00:09:05,360
from each of the outcomes of the project. Usually these projects have five or six

131
00:09:05,360 --> 00:09:08,720
outcomes and then for each outcome they have two or three

132
00:09:08,720 --> 00:09:12,560
indicators. We'll probably just have to pick a few of them

133
00:09:12,560 --> 00:09:16,400
and go through them to make sure we cover all of the outcomes.

134
00:09:16,400 --> 00:09:19,600
And at the end of part two we're going to

135
00:09:19,600 --> 00:09:22,560
answer the following question the host that's me

136
00:09:22,560 --> 00:09:26,160
and the evaluation experts and that is the question

137
00:09:26,160 --> 00:09:29,680
is the project in its current form with its current performance measurement

138
00:09:29,680 --> 00:09:34,400
framework which they are required to follow and develop before they get

139
00:09:34,400 --> 00:09:40,320
funding from the government of Canada is it adequate for the international

140
00:09:40,320 --> 00:09:43,200
development organization to make the claim

141
00:09:43,200 --> 00:09:46,720
that the project is achieving its outcomes. My answer

142
00:09:46,720 --> 00:09:50,880
right now is no. There may be a few of those 12

143
00:09:50,880 --> 00:09:54,240
organizations that you'll see in season one where

144
00:09:54,240 --> 00:09:57,520
maybe some of the indicators get it right

145
00:09:57,520 --> 00:10:01,680
but most of them fall into one of those five problem areas and for each episode

146
00:10:01,680 --> 00:10:06,880
I'm going to post on a website the project's performance

147
00:10:06,880 --> 00:10:11,040
measurement framework as well as a summary of how each of

148
00:10:11,040 --> 00:10:14,640
the outcome indicators in that performance measurement framework falls

149
00:10:14,640 --> 00:10:19,120
into one of those five problem areas.

150
00:10:19,120 --> 00:10:24,880
So just to recap in part two of the episode I will be encouraging the

151
00:10:24,880 --> 00:10:28,240
international development organization to respond to my critique

152
00:10:28,240 --> 00:10:34,640
which includes offering alternative solutions to this problem of how to

153
00:10:34,640 --> 00:10:38,800
improve their evaluation of their project. The

154
00:10:38,800 --> 00:10:43,120
idea is to get a discussion going about how to improve

155
00:10:43,120 --> 00:10:46,320
development evaluation.

156
00:10:46,800 --> 00:10:50,560
So you might be wondering what we plan to cover in season one like I mentioned

157
00:10:50,560 --> 00:10:54,240
we have 12 episodes introducing 12 international development

158
00:10:54,240 --> 00:10:58,720
organizations each one having a project along with its performance measurement

159
00:10:58,720 --> 00:11:03,200
framework and so these projects their cost to the

160
00:11:03,200 --> 00:11:08,000
Canadian taxpayer all 12 of them is around 177 million

161
00:11:08,000 --> 00:11:10,480
dollars.

162
00:11:10,480 --> 00:11:13,520
What sort of services are these projects delivering?

163
00:11:13,520 --> 00:11:17,120
Well some of the projects we'll be covering

164
00:11:17,120 --> 00:11:21,760
focus a lot on training how to grow peanuts

165
00:11:21,760 --> 00:11:26,960
how to tutor midwives how to change men's attitudes to support women's rights

166
00:11:26,960 --> 00:11:31,040
and equality and raising awareness on accessing

167
00:11:31,040 --> 00:11:36,000
reproductive health services and gender-based violence services.

168
00:11:36,000 --> 00:11:39,920
These projects are being delivered in West Africa

169
00:11:39,920 --> 00:11:46,160
in particular Ghana, Ivory Coast, Senegal, Burkina Faso and in East Africa

170
00:11:46,160 --> 00:11:49,840
in Somalia, South Sudan, Tanzania and Kenya

171
00:11:49,840 --> 00:11:54,000
as well as in Asia, Bangladesh and Vietnam.

172
00:11:54,000 --> 00:11:58,480
And finally you may be wondering okay you're going to listen to the podcast

173
00:11:58,480 --> 00:12:02,480
hopefully but in addition to that you may want to get a bit more involved

174
00:12:02,480 --> 00:12:06,240
as you can sense from the podcast there's a bit of

175
00:12:06,240 --> 00:12:11,680
advocacy involved where we're trying to advocate for improvements in development

176
00:12:11,680 --> 00:12:15,680
evaluation. The government of Canada spends over

177
00:12:15,680 --> 00:12:21,040
four billion dollars a year on development projects

178
00:12:21,040 --> 00:12:26,480
all around the world. You may want to send me an email

179
00:12:26,480 --> 00:12:29,920
regarding this podcast your comments and suggestions

180
00:12:29,920 --> 00:12:35,120
on how the podcast can be improved. My email address is evaluate

181
00:12:35,120 --> 00:12:40,080
canadaaid at gmail.com and finally I'm always looking for

182
00:12:40,080 --> 00:12:44,320
evaluation experts who may wish to be on the evaluation

183
00:12:44,320 --> 00:12:49,360
panel in the podcast part two.

184
00:12:50,480 --> 00:12:55,840
And you can email me and I will be happy to send you the performance measurement

185
00:12:55,840 --> 00:13:00,160
frameworks for season one. We've got four

186
00:13:00,160 --> 00:13:03,520
already booked with evaluation experts but I can send

187
00:13:03,520 --> 00:13:06,800
you the remaining eight performance measurement frameworks.

188
00:13:06,800 --> 00:13:11,040
And for season two we will be getting an additional five

189
00:13:11,040 --> 00:13:16,080
performance measurement frameworks. And if you're really interested

190
00:13:16,080 --> 00:13:20,240
and you want a particular project for us to look at

191
00:13:20,240 --> 00:13:24,400
on the podcast you can go to your search engine and enter

192
00:13:24,400 --> 00:13:30,320
global affairs canada project browser and up will come the platform where all

193
00:13:30,320 --> 00:13:33,520
of the projects thousands of them are listed.

194
00:13:33,520 --> 00:13:39,680
And you can select a particular project that you have an interest in and you

195
00:13:39,680 --> 00:13:42,480
will notice that even though there's a project

196
00:13:42,480 --> 00:13:46,080
browser there's no performance measurement framework.

197
00:13:46,080 --> 00:13:50,160
It's not available and so ideally what we want to see

198
00:13:50,160 --> 00:13:53,600
is that performance measurement framework

199
00:13:53,600 --> 00:13:57,680
available on the project browser platform as well as the international

200
00:13:57,680 --> 00:14:01,040
development organizations putting their performance measurement

201
00:14:01,040 --> 00:14:06,400
framework for each of their projects on their websites.

202
00:14:06,400 --> 00:14:10,320
Ideally both the blank ones that we're discussing on the podcast

203
00:14:10,320 --> 00:14:14,240
but as well the data that comes in on the outcome

204
00:14:14,240 --> 00:14:18,560
indicators that they've listed. The irony of course is as you may have

205
00:14:18,560 --> 00:14:22,640
figured this out is from this podcast most of those

206
00:14:22,640 --> 00:14:29,360
outcome indicators are flawed so it wouldn't be very useful to use

207
00:14:29,360 --> 00:14:33,760
the data coming in from those outcome indicators.

208
00:14:33,760 --> 00:14:38,080
As the saying goes garbage in garbage out.

209
00:14:38,080 --> 00:14:41,680
And that's the big issue that I am trying to raise

210
00:14:41,680 --> 00:14:44,960
in this podcast. I look forward to hearing from you.

211
00:14:44,960 --> 00:14:52,240
Thank you for your time. Goodbye and stay tuned for episode

212
00:14:52,240 --> 00:14:56,240
one part one.