1
00:00:00,000 --> 00:00:05,680
I'm your host David Wand and in this episode 2 part 2 we're going to be talking about

2
00:00:05,680 --> 00:00:12,760
the Canadian Bureau for International Education, their project, cost of $5 million to the Canadian

3
00:00:12,760 --> 00:00:15,500
taxpayer in Canada.

4
00:00:15,500 --> 00:00:21,680
And just to give you a recap of part 1, I reviewed the 7 outcomes and the 11 outcome

5
00:00:21,680 --> 00:00:27,120
indicators for that project and came to the conclusion that the Canadian Bureau for International

6
00:00:27,120 --> 00:00:32,320
Education could not make the claim that they are properly measuring the outcomes using

7
00:00:32,320 --> 00:00:35,880
the outcome indicators in its performance measurement framework.

8
00:00:35,880 --> 00:00:41,600
I also contacted the Canadian Bureau for International Education inviting them to attend this podcast

9
00:00:41,600 --> 00:00:46,240
part 2 to respond to my critique of their performance measurement framework.

10
00:00:46,240 --> 00:00:52,780
However, they declined to attend this podcast and they also declined to send a written response

11
00:00:52,780 --> 00:00:55,240
to my critique.

12
00:00:55,240 --> 00:01:00,880
But fortunately today we also have with us an evaluation expert from Amman Jordan and

13
00:01:00,880 --> 00:01:09,040
her name is Hyatt Asgar and she holds a graduate diploma in public policy program evaluation

14
00:01:09,040 --> 00:01:11,560
from Carleton University in Canada.

15
00:01:11,560 --> 00:01:13,840
Welcome Hyatt, how are you?

16
00:01:13,840 --> 00:01:17,000
Thank you, hi David, glad to be with you today.

17
00:01:17,000 --> 00:01:20,280
Good, and we're also glad to hear from you.

18
00:01:20,280 --> 00:01:25,520
And what we're going to do now is go through each of the outcome statements one by one

19
00:01:25,520 --> 00:01:30,520
and then we will have a discussion about why their outcome indicators are not properly

20
00:01:30,520 --> 00:01:32,180
measuring those outcomes.

21
00:01:32,180 --> 00:01:39,080
So let's start with outcome number 1, increased leadership skills of female and male African

22
00:01:39,080 --> 00:01:44,920
leaders of tomorrow, scholars in public administration and public policy.

23
00:01:44,920 --> 00:01:49,840
Hyatt, what outcome indicator do you want to choose and talk about?

24
00:01:49,840 --> 00:01:55,480
Yeah, I'll speak about the indicator number 2, the percentage of schoolers reporting that

25
00:01:55,480 --> 00:02:01,920
leadership capacity building events have increased their ability to take on new leadership responsibilities

26
00:02:01,920 --> 00:02:04,120
in public administration and public policy.

27
00:02:04,120 --> 00:02:07,840
Tell us what's wrong with the indicator in measuring the outcome.

28
00:02:07,840 --> 00:02:14,840
So I'll share some observations from my personal opinion and I think what caught my attention

29
00:02:14,840 --> 00:02:19,180
for these indicators is that we are asking schoolers to report.

30
00:02:19,180 --> 00:02:26,680
So we're speaking about self-reporting, which in like you could see it in other monitoring

31
00:02:26,680 --> 00:02:34,640
and frameworks, it is used, but is that enough to have this self-reporting as a source of

32
00:02:34,640 --> 00:02:36,480
information alone?

33
00:02:36,480 --> 00:02:38,360
This is what is called the reporting bias.

34
00:02:38,360 --> 00:02:43,640
So you are asked the students to report on their ability, how they can see their leadership

35
00:02:43,640 --> 00:02:45,160
skills.

36
00:02:45,160 --> 00:02:52,320
You expect that people will be judging themselves, whether telling you the truth from their own

37
00:02:52,320 --> 00:02:56,680
perspective, whether underestimating themselves or overestimating themselves.

38
00:02:56,680 --> 00:03:01,480
So the self-reporting, I cannot say that it is wrong, it can be used, but it should be

39
00:03:01,480 --> 00:03:07,840
supported by other sources, other assessment methods, like for example, having a different

40
00:03:07,840 --> 00:03:14,220
measure to look at their leadership skills using some sort of leadership assessments.

41
00:03:14,220 --> 00:03:19,140
And here they are speaking about so many available leadership assessments worldwide that the

42
00:03:19,140 --> 00:03:21,400
organization may choose to select.

43
00:03:21,400 --> 00:03:27,760
Also, if they are focusing on a specific sub-factors or sub-leadership factors, they can build

44
00:03:27,760 --> 00:03:29,260
their own leadership assessment.

45
00:03:29,260 --> 00:03:36,380
But what is good about having a supporting measure that they can use a scientific approach

46
00:03:36,380 --> 00:03:40,740
and then they are also welcome, they could use the reporting as a second source and they

47
00:03:40,740 --> 00:03:42,720
compare the two together.

48
00:03:42,720 --> 00:03:45,240
So this is my first observation.

49
00:03:45,240 --> 00:03:47,020
Yeah, that's very good.

50
00:03:47,020 --> 00:03:48,020
Thank you.

51
00:03:48,020 --> 00:03:52,800
And just to recap, of the 11 outcome indicators in this Canadian Bureau for International

52
00:03:52,800 --> 00:03:58,400
Education Performance Measurement Framework, 11, 8 of them were having this problem of

53
00:03:58,400 --> 00:04:04,200
self-reporting bias and 2 others were having this problem of no comparison groups.

54
00:04:04,200 --> 00:04:06,320
And we'll get to those later too.

55
00:04:06,320 --> 00:04:09,520
So the second outcome is an interesting one.

56
00:04:09,520 --> 00:04:15,560
It's increased access of female and male African leaders of tomorrow scholars to opportunities

57
00:04:15,560 --> 00:04:21,920
for exchange and collaboration within the framework of Canadian Pan-African Public Administration

58
00:04:21,920 --> 00:04:23,840
public policy networks.

59
00:04:23,840 --> 00:04:27,140
Now what's interesting about this one is it's really not an outcome.

60
00:04:27,140 --> 00:04:32,220
They want to increase access for these scholars for opportunities to exchange.

61
00:04:32,220 --> 00:04:37,460
And that is basically what the project is doing by sending them from their African countries

62
00:04:37,460 --> 00:04:41,980
over to Canada to complete this master's program at public administration.

63
00:04:41,980 --> 00:04:46,280
So I think they have increased their access, but that's not an outcome.

64
00:04:46,280 --> 00:04:49,980
That's part of the services they're getting by being in the project.

65
00:04:49,980 --> 00:04:53,640
So we're not even going to look at the indicators for outcome 2.

66
00:04:53,640 --> 00:05:01,340
Now outcome 3, however, is an outcome because they expect these African students studying

67
00:05:01,340 --> 00:05:07,340
this master's program at public administration to have, quote, enhanced understanding of

68
00:05:07,340 --> 00:05:13,340
female and male African leaders of tomorrow scholars respecting public administration,

69
00:05:13,340 --> 00:05:19,320
public policy development, and implementation processes, including the integration of gender

70
00:05:19,320 --> 00:05:21,760
equality considerations.

71
00:05:21,760 --> 00:05:26,660
Now this one's interesting, as I mentioned in part one, because there's the self-reporting

72
00:05:26,660 --> 00:05:31,080
bias obviously, because they're asked, and the indicator is.

73
00:05:31,080 --> 00:05:36,680
Percentage of female and male African leaders of tomorrow scholars reporting that their

74
00:05:36,680 --> 00:05:43,680
mentoring relationships with Canadian leaders, practitioners, have strengthened their knowledge

75
00:05:43,680 --> 00:05:49,500
of public administration, public policy development, and implementation processes, including the

76
00:05:49,500 --> 00:05:53,760
integration of gender equality considerations.

77
00:05:53,760 --> 00:05:59,740
So again, we have the same problem, but they're trying to measure strength and knowledge of

78
00:05:59,740 --> 00:06:04,620
not just public administration, but specifically to gender equality.

79
00:06:04,620 --> 00:06:06,760
And they're doing this by self-reporting.

80
00:06:06,760 --> 00:06:11,840
And again, that's not a good measure of whether or not they have, quote, actually enhanced

81
00:06:11,840 --> 00:06:13,460
their understanding.

82
00:06:13,460 --> 00:06:17,540
They could have enhanced their understanding, but we have this problem with the self-reporting

83
00:06:17,540 --> 00:06:18,540
bias.

84
00:06:18,540 --> 00:06:24,300
And I had mentioned in part one that there are lots of gender equality indices out there

85
00:06:24,300 --> 00:06:30,260
where the United Nations Development Program, UNDP, is measuring these at the country level.

86
00:06:30,260 --> 00:06:37,120
What would be of interest to me is if they measured their ability to calculate this index

87
00:06:37,120 --> 00:06:42,780
for the country where they come from in Africa to see how they're doing on this gender equality

88
00:06:42,780 --> 00:06:43,780
measure.

89
00:06:43,780 --> 00:06:48,100
And that would have been a better indicator for me in terms of measuring their enhanced

90
00:06:48,100 --> 00:06:52,860
understanding of, quote, gender equality.

91
00:06:52,860 --> 00:06:56,020
Can I add one small point here?

92
00:06:56,020 --> 00:06:58,400
I'd like to support what you were saying.

93
00:06:58,400 --> 00:07:04,060
What is interesting about using available gender equality indices is that, especially

94
00:07:04,060 --> 00:07:09,660
here where I was speaking about Africa and Canada, so different countries, it provides

95
00:07:09,660 --> 00:07:13,540
them a room also to benchmark, to compare with other countries.

96
00:07:13,540 --> 00:07:20,040
So it's not only looking at your own schoolers, but these indices are global.

97
00:07:20,040 --> 00:07:22,580
They are used in multi-other organizations.

98
00:07:22,580 --> 00:07:27,100
So it would have given them the opportunity to compare even with their learning and their

99
00:07:27,100 --> 00:07:30,940
outcomes with other organizations and other countries as well.

100
00:07:30,940 --> 00:07:31,940
Right.

101
00:07:31,940 --> 00:07:32,940
Yeah.

102
00:07:32,940 --> 00:07:33,940
No, that's good.

103
00:07:33,940 --> 00:07:38,740
And that, given that this project, I think, is attracting African students from, I think

104
00:07:38,740 --> 00:07:42,980
it was 23 African countries, as I described in part one.

105
00:07:42,980 --> 00:07:48,860
Yeah, that would be interesting, especially if those students as part of their MPA program,

106
00:07:48,860 --> 00:07:53,300
because we don't know in their courses in Canada if they had to take, if they had to

107
00:07:53,300 --> 00:07:57,020
learn how to calculate this indices, I had to learn it when I was doing international

108
00:07:57,020 --> 00:07:58,100
development.

109
00:07:58,100 --> 00:08:03,820
But it would be interesting as an exercise for the students who are in this project to

110
00:08:03,820 --> 00:08:05,300
do this comparison.

111
00:08:05,300 --> 00:08:09,700
And I used to do it when I was teaching international development at the University of Toronto,

112
00:08:09,700 --> 00:08:13,780
where each student would pick a developing country and we would go through a variety

113
00:08:13,780 --> 00:08:19,300
of indices such as the gender equality measure and we would graph it to see the comparisons

114
00:08:19,300 --> 00:08:22,340
between developing countries.

115
00:08:22,340 --> 00:08:28,140
Outcome number four is increased knowledge of female and male African leaders of tomorrow,

116
00:08:28,140 --> 00:08:33,300
scholars in governance, public administration, public policy and gender equality.

117
00:08:33,300 --> 00:08:37,220
And I've already sort of alluded to this in the previous outcome.

118
00:08:37,220 --> 00:08:42,300
The indicator here I've chosen is the percentage of female and male African leaders of tomorrow

119
00:08:42,300 --> 00:08:47,780
scholars who have successfully completed a master's level public administration, public

120
00:08:47,780 --> 00:08:49,940
policy program in Canada.

121
00:08:49,940 --> 00:08:54,460
Now you would think logically that completing this master's program in public administration

122
00:08:54,460 --> 00:09:01,020
would be adequate to claim that they actually indeed increase their knowledge of public

123
00:09:01,020 --> 00:09:02,220
administration.

124
00:09:02,220 --> 00:09:04,380
Nothing wrong with that.

125
00:09:04,380 --> 00:09:10,620
But one of the issues here is comparison groups.

126
00:09:10,620 --> 00:09:16,100
There could be other master's programs in public administration where African students

127
00:09:16,100 --> 00:09:19,740
have attended it in Africa because there's a lot of African universities that offer the

128
00:09:19,740 --> 00:09:22,420
same MPA program.

129
00:09:22,420 --> 00:09:27,940
And it would be interesting to compare levels of knowledge of public administration or public

130
00:09:27,940 --> 00:09:33,760
policy with a comparison group and even specific to their ability to calculate these gender

131
00:09:33,760 --> 00:09:35,540
equality indices.

132
00:09:35,540 --> 00:09:44,700
So the first one indicator, even though graduating is a good indicator of increased knowledge,

133
00:09:44,700 --> 00:09:49,760
I would think you would have to go a step further and say there may be MPA programs

134
00:09:49,760 --> 00:09:56,580
out there where students are also increasing their knowledge and at even maybe a greater

135
00:09:56,580 --> 00:10:00,420
rate than the students that are in this project.

136
00:10:00,420 --> 00:10:07,120
So we need to ideally have a comparison group where we can show that this project actually

137
00:10:07,120 --> 00:10:10,580
had a better level of knowledge acquired by the students.

138
00:10:10,580 --> 00:10:17,660
The next outcome, number five, is increased effectiveness of female and male African leaders

139
00:10:17,660 --> 00:10:23,060
of tomorrow scholars to provide leadership within public administration, public policy

140
00:10:23,060 --> 00:10:29,300
networks, Canadian and Pan-African networks.

141
00:10:29,300 --> 00:10:33,620
And the indicator that I'll be looking at is percentage of female and male schoolers

142
00:10:33,620 --> 00:10:39,340
who report making a meaningful contribution to public administration, public policy development

143
00:10:39,340 --> 00:10:42,560
in key sectors in sub-Saharan Africa.

144
00:10:42,560 --> 00:10:49,220
So the first thing before looking at the indicator that I think I would like to comment on as

145
00:10:49,220 --> 00:10:54,460
the outcome itself, because measuring effectiveness and planning to increase effectiveness is

146
00:10:54,460 --> 00:11:01,980
in itself is a very big outcome, is broad, usually, and for evaluators to measure effectiveness

147
00:11:01,980 --> 00:11:07,740
of a training or of a program, we're speaking about multi-month evaluation and even might

148
00:11:07,740 --> 00:11:09,700
reach one-year evaluation.

149
00:11:09,700 --> 00:11:18,340
So it's measure that for an outcome indicator in my perspective might not be enough.

150
00:11:18,340 --> 00:11:22,940
And again, we have the self-reporting bias here again.

151
00:11:22,940 --> 00:11:27,820
So it shows as David mentioned, this is another indicator where we have reporting bias.

152
00:11:27,820 --> 00:11:32,860
Oh, sorry to interrupt, but is it...we've sort of skipped here.

153
00:11:32,860 --> 00:11:37,740
This is outcome six, increased effectiveness of female and male African leaders of tomorrow

154
00:11:37,740 --> 00:11:42,940
scholars to contribute to public administration, public policy development in key sectors in

155
00:11:42,940 --> 00:11:45,060
sub-Saharan Africa.

156
00:11:45,060 --> 00:11:51,100
And the indicator is percentage of female and male scholars making a meaningful contribution

157
00:11:51,100 --> 00:11:53,380
from the performance measurement framework.

158
00:11:53,380 --> 00:11:56,700
Do they have to report this themselves?

159
00:11:56,700 --> 00:11:59,540
Is that what you're saying?

160
00:11:59,540 --> 00:12:04,300
The wording of the indicator and the document they have is percentage of female and male

161
00:12:04,300 --> 00:12:07,300
scholars who report making a meaningful contribution.

162
00:12:07,300 --> 00:12:08,300
Excellent.

163
00:12:08,300 --> 00:12:09,300
Okay.

164
00:12:09,300 --> 00:12:10,300
So thank you for that.

165
00:12:10,300 --> 00:12:11,460
So yeah, who report?

166
00:12:11,460 --> 00:12:13,500
So there's a self-reporting bias.

167
00:12:13,500 --> 00:12:16,780
Okay, great.

168
00:12:16,780 --> 00:12:19,700
And increased effectiveness is not going to measure that.

169
00:12:19,700 --> 00:12:25,900
Yeah, and another important thing is the use of the word meaningful contribution.

170
00:12:25,900 --> 00:12:30,380
So how could we know that they are doing meaningful contribution?

171
00:12:30,380 --> 00:12:32,700
What does meaningful contribution mean to them?

172
00:12:32,700 --> 00:12:37,000
Does it mean like...I think I haven't found any definition of that.

173
00:12:37,000 --> 00:12:40,460
Maybe it's in their documentation, but at least in the BMF it's not there.

174
00:12:40,460 --> 00:12:47,180
I would expect like that there will be a clearly that such meaningful contribution should be

175
00:12:47,180 --> 00:12:52,900
clearly defined and then they can measure it whether through what applies to the African

176
00:12:52,900 --> 00:12:53,900
context.

177
00:12:53,900 --> 00:12:59,700
Like they can say, okay, the meaningful contribution could be to what extent they are doing quality

178
00:12:59,700 --> 00:13:05,780
written products or briefing notes or white papers in their work, or maybe to what extent

179
00:13:05,780 --> 00:13:09,760
they are pushing the agenda, the policy agenda in a specific sector.

180
00:13:09,760 --> 00:13:14,300
So I know it varies based on the context, but at least there should be boundaries or

181
00:13:14,300 --> 00:13:16,260
a clear definition of it.

182
00:13:16,260 --> 00:13:21,660
So that they can, even if we're speaking about self-reporting, so that student can't report

183
00:13:21,660 --> 00:13:22,660
in it.

184
00:13:22,660 --> 00:13:30,140
But again, preferably it can be looking at actual tangible deliverables that these students

185
00:13:30,140 --> 00:13:31,140
are doing.

186
00:13:31,140 --> 00:13:32,140
Right.

187
00:13:32,140 --> 00:13:33,140
Yeah.

188
00:13:33,140 --> 00:13:37,180
They should have a definition of what they mean by meaningful contribution, maybe a certain

189
00:13:37,180 --> 00:13:44,100
number of briefing notes or white papers or whatever analysis that they're doing when

190
00:13:44,100 --> 00:13:46,460
they graduate from the MPA program.

191
00:13:46,460 --> 00:13:51,300
That would be a good measure of meaningful contribution, at least try to attempt to measure

192
00:13:51,300 --> 00:13:52,820
it a bit more objectively.

193
00:13:52,820 --> 00:13:53,820
Right.

194
00:13:53,820 --> 00:14:00,740
Now, outcome number five, just to go back, increased effectiveness of female and male

195
00:14:00,740 --> 00:14:06,220
African leaders of tomorrow scholars to provide leadership within public administration, public

196
00:14:06,220 --> 00:14:10,860
policy networks, Canadian, pan-African networks.

197
00:14:10,860 --> 00:14:12,460
And we have two indicators there.

198
00:14:12,460 --> 00:14:17,120
I think you had mentioned you would want to talk to the number of inputs or the percentage

199
00:14:17,120 --> 00:14:24,340
of male and female scholars actively involved in leadership functions within public administration,

200
00:14:24,340 --> 00:14:25,340
public policy networks.

201
00:14:25,340 --> 00:14:29,780
My understanding for that one is self-reporting also, right?

202
00:14:29,780 --> 00:14:30,780
Yes.

203
00:14:30,780 --> 00:14:37,180
And sorry here, it seems that there are some confusion in terms of the order, but I can

204
00:14:37,180 --> 00:14:42,820
speak about the number of inputs such as paper studies, presentation lectures provided by

205
00:14:42,820 --> 00:14:47,540
female and male schoolers to public administration networks.

206
00:14:47,540 --> 00:14:56,380
So here it is, we are counting number of papers and to measure such a high level outcome,

207
00:14:56,380 --> 00:15:00,040
would the number of papers in itself be enough?

208
00:15:00,040 --> 00:15:06,820
So I could develop as a school or even a public policy official five blogs while another person

209
00:15:06,820 --> 00:15:15,740
might come up with a more meaningful policy paper that is being provided to the government.

210
00:15:15,740 --> 00:15:23,120
So number in itself, like I prefer that if they have it to look at more a different way

211
00:15:23,120 --> 00:15:31,700
of put this indicator and also the comparison group, like how can they, is that, are there

212
00:15:31,700 --> 00:15:36,300
any other group of people or graduates?

213
00:15:36,300 --> 00:15:39,740
How is that about like the papers they are developing?

214
00:15:39,740 --> 00:15:42,540
Are they of different quality?

215
00:15:42,540 --> 00:15:49,100
Are the graduates of this program providing more quality papers, more quality or more

216
00:15:49,100 --> 00:15:54,780
like papers that makes real change to their entities they are working on?

217
00:15:54,780 --> 00:15:58,700
So yeah, I don't know if you have anything else to add to this also, David.

218
00:15:58,700 --> 00:16:04,720
The only thing I would add is they talk about number of inputs and we talked about it earlier.

219
00:16:04,720 --> 00:16:09,060
This is kind of in the right direction where they are suggesting if these students when

220
00:16:09,060 --> 00:16:16,040
they graduate can produce high quality knowledge products, what you could do is maybe set a

221
00:16:16,040 --> 00:16:21,820
standard of, for example, in Canada there is the institution of public administration

222
00:16:21,820 --> 00:16:25,540
of Canada and I think they still have an academic journal.

223
00:16:25,540 --> 00:16:28,780
There is also a magazine called Policy Options.

224
00:16:28,780 --> 00:16:36,420
There could be equivalent sort of magazines or academic journals in the African country

225
00:16:36,420 --> 00:16:43,820
where the students come from where they could track their contribution to these particular

226
00:16:43,820 --> 00:16:48,580
journals as a measure of increased effectiveness.

227
00:16:48,580 --> 00:16:54,160
But to be fair to the Canadian Bureau for International Education, it's a bit difficult,

228
00:16:54,160 --> 00:16:58,340
but they should try at least to have an attempt to make a better indicator rather than just

229
00:16:58,340 --> 00:16:59,900
saying number of inputs.

230
00:16:59,900 --> 00:17:04,860
They could keep it at number of inputs as long as they have specified criteria or what's

231
00:17:04,860 --> 00:17:11,220
considered to be a high quality input and they could apply that across all 23 African

232
00:17:11,220 --> 00:17:13,460
countries where these students come from.

233
00:17:13,460 --> 00:17:17,900
And then they could argue that when they started the program they didn't produce any of these

234
00:17:17,900 --> 00:17:23,740
inputs and then when they finished the program and further on they produced these inputs

235
00:17:23,740 --> 00:17:26,500
and that could be a measure of increased effectiveness.

236
00:17:26,500 --> 00:17:35,420
Yeah, and the last outcome is outcome number seven and it is strengthened economic and

237
00:17:35,420 --> 00:17:43,260
social development, key public administration and policy sectors in sub-Saharan Africa.

238
00:17:43,260 --> 00:17:46,500
And the measure here they have only one indicator.

239
00:17:46,500 --> 00:17:52,060
It's the percentage of female and male African leaders of tomorrow scholars who hold senior

240
00:17:52,060 --> 00:17:57,900
level decision making management positions in sub-Saharan Africa.

241
00:17:57,900 --> 00:18:03,340
This is fine as an indicator, but the challenge here again is graduates are asked if you look

242
00:18:03,340 --> 00:18:07,580
at the performance measurement framework, this is why you need to get the performance

243
00:18:07,580 --> 00:18:12,540
measurement framework to actually see the details of how do they come up with this indicator.

244
00:18:12,540 --> 00:18:18,220
They ask the graduates themselves to report where they are, the entry, middle or senior

245
00:18:18,220 --> 00:18:20,660
level in the government.

246
00:18:20,660 --> 00:18:27,700
And they shouldn't be asked to report especially when the criteria for what's entry level,

247
00:18:27,700 --> 00:18:30,140
middle or senior level is not defined.

248
00:18:30,140 --> 00:18:33,820
And they should try to do that across all 23 African countries.

249
00:18:33,820 --> 00:18:36,400
And it's possible to be able to do that.

250
00:18:36,400 --> 00:18:42,980
And then what you do is interview the graduates and you ask them through a checklist of a

251
00:18:42,980 --> 00:18:47,140
series of questions and then from there you could put them in the box whether they would

252
00:18:47,140 --> 00:18:51,020
be considered to be entry, middle or senior level.

253
00:18:51,020 --> 00:18:55,460
And the interesting thing I would add to this is that's the first thing they have to do

254
00:18:55,460 --> 00:19:01,740
to improve the indicator is get an objective measure and then follow up with the graduates

255
00:19:01,740 --> 00:19:07,220
when they're back in their African countries in the civil service and ask them, you know,

256
00:19:07,220 --> 00:19:08,900
a series of questions.

257
00:19:08,900 --> 00:19:14,240
I'm repeating myself to get into these three boxes, entry, middle or senior level.

258
00:19:14,240 --> 00:19:19,300
But the other thing they should be doing is comparing these graduates of MPA programs

259
00:19:19,300 --> 00:19:24,260
with in the same country for each of the 23 African countries where these graduates went

260
00:19:24,260 --> 00:19:31,820
back, they should also be asking graduates from MPA programs outside of this project

261
00:19:31,820 --> 00:19:33,980
and see how well they're doing.

262
00:19:33,980 --> 00:19:35,300
Same level of education.

263
00:19:35,300 --> 00:19:40,900
They just find your MPA students who graduated working in the same government in the same

264
00:19:40,900 --> 00:19:47,300
African country and ask them also the same questions and to see what percentage of those

265
00:19:47,300 --> 00:19:54,220
graduates are occupying senior, middle or entry level positions because again, without

266
00:19:54,220 --> 00:20:01,740
a comparison group, this project may be not doing as well as graduates from MPA programs

267
00:20:01,740 --> 00:20:06,300
outside of this project, which is fine because what you're doing here is providing these

268
00:20:06,300 --> 00:20:12,100
Africans with an opportunity they otherwise may not be able to get because the tuition

269
00:20:12,100 --> 00:20:16,260
at MPA programs is very expensive for foreign students.

270
00:20:16,260 --> 00:20:21,580
So the issue is if they're going to make the claim that they're strengthened development

271
00:20:21,580 --> 00:20:26,840
by increasing the percentage who hold senior positions from this project, they need to

272
00:20:26,840 --> 00:20:28,500
use a comparison group.

273
00:20:28,500 --> 00:20:33,380
And I know from personal experience working in Rwanda and Nigeria at the National Institute

274
00:20:33,380 --> 00:20:42,180
of Statistics, I worked with very bright, talented senior managers who were graduates of master's

275
00:20:42,180 --> 00:20:49,980
programs in statistics from local universities as well as international universities also.

276
00:20:49,980 --> 00:20:56,560
So there's a huge variety of civil servants in the National Institute of Statistics in

277
00:20:56,560 --> 00:21:03,340
Rwanda as well as in Nigeria who had education all around the world and including in Nigeria.

278
00:21:03,340 --> 00:21:11,300
So, I think they need to show that this project is even better at strengthening economic and

279
00:21:11,300 --> 00:21:14,020
social development of their graduates.

280
00:21:14,020 --> 00:21:19,180
But another option is to just say, look, we're in the business of charity and we're just

281
00:21:19,180 --> 00:21:26,100
going to make sure that these students get their tuition paid for by MasterCard Foundation,

282
00:21:26,100 --> 00:21:29,180
which is what MasterCard Foundation is doing here.

283
00:21:29,180 --> 00:21:34,220
So you could argue that maybe this project isn't necessary at all, the $5 million, they

284
00:21:34,220 --> 00:21:41,700
just need the $5 million or whatever it is for the tuition to pay for the 120 to 130

285
00:21:41,700 --> 00:21:42,700
students.

286
00:21:42,700 --> 00:21:45,420
So that'll be an ongoing debate.

287
00:21:45,420 --> 00:21:46,420
But that's the idea.

288
00:21:46,420 --> 00:21:47,420
Yeah.

289
00:21:47,420 --> 00:21:52,060
I have one, like I'm curious to know from this outcome, like on this indicators, how

290
00:21:52,060 --> 00:21:57,300
they are seeing the link between this indicator and the outcome because measuring the strength

291
00:21:57,300 --> 00:22:02,740
in economic and social development, how we envision this to happen by having these seniors

292
00:22:02,740 --> 00:22:04,420
in these positions.

293
00:22:04,420 --> 00:22:11,380
So I think there might be something in between needed and especially which could be another

294
00:22:11,380 --> 00:22:17,020
also question that I'm curious to, like I cannot find an answer to as they are addressing

295
00:22:17,020 --> 00:22:22,500
123 to I think 130 students from 23 countries.

296
00:22:22,500 --> 00:22:28,700
So if we do the formula where maybe in some countries they will only have five graduates,

297
00:22:28,700 --> 00:22:33,620
so do we expect these five will have that contribution to strengthen the economic and

298
00:22:33,620 --> 00:22:34,620
social developments?

299
00:22:34,620 --> 00:22:37,500
Will they have the ability to do so?

300
00:22:37,500 --> 00:22:41,700
So some like more clarification, further information might be needed and other indicators might

301
00:22:41,700 --> 00:22:44,620
need to support that as well.

302
00:22:44,620 --> 00:22:45,620
Very good point.

303
00:22:45,620 --> 00:22:52,220
It ties in nicely with a comment I made in part one, which is that Canadian immigration

304
00:22:52,220 --> 00:23:00,380
policy right now is to encourage a lot of young, bright, talented students from developing

305
00:23:00,380 --> 00:23:07,460
countries to come to Canada and complete their graduate or undergraduate education, graduate

306
00:23:07,460 --> 00:23:10,460
and stay in Canada.

307
00:23:10,460 --> 00:23:16,000
And that conflicts directly with the objectives of this program, which is to strengthen the

308
00:23:16,000 --> 00:23:22,140
development of key public administration and policy sectors in sub-Saharan Africa.

309
00:23:22,140 --> 00:23:27,820
So this is another interesting point, which is that a lot of graduates in international

310
00:23:27,820 --> 00:23:33,380
development in Canada are perfect for staying in the country because they have the skill

311
00:23:33,380 --> 00:23:39,220
set, the high levels of education, they're young, and that's part of the Canadian immigration

312
00:23:39,220 --> 00:23:40,900
policy right now.

313
00:23:40,900 --> 00:23:46,300
And it would conflict exactly with what you've said, where maybe they only have five in a

314
00:23:46,300 --> 00:23:51,220
particular country and even worse, some of the students graduate in Canada with their

315
00:23:51,220 --> 00:23:57,580
MPAs and they don't go back because there's an aging population in Canada where there's

316
00:23:57,580 --> 00:24:04,820
a huge demographic boom around the 60 to 65 age group that are retiring from the public

317
00:24:04,820 --> 00:24:10,920
sector and the age group behind them is smaller because of the birth rate.

318
00:24:10,920 --> 00:24:14,080
So that's another interesting point to make.

319
00:24:14,080 --> 00:24:19,500
And it raises the whole issue again of is this really a development project?

320
00:24:19,500 --> 00:24:23,940
And it could be just a charity project and there's nothing wrong with having a charity

321
00:24:23,940 --> 00:24:32,500
project by giving these students an opportunity to avoid having to pay $18,000 Canadian in

322
00:24:32,500 --> 00:24:36,540
tuition, foreign international student tuition for an MPA program.

323
00:24:36,540 --> 00:24:40,780
I think if you go on the websites here in Canada, I think that's what the foreign tuition

324
00:24:40,780 --> 00:24:47,580
at your university, Carleton, I think it's about $9,000 a semester for foreign students.

325
00:24:47,580 --> 00:24:48,980
So it's very expensive.

326
00:24:48,980 --> 00:24:54,600
So you could argue, you know, that's fine, but it conflicts with the outcomes that they're

327
00:24:54,600 --> 00:24:56,420
claiming in this project.

328
00:24:56,420 --> 00:24:58,380
Yeah, yeah.

329
00:24:58,380 --> 00:25:03,580
Now that we've done all seven outcomes, we took out outcome number two, because it was

330
00:25:03,580 --> 00:25:07,180
access and it's actually an output.

331
00:25:07,180 --> 00:25:11,700
Are there any other outcome indicators that you wanted to talk about or are we done?

332
00:25:11,700 --> 00:25:12,700
Okay.

333
00:25:12,700 --> 00:25:18,940
Yes, there's one under outcome number one.

334
00:25:18,940 --> 00:25:20,380
Case study experience.

335
00:25:20,380 --> 00:25:21,380
That's right.

336
00:25:21,380 --> 00:25:22,380
Yeah.

337
00:25:22,380 --> 00:25:23,380
Yeah.

338
00:25:23,380 --> 00:25:27,060
Percentage of female and male African leaders at Tomorrow Scholars reporting that the case

339
00:25:27,060 --> 00:25:33,940
study experience has increased their capacity to address policy changes within their home

340
00:25:33,940 --> 00:25:35,120
environment.

341
00:25:35,120 --> 00:25:39,900
And that indicator was for measuring increased leadership skills.

342
00:25:39,900 --> 00:25:41,600
Yes.

343
00:25:41,600 --> 00:25:47,220
So what I want to mention first is that what I liked, like this initiative looks at the

344
00:25:47,220 --> 00:25:50,300
different learning development stages.

345
00:25:50,300 --> 00:25:58,060
So it looks into the direct attendance of events, looking into mentoring, applying the

346
00:25:58,060 --> 00:26:02,580
learning through case studies and then providing mentoring along the way.

347
00:26:02,580 --> 00:26:07,420
These indicators in particular, like there is this case study.

348
00:26:07,420 --> 00:26:13,700
I did not get from the BFM how exactly they're doing that, even with the outputs that it

349
00:26:13,700 --> 00:26:18,420
seems that they will be applying, doing, exploring or doing a case study, applying the case study

350
00:26:18,420 --> 00:26:21,180
experience on the country.

351
00:26:21,180 --> 00:26:27,860
And there are some audience who will be, I think will be looking into their presentation,

352
00:26:27,860 --> 00:26:31,500
but it's not 100% clear.

353
00:26:31,500 --> 00:26:37,040
But if they are applying that into a case study and to their countries, it will be good

354
00:26:37,040 --> 00:26:43,940
to know if are the people like, are there external people who are evaluating them?

355
00:26:43,940 --> 00:26:49,260
Because this indicator again is a self-reporting and to self-report, like to report on their

356
00:26:49,260 --> 00:26:54,500
own experience with a case study will give us some information from them on the learning

357
00:26:54,500 --> 00:26:55,500
along the way.

358
00:26:55,500 --> 00:27:01,300
But was that really adding value to their leadership skills and how you need someone

359
00:27:01,300 --> 00:27:09,020
else to provide you this assessment and someone with experience and with external backgrounds?

360
00:27:09,020 --> 00:27:13,260
And yeah, so this is generally on this indicator.

361
00:27:13,260 --> 00:27:15,400
So, yes.

362
00:27:15,400 --> 00:27:18,260
And it's also saying within their home environment.

363
00:27:18,260 --> 00:27:20,340
So I'm wondering how they do that.

364
00:27:20,340 --> 00:27:23,020
Yeah, they would have to be back in Africa, right?

365
00:27:23,020 --> 00:27:24,020
Exactly.

366
00:27:24,020 --> 00:27:26,980
This is why I said it's not 100% clear.

367
00:27:26,980 --> 00:27:31,860
So I think there are some like, there needs to be some more clarity in the PFM in general.

368
00:27:31,860 --> 00:27:34,660
And this is a general observation.

369
00:27:34,660 --> 00:27:35,660
No that's true.

370
00:27:35,660 --> 00:27:36,660
Yeah.

371
00:27:36,660 --> 00:27:43,220
So I think we've covered the performance measurement framework and thank you for your input, Hyatt.

372
00:27:43,220 --> 00:27:50,180
And what we're going to do now is we will send this podcast part two, episode two, to

373
00:27:50,180 --> 00:27:52,380
the Canadian Bureau for International Education.

374
00:27:52,380 --> 00:27:58,980
But more importantly, as noted in the trailer for this podcast, we need to advocate for

375
00:27:58,980 --> 00:27:59,980
two things.

376
00:27:59,980 --> 00:28:06,120
We'll be sending this podcast recording to the Minister for International Development,

377
00:28:06,120 --> 00:28:11,340
the Honorable Ahmed Hossein, as well as the shadow critics from the Conservative Party

378
00:28:11,340 --> 00:28:14,840
of the New Democratic Party on international development.

379
00:28:14,840 --> 00:28:16,420
Because this is a universal problem.

380
00:28:16,420 --> 00:28:19,580
It doesn't matter what political party you support.

381
00:28:19,580 --> 00:28:25,700
We need to have proper evaluation of development projects funded by the Canadian government,

382
00:28:25,700 --> 00:28:31,080
regardless of who the Canadian government is, Conservative, Liberal, NDP, Green, whatever.

383
00:28:31,080 --> 00:28:38,140
So we'll be recommending one, that performance measurement frameworks be required to be put

384
00:28:38,140 --> 00:28:43,740
on the websites of the organizations such as the Canadian Bureau of International Education,

385
00:28:43,740 --> 00:28:47,420
that they put their performance measurement framework for all their projects funded by

386
00:28:47,420 --> 00:28:53,460
the government of Canada on their websites, both the blank ones and the data that's going

387
00:28:53,460 --> 00:28:55,940
to be coming in on these indicators.

388
00:28:55,940 --> 00:29:01,340
And number two, we want to recommend, since the podcast has been raising this, that the

389
00:29:01,340 --> 00:29:07,180
government of Canada follow their own guidelines on how to properly design outcome indicators

390
00:29:07,180 --> 00:29:13,380
to properly measure the achievement of the outcomes for the projects that they fund.

391
00:29:13,380 --> 00:29:17,980
Because we're talking about five billion, between four and five billion dollars a year,

392
00:29:17,980 --> 00:29:23,660
the government of Canada funds international development projects all around the world.

393
00:29:23,660 --> 00:29:26,100
So we'll be advocating for that.

394
00:29:26,100 --> 00:29:31,140
Hopefully someday, everyone will be able to see a performance measurement framework on

395
00:29:31,140 --> 00:29:34,860
every organization's website.

396
00:29:34,860 --> 00:29:43,340
So Hyatt, thank you so much for taking your time to provide your expertise and input and

397
00:29:43,340 --> 00:29:49,860
perhaps you'll return again for another episode because I've got another 12, well I've got

398
00:29:49,860 --> 00:29:57,180
a total of 12 lined up for season one and then I have another 12 lined up for season

399
00:29:57,180 --> 00:29:58,180
two.

400
00:29:58,180 --> 00:30:01,300
So we'll be doing this for a long, long time.

401
00:30:01,300 --> 00:30:07,700
Yeah, no, sure, and thank you for your efforts and I think you are doing something very important.

402
00:30:07,700 --> 00:30:12,780
And as mentioned, there are a lot of good things in this PMF, but also again, there

403
00:30:12,780 --> 00:30:14,820
is room for improvement.

404
00:30:14,820 --> 00:30:20,620
There are ways to present even the way that this PMF is presented if it's supported by

405
00:30:20,620 --> 00:30:26,380
a more easy to read theory of change or visual, will even maybe answer some of the questions

406
00:30:26,380 --> 00:30:28,140
that the reader might have.

407
00:30:28,140 --> 00:30:32,980
But yeah, I really appreciate your time and involving me in this.

408
00:30:32,980 --> 00:30:34,420
So thank you so much.

409
00:30:34,420 --> 00:30:41,140
Well, thank you and we'll keep in touch and feel free to also, I want to add that if anybody

410
00:30:41,140 --> 00:30:45,060
listening out there and I keep repeating this, I'll just say it one more time.

411
00:30:45,060 --> 00:30:48,300
If you're interested in getting any of the performance measurement frameworks that we've

412
00:30:48,300 --> 00:30:53,460
discussed on any of the episodes along with the summary of the outcome indicators and

413
00:30:53,460 --> 00:30:58,260
how they fall into one of these five problem areas of not properly measuring the outcomes

414
00:30:58,260 --> 00:31:04,580
for the project, you can send me an email at evaluatecanadaaid at gmail.com and I'd

415
00:31:04,580 --> 00:31:06,700
be happy to send them to you.

416
00:31:06,700 --> 00:31:11,860
And I'm always looking for bright, talented evaluation experts like Hyatt who would be

417
00:31:11,860 --> 00:31:14,980
interested in being on future episodes of the podcast.

418
00:31:14,980 --> 00:31:20,260
I can also send you the future performance measurement frameworks that I have that I'll

419
00:31:20,260 --> 00:31:22,420
be featuring in future episodes.

420
00:31:22,420 --> 00:31:30,060
So thank you again and stay tuned for episode four.

421
00:31:30,060 --> 00:31:57,300
We've already done episode three, but episode four will be coming up also.