1
00:00:00,000 --> 00:00:05,000
Welcome to the Improving Development Evaluation podcast.

2
00:00:07,280 --> 00:00:09,960
I'm your host David Wand and welcome to season two,

3
00:00:09,960 --> 00:00:13,040
episode two, where we're going to feature

4
00:00:13,040 --> 00:00:15,480
the international development organization

5
00:00:15,480 --> 00:00:18,880
called Doctors of the World.

6
00:00:18,880 --> 00:00:20,440
And if I recall correctly,

7
00:00:20,440 --> 00:00:22,960
they are located in Montreal, Canada.

8
00:00:22,960 --> 00:00:33,560
They have received $14,886,968 according to the Government of Canada Global Affairs Canada

9
00:00:33,560 --> 00:00:36,880
project browser website.

10
00:00:36,880 --> 00:00:43,400
For a project they are delivering in the West African country of Burkina Faso.

11
00:00:43,400 --> 00:00:50,200
And the name of that project is called Access to Health Rights in Northern Burkina Faso.

12
00:00:50,200 --> 00:00:54,640
And you can learn more about Doctors of the World, the international development organization

13
00:00:54,640 --> 00:01:10,920
at their website www.medecinsdumonde.ca.

14
00:01:10,920 --> 00:01:16,400
And in this project of $14 million, courtesy of the Canadian taxpayer,

15
00:01:16,400 --> 00:01:23,160
there were eight expected outcomes to be achieved from delivering services in Burkina Faso

16
00:01:23,160 --> 00:01:25,560
for that cost of $14 million.

17
00:01:25,560 --> 00:01:32,480
And for those eight outcomes, they had 15 outcome indicators that they used

18
00:01:32,480 --> 00:01:37,760
from the performance measurement framework to see whether or not those eight expected

19
00:01:37,760 --> 00:01:40,320
outcomes had been achieved.

20
00:01:40,320 --> 00:01:49,280
My review of those 15 outcome indicators indicated that 13 out of the 15 were not valid measures

21
00:01:49,280 --> 00:01:56,520
of the expected outcomes and therefore Doctors of the World cannot make the claim that even

22
00:01:56,520 --> 00:02:03,120
if those outcomes were achieved, they cannot claim that the project costing $14 million

23
00:02:03,120 --> 00:02:10,280
was responsible for the achievement of those expected outcomes.

24
00:02:10,280 --> 00:02:14,280
Before I go into detail about those outcome indicators and the outcomes, I'm going to

25
00:02:14,280 --> 00:02:18,920
give a brief description of what the project from the performance measurement framework

26
00:02:18,920 --> 00:02:21,160
actually delivered.

27
00:02:21,160 --> 00:02:25,920
And at any time, if you're interested in getting the performance measurement framework plus

28
00:02:25,920 --> 00:02:34,960
the critique in an Excel sheet that I have, you can email me at evaluatecanadaaid.gmail.com

29
00:02:34,960 --> 00:02:38,200
and I would be happy to email those documents to you.

30
00:02:38,200 --> 00:02:43,240
So let's begin with a brief description of the project.

31
00:02:43,240 --> 00:02:49,960
The project in Burkina Faso delivered by Doctors of the World has roughly 16 target groups

32
00:02:49,960 --> 00:02:55,480
and each one of those target groups receives a certain amount of services.

33
00:02:55,480 --> 00:03:01,740
And with those services, they expect some of those target groups to achieve certain

34
00:03:01,740 --> 00:03:03,900
expected outcomes.

35
00:03:03,900 --> 00:03:10,240
So starting with target group number one, International Dialogue Forum participants.

36
00:03:10,240 --> 00:03:14,640
They are supposed to receive from Doctors of the World presentations at these international

37
00:03:14,640 --> 00:03:18,680
forums to advocate for access to health.

38
00:03:18,680 --> 00:03:24,840
Second target group, authorities and stakeholders in Burkina Faso at local and national level

39
00:03:24,840 --> 00:03:30,680
and they are to receive from Doctors of the World presentations to this target group to

40
00:03:30,680 --> 00:03:34,160
advocate for access to health.

41
00:03:34,160 --> 00:03:38,360
Target group number three are community and religious leaders and they are to receive

42
00:03:38,360 --> 00:03:43,360
support in quote, facilitating awareness sessions.

43
00:03:43,360 --> 00:03:46,560
Target group number four, peer educators.

44
00:03:46,560 --> 00:03:53,040
They're supposed to receive trained sessions to promote to men and boys their responsibility

45
00:03:53,040 --> 00:03:59,140
on their right to health, sexual reproductive health and gender based violence.

46
00:03:59,140 --> 00:04:01,880
Target group number five, men and boys.

47
00:04:01,880 --> 00:04:06,920
They are receiving training on the right to health, sexual reproductive health and fighting

48
00:04:06,920 --> 00:04:11,960
gender based violence in quote, targeted areas.

49
00:04:11,960 --> 00:04:15,600
Target group number six, civil society organizations.

50
00:04:15,600 --> 00:04:20,160
They are trained on how to promote health rights, sexual reproductive health and fighting

51
00:04:20,160 --> 00:04:23,040
gender based violence.

52
00:04:23,040 --> 00:04:27,960
Target group number seven are women and their allies and they're trained and mobilized for

53
00:04:27,960 --> 00:04:33,400
the right to health, sexual reproductive health and the fight against gender based violence.

54
00:04:33,400 --> 00:04:39,480
Target group number eight are women and girls and they are sensitized and supported on health

55
00:04:39,480 --> 00:04:44,400
and gender based violence in quote, targeted areas.

56
00:04:44,400 --> 00:04:49,440
Target group number nine are families with children under five and in targeted areas

57
00:04:49,440 --> 00:04:57,940
they are trained on the mid upper arm circumference measure to enable them to detect nutritional

58
00:04:57,940 --> 00:05:01,920
edema and make referrals.

59
00:05:01,920 --> 00:05:07,360
Target group 10 are communities, women, men and adolescents in these communities.

60
00:05:07,360 --> 00:05:12,920
They are to be sensitized on primary health care services, sexual reproductive health

61
00:05:12,920 --> 00:05:15,440
and gender based violence.

62
00:05:15,440 --> 00:05:21,680
Target group 11 are community based health agents and community relays and they are to

63
00:05:21,680 --> 00:05:27,680
be trained on prevention, care and referral for primary health care, sexual reproductive

64
00:05:27,680 --> 00:05:31,520
health and gender based violence.

65
00:05:31,520 --> 00:05:35,680
Target group number 12, it's not clear who exactly they are but these are people that

66
00:05:35,680 --> 00:05:43,560
are accepting referrals to treat women and girls and they're trained on referrals treated

67
00:05:43,560 --> 00:05:48,140
with no financial charge to patients.

68
00:05:48,140 --> 00:05:54,440
So I guess what that means is they're treating patients that they've received referrals from

69
00:05:54,440 --> 00:05:59,280
women and girls where they're not going to be charging them for services.

70
00:05:59,280 --> 00:06:02,000
Target group number 13 are health providers.

71
00:06:02,000 --> 00:06:09,800
They're trained in epidemic management and rapid response during a humanitarian emergency.

72
00:06:09,800 --> 00:06:16,440
Target group number 14 is remote areas and this is where mobile clinics go to those remote

73
00:06:16,440 --> 00:06:23,560
areas deployed to deliver free medical services including sexual reproductive health services.

74
00:06:23,560 --> 00:06:30,200
Target group 15 are supervision teams and they're trained on quote provider supervision

75
00:06:30,200 --> 00:06:36,840
techniques and finally target group 16, I don't know what the acronym stands for FOSA,

76
00:06:36,840 --> 00:06:38,420
FOSA.

77
00:06:38,420 --> 00:06:44,480
They are supervised and supported to deliver primary health care and treat gender based

78
00:06:44,480 --> 00:06:46,160
violence cases.

79
00:06:46,160 --> 00:06:50,960
So that gives you an idea of the target groups and the services delivered at a cost of roughly

80
00:06:50,960 --> 00:06:52,960
$14 million.

81
00:06:52,960 --> 00:07:00,760
So now we're going to look at the eight outcomes and for those eight outcomes the 15 indicators

82
00:07:00,760 --> 00:07:08,840
that were used to measure whether or not those eight outcomes were achieved and there could

83
00:07:08,840 --> 00:07:14,700
even be more but from my review of the performance measurement framework this is what I figured

84
00:07:14,700 --> 00:07:18,880
out were the eight expected outcomes.

85
00:07:18,880 --> 00:07:27,360
The first one was and I quote increased mobilization of actors to take into account gender inequalities

86
00:07:27,360 --> 00:07:33,720
in particular sexual reproductive health issues in emergency response.

87
00:07:33,720 --> 00:07:41,340
The outcome indicator number one, actors involved is measured enough and also there's no comparison

88
00:07:41,340 --> 00:07:48,860
or control group outside of the targeted areas for this project so we don't know if the number

89
00:07:48,860 --> 00:07:54,080
of actors outside of the project also were involved in issues of gender inequality and

90
00:07:54,080 --> 00:07:57,520
access to health in the response emergency.

91
00:07:57,520 --> 00:08:03,760
So keeping with the trailer for season two this would violate the development assistance

92
00:08:03,760 --> 00:08:09,660
committee criteria on impact because there's no comparison group or control group so they

93
00:08:09,660 --> 00:08:14,660
can't make a claim that any difference they see in this indicator such as an increase

94
00:08:14,660 --> 00:08:22,360
in the number of actors it could happen anyways regardless of the project so it violates the

95
00:08:22,360 --> 00:08:29,720
impact criteria for the development assistance committee and this is a big thing in international

96
00:08:29,720 --> 00:08:39,200
development where you'll see they often go to these five questions or issues of evaluation

97
00:08:39,200 --> 00:08:45,980
in the development assistance committee criteria one relevance two coherence three effectiveness

98
00:08:45,980 --> 00:08:51,220
four efficiency and five impact and six sustainability.

99
00:08:51,220 --> 00:08:55,580
So in this performance measurement framework in this project most of the violations are

100
00:08:55,580 --> 00:09:01,560
on impact they just simply can't claim that the project is responsible for achieving the

101
00:09:01,560 --> 00:09:09,080
outcomes so indicator number two for this same outcome is the number of advocacy priorities

102
00:09:09,080 --> 00:09:15,520
taking into account gender inequalities in particular sexual reproductive health issues

103
00:09:15,520 --> 00:09:22,320
in the emergency response presented to the authorities for consideration in local and

104
00:09:22,320 --> 00:09:28,000
national policies again problem is it's not measured frequently enough and there's no

105
00:09:28,000 --> 00:09:34,500
comparison group if we look at the performance measurement framework I can't see it right

106
00:09:34,500 --> 00:09:39,680
here I'm claiming it's not measured very frequently it's probably measured only once every it's

107
00:09:39,680 --> 00:09:47,200
measured up every six months the first one that we just covered so anything could happen

108
00:09:47,200 --> 00:09:52,440
in six months between those measures actors in other communities outside of the project

109
00:09:52,440 --> 00:09:58,320
could also be getting mobilized and it's the same for this indicator here indicator number

110
00:09:58,320 --> 00:10:05,500
two they only measuring it every six months and the other thing that's interesting here

111
00:10:05,500 --> 00:10:11,000
and it's it's been done before is presented to the authorities for consideration in local

112
00:10:11,000 --> 00:10:17,520
and national policies a policy is just a piece of paper what you could do instead is just

113
00:10:17,520 --> 00:10:24,300
see if expenditures addressing gender gender inequalities has gone up actually expenditures

114
00:10:24,300 --> 00:10:29,360
not the budget that they claim they're going to spend but after what have they actually

115
00:10:29,360 --> 00:10:35,440
follow the money and see if it's been spent so for example did they spend money to increase

116
00:10:35,440 --> 00:10:41,440
enrollment of girls in primary schools or did the number of girls or women with access

117
00:10:41,440 --> 00:10:46,840
to sexual reproductive health services the number of visits free of charge actually go

118
00:10:46,840 --> 00:10:52,960
up thanks to maybe an increase in expenditures in that area by the government so that would

119
00:10:52,960 --> 00:11:00,160
be better measures rather than this advocacy priority taking into account gender inequalities

120
00:11:00,160 --> 00:11:07,880
to see if after the advocacy did they actually result in increased expenditures and again

121
00:11:07,880 --> 00:11:12,360
you could still argue even if it goes up the expenditures it may have nothing to do with

122
00:11:12,360 --> 00:11:16,560
the project that's why you need to have some sort of comparison group or control group

123
00:11:16,560 --> 00:11:25,040
the next outcome is called increased co-responsibility of men and adolescents in favor of access

124
00:11:25,040 --> 00:11:34,960
for all to the right to health in the targeted areas and the outcome indicator for this outcome

125
00:11:34,960 --> 00:11:40,720
this is outcome indicator number three for outcome number two proportion of men and adolescents

126
00:11:40,720 --> 00:11:49,380
who participated in project activities who report having adopted behaviors in favor of

127
00:11:49,380 --> 00:11:56,640
equitable gender norms obviously there's self-reporting bias there they're going to say sure sure

128
00:11:56,640 --> 00:12:02,160
I adopted the right behaviors because it's hard to measure the behaviors directly so

129
00:12:02,160 --> 00:12:07,480
self-reporting is going to have a bias so even more importantly here you need another

130
00:12:07,480 --> 00:12:13,520
comparison group outside of the project where men and boys weren't being engaged with to

131
00:12:13,520 --> 00:12:21,200
change their attitudes right so that is what you need to do here and again this violates

132
00:12:21,200 --> 00:12:28,700
the impact criteria the DAC development existence committee criteria because we don't have a

133
00:12:28,700 --> 00:12:35,200
control group we can't claim that the men and boys outside of the quote targeted areas

134
00:12:35,200 --> 00:12:41,920
also had a preferable shift in attitude we just don't know so we need to measure men

135
00:12:41,920 --> 00:12:47,400
and boys in areas outside of where the project is going on and this is a challenge of course

136
00:12:47,400 --> 00:12:53,560
but it should be at least attempted so again for that increased co-responsibility outcome

137
00:12:53,560 --> 00:12:58,160
there's another indicator they use outcome indicator number four proportion of men and

138
00:12:58,160 --> 00:13:04,600
adolescents participating in activities that have quote increased their knowledge of the

139
00:13:04,600 --> 00:13:09,880
right to health including sexual reproductive health and the fight against gender-based

140
00:13:09,880 --> 00:13:16,400
violence now here you're measuring knowledge levels which is fine but again what you need

141
00:13:16,400 --> 00:13:23,040
to do is have a comparison group because their knowledge levels could go up anyways in other

142
00:13:23,040 --> 00:13:29,300
areas where they're not operating the project so again this violates the impact criteria

143
00:13:29,300 --> 00:13:35,440
for the development assistance committee criteria on evaluation the next outcome number three

144
00:13:35,440 --> 00:13:42,760
is strengthened capacities of women and adolescent girls affected by the conflict to exercise

145
00:13:42,760 --> 00:13:49,360
their right to health in particular with regard to sexual reproductive health in the targeted

146
00:13:49,360 --> 00:13:57,480
areas and the outcome indicator number five for that is proportion of women and adolescent

147
00:13:57,480 --> 00:14:04,200
girls participating in the empowerment program who have quote increased their knowledge of

148
00:14:04,200 --> 00:14:10,840
sexual reproductive health so again if you want to show an increase in knowledge you

149
00:14:10,840 --> 00:14:15,960
have to show due to the project so ideally what you'd like to do is find a comparison

150
00:14:15,960 --> 00:14:21,000
group of women and adolescent girls outside of the project and see how their knowledge

151
00:14:21,000 --> 00:14:27,720
levels are going before during and after the project and then you could show again that

152
00:14:27,720 --> 00:14:36,400
the project is having some statistically significant one-tail positive effect right just to find

153
00:14:36,400 --> 00:14:41,320
some women and girls not in the project areas another option is if you don't want to use

154
00:14:41,320 --> 00:14:46,560
control groups you could just measure more frequently and just show that the knowledge

155
00:14:46,560 --> 00:14:53,360
levels are going up rather than just a pre and a post test level I again I'd have to

156
00:14:53,360 --> 00:15:01,400
look at the performance measurement framework in detail to see how frequently they measure

157
00:15:01,400 --> 00:15:07,920
this another outcome indicator for this strength and capacity of women and girls specific to

158
00:15:07,920 --> 00:15:14,400
their knowledge levels is quote number of new initiatives in favor of sexual reproductive

159
00:15:14,400 --> 00:15:21,600
health or the right to health taken by influential women and their allies so this is a separate

160
00:15:21,600 --> 00:15:28,780
group of women who take it up and again there's no comparison group here so this violates

161
00:15:28,780 --> 00:15:34,120
the impact criteria and here I have made a note according to the performance measurement

162
00:15:34,120 --> 00:15:42,040
framework this is only measured once a year so we could experience this increase anyways

163
00:15:42,040 --> 00:15:49,480
in between the time that the women and girls are trained on their capacity for their right

164
00:15:49,480 --> 00:15:56,080
to health in targeted areas and whether or not these new initiatives have been taken

165
00:15:56,080 --> 00:16:02,080
up by influential women and their allies so it's only measured once a year anyways so

166
00:16:02,080 --> 00:16:07,280
even without the control group once a year is not frequent enough outcome number four

167
00:16:07,280 --> 00:16:14,200
women and adolescent girls sensitized and supported in terms of health and gender-based

168
00:16:14,200 --> 00:16:21,800
violence in targeted areas and outcome indicator number seven for that outcome number four

169
00:16:21,800 --> 00:16:27,840
is proportion of women and adolescent girls in the community who have completed the health

170
00:16:27,840 --> 00:16:35,160
sexual reproductive health and gender-based violence empowerment program in targeted areas

171
00:16:35,160 --> 00:16:44,720
that's just attendance showing up but that doesn't indicate how many quote got sensitized

172
00:16:44,720 --> 00:16:50,520
they showed up for the program but did they get sensitized whatever that means from receiving

173
00:16:50,520 --> 00:16:56,360
that empowerment program so it's just attendance so what they need to do is find a better indicator

174
00:16:56,360 --> 00:17:02,880
to actually measure that sensitization due to the empowerment program that they have

175
00:17:02,880 --> 00:17:08,960
received and I also looked at the performance measurement framework and noticed the annual

176
00:17:08,960 --> 00:17:15,040
empowerment program activity reports once a year so again it's not very frequent if

177
00:17:15,040 --> 00:17:20,280
they want to show that their sensitivity levels have gone up or their empowerment levels have

178
00:17:20,280 --> 00:17:26,220
gone up they have to measure more frequently than just once a year and then get a better

179
00:17:26,220 --> 00:17:31,720
indicator rather than just showing up attendance who completed the program which is what they

180
00:17:31,720 --> 00:17:40,960
currently have and also maybe have a comparison group if they can here is it's a good example

181
00:17:40,960 --> 00:17:45,440
of how the indicator fails to validly measure the outcome in the first place where they're

182
00:17:45,440 --> 00:17:52,560
just taking attendance rather than measuring any levels of sensitivity or empowerment so

183
00:17:52,560 --> 00:17:59,600
outcome number five community and religious leaders quote strengthened and mobilized for

184
00:17:59,600 --> 00:18:05,840
the promotion of the right to health for all including sexual reproductive health including

185
00:18:05,840 --> 00:18:12,400
the fight against gender-based violence this was initially reported as an output but strengthening

186
00:18:12,400 --> 00:18:19,680
suggests the ability to advocate and promote so I considered it an outcome the outcome

187
00:18:19,680 --> 00:18:24,880
indicator for this is the number of community and religious leaders supported in facilitating

188
00:18:24,880 --> 00:18:31,040
awareness sessions this would suggest just attendance they're not really looking at their

189
00:18:31,040 --> 00:18:38,040
ability to mobilize their ability to advocate and their ability to promote so again the

190
00:18:38,040 --> 00:18:44,000
indicator fails to measure any increases in the ability to promote health rights for all

191
00:18:44,000 --> 00:18:50,400
it only shows the number who were supporting in facilitating awareness sessions but no

192
00:18:50,400 --> 00:18:56,520
measure showing that their ability to promote health rights for all has been strengthened

193
00:18:56,520 --> 00:19:01,280
so they should introduce a measure that shows that those who attended the community and

194
00:19:01,280 --> 00:19:08,200
religious leaders who delivered the awareness sessions experienced increases in awareness

195
00:19:08,200 --> 00:19:13,600
preferably an indicator and measurement tool that has been shown to be a valid measure

196
00:19:13,600 --> 00:19:20,360
of the ability to promote health rights for all and increase awareness plus measuring

197
00:19:20,360 --> 00:19:26,200
those who attended on their awareness levels against a non-project target group so outcome

198
00:19:26,200 --> 00:19:33,640
number six increased capacity of conflict affected communities especially women and

199
00:19:33,640 --> 00:19:40,080
adolescent girls to utilize primary health care services including sexual reproductive

200
00:19:40,080 --> 00:19:47,880
health outcome indicator nine for this outcome number six proportion of women men and adolescents

201
00:19:47,880 --> 00:19:55,160
targeted by the project who can name at least three types of modern contraceptive methods

202
00:19:55,160 --> 00:20:02,960
again this is a knowledge indicator so if you want to show that the project services

203
00:20:02,960 --> 00:20:10,720
targeting these particular individuals has increased their knowledge on contraception

204
00:20:10,720 --> 00:20:18,040
you need to have a comparison group or a control group so this again violates the impact criteria

205
00:20:18,040 --> 00:20:25,040
for the development assistance committee criteria on evaluation the performance measurement

206
00:20:25,040 --> 00:20:31,440
framework shows that the percent is measured at the start middle and end of the project

207
00:20:31,440 --> 00:20:38,000
but no mention of a controller comparison group so with a control group and the statistical

208
00:20:38,000 --> 00:20:44,360
sampling they mention in the performance measurement framework they actually mention it they could

209
00:20:44,360 --> 00:20:51,200
complete if we follow their interest in statistics a one-tailed hypothesis test and see with

210
00:20:51,200 --> 00:20:59,200
95 percent confidence if they can claim that the percent increases were statistically significant

211
00:20:59,200 --> 00:21:06,280
and due to the project training so that's what they have to do otherwise right now it

212
00:21:06,280 --> 00:21:13,560
violates the DAC criteria on impact because they're not showing that their project group

213
00:21:13,560 --> 00:21:18,280
is statistically significantly greater in terms of the percentages who can name at least

214
00:21:18,280 --> 00:21:24,900
three types of modern contraceptive methods another outcome indicator for that outcome

215
00:21:24,900 --> 00:21:31,080
number six this is outcome number indicator number ten proportion of women men and adolescents

216
00:21:31,080 --> 00:21:37,560
quote who have improved their knowledge during training and awareness raising on the use

217
00:21:37,560 --> 00:21:44,380
of primary health care services including sexual reproductive health same issue no comparison

218
00:21:44,380 --> 00:21:48,560
or control group and they according to the performance measurement framework they only

219
00:21:48,560 --> 00:21:53,120
measure this once before the training and after the training so that could show an increase

220
00:21:53,120 --> 00:22:01,300
due to the training but if we don't have a control group or a comparison group you could

221
00:22:01,300 --> 00:22:06,000
show also another group of women who didn't show up for the training somewhere else in

222
00:22:06,000 --> 00:22:14,420
the country who also just over time improved increased their knowledge levels and awareness

223
00:22:14,420 --> 00:22:23,080
levels so that has to be taken into account and once again that violates the DAC development

224
00:22:23,080 --> 00:22:29,720
assistance committee criteria on impact evaluation outcome number seven improving the social

225
00:22:29,720 --> 00:22:35,360
community and political environment in favor of the right to health in particular sexual

226
00:22:35,360 --> 00:22:41,200
and reproductive rights and gender equality for women and adolescent girls affected by

227
00:22:41,200 --> 00:22:48,440
the conflict in northern Burkina Faso outcome indicator 11 for this outcome number seven

228
00:22:48,440 --> 00:22:55,720
proportion of women men and adolescents targeted by the project having improved their behavior

229
00:22:55,720 --> 00:23:02,260
in favor of sexual and reproductive rights this is tricky because I'm assuming having

230
00:23:02,260 --> 00:23:08,680
improved their behavior this has to be self-reporting which is biased and again no control group

231
00:23:08,680 --> 00:23:13,800
or comparison group outside of the project so even if these percentages increase you

232
00:23:13,800 --> 00:23:18,000
need a control group where these women men and boys have not received the various trainings

233
00:23:18,000 --> 00:23:24,040
etc from this project their behaviors could also increase so the DAC impact criteria is

234
00:23:24,040 --> 00:23:30,320
not met here also same in outcome number seven we have another outcome indicator number 12

235
00:23:30,320 --> 00:23:37,880
number of sexual reproductive health initiatives undertaken by civil society organizations

236
00:23:37,880 --> 00:23:44,640
that seek to mitigate traditional practices harmful to women and adolescent girls according

237
00:23:44,640 --> 00:23:51,760
to the performance measurement framework this is only measured once per year and again there's

238
00:23:51,760 --> 00:23:57,720
no control group so even if you don't have a control group or a comparison group outside

239
00:23:57,720 --> 00:24:05,360
of the project you could measure it more frequently and that might help in allaying concerns that

240
00:24:05,360 --> 00:24:13,080
these initiatives would happen anyways regardless of the trainings that these women men and

241
00:24:13,080 --> 00:24:20,880
adolescents or CSO civil society organizations receive to uptake these initiatives so that's

242
00:24:20,880 --> 00:24:28,040
the other problem there again no control group no comparison group so the DAC criteria of

243
00:24:28,040 --> 00:24:36,680
impact cannot be claimed to be achieved another indicator for this same outcome outcome indicator

244
00:24:36,680 --> 00:24:44,400
number 13 number of national policies or guidelines adopted or revised taking into account gender

245
00:24:44,400 --> 00:24:51,560
inequalities in particular sexual reproductive health issues in the emergency response outcome

246
00:24:51,560 --> 00:24:55,720
indicator third now again you could argue here that the number of national policies

247
00:24:55,720 --> 00:25:03,680
adopted or revised is really not related to the outcome of improving the political environment

248
00:25:03,680 --> 00:25:09,400
in favor of the right to health in particular the right to access to sexual reproductive

249
00:25:09,400 --> 00:25:16,360
health services because as I mentioned earlier a policy is just a piece of paper a better

250
00:25:16,360 --> 00:25:25,120
measure would be an actual increase in the amount and percentage of the government budget

251
00:25:25,120 --> 00:25:31,960
that was actually spent on women and girls specific to accessing sexual reproductive

252
00:25:31,960 --> 00:25:36,600
health and gender based violence services so even though they have a policy on a piece

253
00:25:36,600 --> 00:25:41,740
of paper do they actually put their money where their policy is this is a common sort

254
00:25:41,740 --> 00:25:50,800
of indicator that many governance organizations when monitoring governance and government

255
00:25:50,800 --> 00:25:57,060
institutional performance look at because they can have all the policies in the world

256
00:25:57,060 --> 00:26:02,040
but really what you need to do is follow the money especially when it comes to in this

257
00:26:02,040 --> 00:26:10,760
project they are doing a large portion of this 14 million dollars in advocacy so really

258
00:26:10,760 --> 00:26:17,860
what you should be doing is tracking government expenditures outcome number eight increased

259
00:26:17,860 --> 00:26:23,720
enjoyment of rights relating to health in particular sexual and reproductive health

260
00:26:23,720 --> 00:26:30,920
of women men girls and boys affected by the conflicts in northern Burkina Faso here we

261
00:26:30,920 --> 00:26:38,240
have two indicators and I look at these indicators and I think they're fairly universal in this

262
00:26:38,240 --> 00:26:44,480
area and the first indicator outcome indicator number 14 of the project is percentage age

263
00:26:44,480 --> 00:26:52,360
15 to 19 who have given birth to their first child who or who are pregnant with their first

264
00:26:52,360 --> 00:26:57,760
child and the idea clearly there is to try to reduce this percentage and I think it's

265
00:26:57,760 --> 00:27:05,240
a universal global indicator that's used which is fine the indicator is a valid measure of

266
00:27:05,240 --> 00:27:14,500
the outcome of increased access and enjoyment of rights relating to health and what is tricky

267
00:27:14,500 --> 00:27:20,040
here is this is the only indicator where you could argue it is a valid measure but again

268
00:27:20,040 --> 00:27:26,320
all of the services I described for those 16 target groups that this 14 million dollars

269
00:27:26,320 --> 00:27:31,360
is paying for in the delivery of services to those target groups and you're hoping that

270
00:27:31,360 --> 00:27:38,880
this indicator will go down that's fine but to show that you need to have some sort of

271
00:27:38,880 --> 00:27:45,400
comparison group or ideally in the demographic health survey they call it where they measure

272
00:27:45,400 --> 00:27:53,360
this one of the questions on that demographic health survey should be by the way did you

273
00:27:53,360 --> 00:28:01,000
participate in any way shape or form in the doctors of the world project paid for by global

274
00:28:01,000 --> 00:28:07,280
affairs Canada and right there you could do a split and see without getting too methodologically

275
00:28:07,280 --> 00:28:14,520
complex here because I don't claim I'm a statistician happy to get email saying sorry that's not

276
00:28:14,520 --> 00:28:20,720
how you do it but at least we could compare the two groups when we do the survey and those

277
00:28:20,720 --> 00:28:26,360
who say no what was their rate of pregnancy between age 15 to 19 compared to the group

278
00:28:26,360 --> 00:28:34,000
that said yeah I did show up at those awareness sessions on contraception and then you could

279
00:28:34,000 --> 00:28:39,480
do a comparison but right now I don't see that in the performance measurement framework

280
00:28:39,480 --> 00:28:44,440
it's a great indicator but we don't know if the project for 14 million dollars actually

281
00:28:44,440 --> 00:28:52,840
contributed to a reduction in that percentage age 15 to 19 that reported being pregnant

282
00:28:52,840 --> 00:28:59,520
we just don't know and finally the last outcome indicator number 15 proportion of women who

283
00:28:59,520 --> 00:29:06,160
have decided to use family planning alone or jointly with their husband or partner again

284
00:29:06,160 --> 00:29:12,760
an excellent indicator that suggests if this project is successful and has an impact quote

285
00:29:12,760 --> 00:29:18,720
unquote this indicator would be going up the proportion of women who have decided to use

286
00:29:18,720 --> 00:29:23,960
family planning would be going up and it would be going up statistically significantly compared

287
00:29:23,960 --> 00:29:30,960
to a group of women who were not participating in the project so ideally what you want to

288
00:29:30,960 --> 00:29:36,400
know if this question is being asked in the demographic health survey did they participate

289
00:29:36,400 --> 00:29:42,520
in this doctors of the world Burkina Faso project in any way and then you could see

290
00:29:42,520 --> 00:29:49,280
if the project was in any way responsible for contributing to this increased improvement

291
00:29:49,280 --> 00:29:55,840
and performance on that indicator so that's my summary and as I'll repeat I would say

292
00:29:55,840 --> 00:30:02,320
in its current form right now the performance measurement framework is flawed and the project

293
00:30:02,320 --> 00:30:09,520
and doctors of the world cannot make the claim that their 14 million dollar project is achieving

294
00:30:09,520 --> 00:30:16,560
the outcomes that they claim they're achieving in the project so now I'm going to send this

295
00:30:16,560 --> 00:30:21,400
episode to the Minister for International Development along with the shadow critics

296
00:30:21,400 --> 00:30:27,720
and I'm also going to invite doctors of the world to provide a written response to this

297
00:30:27,720 --> 00:30:34,160
critique and also an invitation if they wish to be on the podcast to respond to the critique

298
00:30:34,160 --> 00:30:41,400
and if that happens we'll be doing another episode thank you for listening and stay tuned

299
00:30:41,400 --> 00:30:50,080
for a further episode and I do have some good news that we have located in the United States

300
00:30:50,080 --> 00:30:58,120
the actual grant for an American USAID project that was funded to an implementing partner

301
00:30:58,120 --> 00:31:04,980
and so hopefully that will in turn result in getting the activity monitoring evaluation

302
00:31:04,980 --> 00:31:10,920
learning plan and then we can critique that which is similar to these Canadian performance

303
00:31:10,920 --> 00:31:20,480
measurement frameworks bye for now