1 00:00:00,000 --> 00:00:03,260 Improving Development Evaluation Podcast. 2 00:00:03,260 --> 00:00:06,880 I'm your host David Wand and welcome to episode seven, 3 00:00:06,880 --> 00:00:11,880 part one, where we introduce you to another new 4 00:00:12,040 --> 00:00:15,000 international development organization. 5 00:00:15,000 --> 00:00:17,480 And that international development organization 6 00:00:17,480 --> 00:00:22,480 for episode seven is Equitas, E-Q-U-I-T-A-S. 7 00:00:23,520 --> 00:00:26,640 Their headquarters is in Montreal, Canada, 8 00:00:26,640 --> 00:00:31,640 and you can learn more about Equitas at www.equitas.org. 9 00:00:35,480 --> 00:00:38,620 The name of the project that they have been provided 10 00:00:38,620 --> 00:00:41,620 with funding to the tune of $18,280,395 11 00:00:45,460 --> 00:00:48,640 from the taxpayers of Canada is called 12 00:00:48,640 --> 00:00:52,220 Achieving Equality Through Human Rights Education. 13 00:00:52,220 --> 00:00:57,140 And they're delivering this project in Burkina Faso, 14 00:00:57,140 --> 00:01:01,660 Haiti, Kenya, Senegal, and Tanzania. 15 00:01:01,660 --> 00:01:06,300 But they're also spending 35% of that 18 million 16 00:01:06,300 --> 00:01:10,380 in regions outside of those five countries, 17 00:01:10,380 --> 00:01:15,200 including north of Sahara in Africa and south of Sahara, 18 00:01:15,200 --> 00:01:20,200 as well as about 20% out of that 35%. 19 00:01:20,200 --> 00:01:24,740 And then percent in Asia and 4% in Americas. 20 00:01:24,740 --> 00:01:28,740 In this project, there were 10 outcomes. 21 00:01:28,740 --> 00:01:31,020 And for each outcome in the performance 22 00:01:31,020 --> 00:01:34,780 measurement framework, they had two outcome indicators 23 00:01:34,780 --> 00:01:37,720 for a total of 20 outcome indicators, 24 00:01:37,720 --> 00:01:41,940 measuring whether or not they achieved those 10 outcomes. 25 00:01:43,200 --> 00:01:46,020 And as I've mentioned before, if any of you are interested 26 00:01:46,020 --> 00:01:49,100 in receiving the performance measurement framework 27 00:01:49,100 --> 00:01:50,940 for this project or any other project 28 00:01:50,940 --> 00:01:53,460 that I've covered so far in the podcast, 29 00:01:53,460 --> 00:01:58,460 as well as the Excel summary of the outcome indicators 30 00:01:59,040 --> 00:02:01,180 for the project, feel free to email me 31 00:02:01,180 --> 00:02:05,620 at evaluatecanadaaid at gmail.com. 32 00:02:05,620 --> 00:02:07,640 I'll put that in the episode notes. 33 00:02:09,240 --> 00:02:13,660 So before we begin a brief description of the project 34 00:02:13,660 --> 00:02:16,020 in terms of the services they deliver, 35 00:02:16,020 --> 00:02:20,460 I'll jump to the chase here, cut to the chase 36 00:02:20,460 --> 00:02:24,740 and tell you that none of the 20 outcome indicators 37 00:02:24,740 --> 00:02:27,660 adequately measure the outcomes. 38 00:02:27,660 --> 00:02:31,700 So Equitas cannot make the claim that they're achieving 39 00:02:31,700 --> 00:02:34,180 the outcomes in their performance measurement framework 40 00:02:34,180 --> 00:02:38,380 simply because the way they're going about measuring them 41 00:02:38,380 --> 00:02:39,380 is inadequate. 42 00:02:40,620 --> 00:02:43,740 So with that summary, we're gonna move on 43 00:02:43,740 --> 00:02:48,740 to a brief description of the services that this project 44 00:02:49,060 --> 00:02:54,060 entitled Achieving Equality Through Human Rights Education, 45 00:02:54,180 --> 00:02:55,880 what services they're delivering. 46 00:02:57,620 --> 00:03:02,620 They have approximately nine target groups in the project. 47 00:03:03,860 --> 00:03:06,420 So starting with the first target group, 48 00:03:06,420 --> 00:03:10,060 women's organizations, they receive quote, 49 00:03:10,060 --> 00:03:13,960 support to participate in events on advancing 50 00:03:13,960 --> 00:03:16,860 gender equality, the problem there is we don't know 51 00:03:16,860 --> 00:03:20,600 what they mean by support, there's no clear description 52 00:03:20,600 --> 00:03:22,240 of what they mean by support. 53 00:03:23,400 --> 00:03:25,940 Women's organizations also receive training 54 00:03:25,940 --> 00:03:29,420 on engaging duty bearers to fulfill their human rights 55 00:03:29,420 --> 00:03:31,800 obligations to advance gender equality. 56 00:03:33,460 --> 00:03:36,460 Women's organizations also receive support 57 00:03:36,460 --> 00:03:39,620 in their continued engagement with duty bearers 58 00:03:39,620 --> 00:03:41,300 on the advancement of gender equality, 59 00:03:41,300 --> 00:03:44,580 but again, it's not clear in their performance measurement 60 00:03:44,580 --> 00:03:48,220 framework what exactly they mean by support. 61 00:03:48,220 --> 00:03:50,920 Women's organizations also develop action plans 62 00:03:50,920 --> 00:03:52,820 to advance gender equality. 63 00:03:53,700 --> 00:03:57,740 They also participate in a number of joint 64 00:03:57,740 --> 00:04:02,740 awareness activities and they also share globally 65 00:04:03,860 --> 00:04:06,780 good practices in human rights based approach 66 00:04:06,780 --> 00:04:09,940 in gender equality, they share those globally 67 00:04:10,820 --> 00:04:14,140 and they also do context analysis related 68 00:04:14,140 --> 00:04:15,460 to gender equality. 69 00:04:17,780 --> 00:04:20,420 The second target group are called duty bearers 70 00:04:22,420 --> 00:04:25,020 and they are interviewed when women's organizations 71 00:04:25,020 --> 00:04:28,460 contact them on their advancement of gender equality. 72 00:04:29,620 --> 00:04:34,620 They also develop action plans to advance gender equality. 73 00:04:34,620 --> 00:04:37,560 They also receive training on integrating 74 00:04:37,560 --> 00:04:40,600 a human rights based approach into their work 75 00:04:40,600 --> 00:04:42,600 to advance gender equality. 76 00:04:42,600 --> 00:04:45,120 Local structures are also supported to become 77 00:04:45,120 --> 00:04:49,400 more inclusive of women and while that is going on, 78 00:04:49,400 --> 00:04:51,080 the duty bearers are interviewed. 79 00:04:51,080 --> 00:04:54,400 Also duty bearers attend community forums 80 00:04:54,400 --> 00:04:56,760 that are promoting gender equality 81 00:04:56,760 --> 00:04:58,380 and women's participation. 82 00:04:59,840 --> 00:05:03,240 They're also involved in promoting gender equality 83 00:05:03,240 --> 00:05:06,480 through community actions and they also are involved 84 00:05:06,480 --> 00:05:09,640 in the context analysis on gender equality. 85 00:05:09,640 --> 00:05:13,760 A third target group are human rights organizations 86 00:05:13,760 --> 00:05:17,000 and they participate in a number of joint awareness 87 00:05:17,000 --> 00:05:20,800 raising activities for advancing gender equality. 88 00:05:20,800 --> 00:05:25,200 They also are sharing globally good practices 89 00:05:25,200 --> 00:05:28,300 in human rights based approach in gender equality. 90 00:05:28,300 --> 00:05:33,300 A fourth target group are civil society organizations 91 00:05:33,340 --> 00:05:35,740 and they are supported by the project 92 00:05:35,740 --> 00:05:40,340 to participate in alliances to advance gender equality 93 00:05:40,340 --> 00:05:44,020 but again, we don't know exactly what that support entails 94 00:05:44,020 --> 00:05:45,780 and in a good performance framework, 95 00:05:46,900 --> 00:05:49,820 measurement framework for outputs which these are, 96 00:05:49,820 --> 00:05:52,260 you would think they would quantify this 97 00:05:52,260 --> 00:05:55,740 by figuring out what they mean by support 98 00:05:55,740 --> 00:05:58,660 so that it could be better monitored. 99 00:05:58,660 --> 00:06:02,060 A fifth target group are partners, 100 00:06:02,060 --> 00:06:04,460 beneficiaries and intermediaries. 101 00:06:04,460 --> 00:06:07,020 They're also involved in the context analysis 102 00:06:07,020 --> 00:06:08,820 on gender equality. 103 00:06:08,820 --> 00:06:12,620 They're also involved in sharing globally good practices 104 00:06:12,620 --> 00:06:15,420 in human rights based approach gender equality 105 00:06:15,420 --> 00:06:19,060 and they're also participating in the community forums. 106 00:06:19,060 --> 00:06:22,660 A sixth target group are participants 107 00:06:22,660 --> 00:06:27,660 and these participants participate in two training sessions, 108 00:06:27,740 --> 00:06:32,140 one at the regional level held in East and West Africa 109 00:06:32,140 --> 00:06:37,140 on human rights training but also taxpayers be aware, 110 00:06:37,980 --> 00:06:40,660 there's one being held in Canada 111 00:06:40,660 --> 00:06:42,740 on training on human rights 112 00:06:42,740 --> 00:06:45,820 and according to the performance measurement framework, 113 00:06:45,820 --> 00:06:49,900 461 people are gonna show up 114 00:06:49,900 --> 00:06:52,940 and if you do this math, that's about 1500 bucks 115 00:06:52,940 --> 00:06:56,580 for an airfare, I mean Haiti's probably a bit cheaper 116 00:06:56,580 --> 00:07:00,060 but return airfare from Africa to Canada 117 00:07:00,060 --> 00:07:04,020 is about 1500 Canadian, multiply that by 461 118 00:07:04,020 --> 00:07:06,580 and that lovely little human rights training session 119 00:07:06,580 --> 00:07:11,580 in Canada is costing you $691,500 Canadian dollars. 120 00:07:13,580 --> 00:07:16,300 That's quite an expensive output. 121 00:07:16,300 --> 00:07:20,100 Number seven target group, leaders of local structures. 122 00:07:20,100 --> 00:07:24,660 They also are supported to become more inclusive of women. 123 00:07:24,660 --> 00:07:29,220 Problem is again, they don't provide detail exactly 124 00:07:29,220 --> 00:07:32,500 of what they mean by support 125 00:07:32,500 --> 00:07:34,180 and if you look at the indicators, 126 00:07:34,180 --> 00:07:36,900 it's not clear how many sessions 127 00:07:36,900 --> 00:07:39,700 or what they're doing there. 128 00:07:39,700 --> 00:07:44,040 The eighth target group are the partners in the countries, 129 00:07:44,040 --> 00:07:44,940 the developing countries 130 00:07:44,940 --> 00:07:47,780 and they receive organizational capacity building, 131 00:07:47,780 --> 00:07:50,620 training from Equitas 132 00:07:50,620 --> 00:07:54,300 and finally target group number nine are coaches 133 00:07:54,300 --> 00:07:57,580 and they are trained on community mobilization 134 00:07:57,580 --> 00:08:02,140 and human rights education to coach local organizations. 135 00:08:02,140 --> 00:08:04,260 I think they mean the local structures 136 00:08:04,260 --> 00:08:05,940 but again, it's not clear. 137 00:08:05,940 --> 00:08:08,940 The funny thing about this is even though the coaches 138 00:08:08,940 --> 00:08:12,840 are trained on this, after the training, 139 00:08:12,840 --> 00:08:14,300 everybody goes home. 140 00:08:14,300 --> 00:08:18,500 There's no indication that the coaches are measured 141 00:08:18,500 --> 00:08:19,980 on whether or not they've learned 142 00:08:19,980 --> 00:08:22,020 what they were supposed to learn in the training. 143 00:08:22,020 --> 00:08:25,660 So that's a brief description of the services 144 00:08:25,660 --> 00:08:28,460 that the project is delivering. 145 00:08:28,460 --> 00:08:31,380 Now I thought I would just give you just two examples 146 00:08:31,380 --> 00:08:33,300 because in part two, we're gonna go through 147 00:08:33,300 --> 00:08:38,300 all 20 outcome indicators with another evaluation expert 148 00:08:39,060 --> 00:08:42,420 and we're also going to invite Equitas to attend part two 149 00:08:42,420 --> 00:08:44,180 to respond to our critique 150 00:08:44,180 --> 00:08:47,180 because they may have some valid explanations 151 00:08:47,180 --> 00:08:49,660 as to our critique. 152 00:08:49,660 --> 00:08:54,180 So let's start with the outcome increased commitment 153 00:08:54,180 --> 00:08:58,020 of select women's organizations to engage duty bearers 154 00:08:58,020 --> 00:09:03,020 in dialogue on advancing gender equality in target countries. 155 00:09:03,820 --> 00:09:06,300 One of the indicators for that outcome is 156 00:09:07,260 --> 00:09:11,520 number of plans implemented by select women's organizations 157 00:09:11,520 --> 00:09:14,380 to engage duty bearers in dialogue 158 00:09:14,380 --> 00:09:19,160 on advancing gender equality in target countries. 159 00:09:20,500 --> 00:09:23,580 Now the problem with that indicator is, 160 00:09:23,580 --> 00:09:26,100 and they even mentioned this in the performance 161 00:09:26,100 --> 00:09:29,500 measurement framework for another indicator, 162 00:09:29,500 --> 00:09:34,500 that they actually are concerned about quality criteria 163 00:09:34,500 --> 00:09:37,240 but they don't put it in this indicator. 164 00:09:37,240 --> 00:09:40,740 So the number of plans is not adequate. 165 00:09:40,740 --> 00:09:42,740 I mean the plans could be garbage, right? 166 00:09:42,740 --> 00:09:46,680 You need to have some sort of minimum quality criteria 167 00:09:46,680 --> 00:09:51,340 where you can say that these plans actually show 168 00:09:51,340 --> 00:09:54,020 that the women actually indeed are committed 169 00:09:54,020 --> 00:09:59,020 and they've made effort to produce high quality action plans 170 00:09:59,300 --> 00:10:03,140 to advocate for gender equality, right? 171 00:10:03,140 --> 00:10:07,180 Even better, a better indicator would be measure the percent 172 00:10:07,180 --> 00:10:10,020 of plans that achieve quality criteria 173 00:10:10,020 --> 00:10:12,180 to engage effectively. 174 00:10:13,180 --> 00:10:14,660 The other thing that's interesting, 175 00:10:14,660 --> 00:10:16,820 and you wouldn't know this unless you saw 176 00:10:16,820 --> 00:10:19,220 the performance measurement framework, 177 00:10:19,220 --> 00:10:22,580 is that in this outcome and in this indicator, 178 00:10:22,580 --> 00:10:25,820 they also interview the other side, 179 00:10:25,820 --> 00:10:29,300 that is the duty bearers that these women organizations 180 00:10:29,300 --> 00:10:34,300 go to to engage in dialogue on advancing gender equality. 181 00:10:35,120 --> 00:10:39,120 So I would say forget the women, don't interview them, 182 00:10:39,120 --> 00:10:42,100 don't measure them, or if you're gonna measure them, 183 00:10:42,100 --> 00:10:44,660 measure the quality of their plans, 184 00:10:44,660 --> 00:10:47,940 go straight to the duty bearers and ask them, 185 00:10:47,940 --> 00:10:49,620 not the women's organizations, 186 00:10:49,620 --> 00:10:53,100 those women organizations that contacted you, 187 00:10:53,100 --> 00:10:56,960 was their engagement with you of sufficient quality 188 00:10:56,960 --> 00:10:59,220 to advance gender equality? 189 00:10:59,220 --> 00:11:01,320 Do you think they knew what they were doing? 190 00:11:01,320 --> 00:11:02,860 What they were talking about? 191 00:11:02,860 --> 00:11:05,300 Did they do their research, et cetera, et cetera? 192 00:11:05,300 --> 00:11:09,420 You could go to the duty bearers instead. 193 00:11:09,420 --> 00:11:12,700 Now, the interesting thing is they interview 194 00:11:12,700 --> 00:11:17,700 the duty bearers, but we don't know if that is of any use, 195 00:11:19,640 --> 00:11:23,860 because the two indicators here for this outcome, 196 00:11:23,860 --> 00:11:25,980 they both talk about the women's organizations. 197 00:11:25,980 --> 00:11:28,840 They don't talk about the percentage of duty bearers, 198 00:11:28,840 --> 00:11:33,140 for example, who report that the women who engage with them 199 00:11:33,140 --> 00:11:35,920 in dialogue on promoting gender equality 200 00:11:35,920 --> 00:11:39,000 were of sufficient quality in their engagement. 201 00:11:39,000 --> 00:11:43,460 Neither of the outcome indicators even mentions that, 202 00:11:43,460 --> 00:11:45,480 that they're gonna measure their duty bearers, 203 00:11:45,480 --> 00:11:47,220 but it does say in the PMF 204 00:11:47,220 --> 00:11:48,660 that they're gonna interview them. 205 00:11:48,660 --> 00:11:51,500 So it makes you wonder, why are they interviewing them 206 00:11:51,500 --> 00:11:55,200 if they're not included in either of the indicators 207 00:11:55,200 --> 00:11:57,300 that they're using to measure the outcome? 208 00:11:58,260 --> 00:12:02,740 So that's one example where the outcome indicator is flawed. 209 00:12:02,740 --> 00:12:06,980 It doesn't adequately measure any increased commitment. 210 00:12:06,980 --> 00:12:09,320 And even if it did, it would be better to go 211 00:12:09,320 --> 00:12:12,580 to the duty bearers as a more objective measure, 212 00:12:12,580 --> 00:12:15,860 because you may get some self-reporting bias 213 00:12:15,860 --> 00:12:17,680 from the women organizations. 214 00:12:17,680 --> 00:12:19,520 So let's go to the next one. 215 00:12:19,520 --> 00:12:22,900 The next outcome statement I'm gonna look at 216 00:12:22,900 --> 00:12:27,040 is increased capacity of select women's organizations 217 00:12:27,040 --> 00:12:30,200 to engage duty bearers in better fulfilling 218 00:12:30,200 --> 00:12:33,700 their human rights obligations to advance gender equality 219 00:12:33,700 --> 00:12:35,500 in target countries. 220 00:12:35,500 --> 00:12:39,780 And the indicator there is level of perceived capacity 221 00:12:39,780 --> 00:12:43,660 on a four point scale of select women's organizations 222 00:12:43,660 --> 00:12:46,580 to engage duty bearers in fulfilling 223 00:12:46,580 --> 00:12:50,740 their human rights obligations to advance gender equality. 224 00:12:52,660 --> 00:12:57,460 So we're talking about capacity to lobby these duty bearers. 225 00:12:57,460 --> 00:13:00,900 But perceived capacity, as we've mentioned several times 226 00:13:00,900 --> 00:13:03,300 on this podcast and other episodes, 227 00:13:03,300 --> 00:13:05,120 it's self-reporting bias. 228 00:13:05,120 --> 00:13:07,780 The women, of course, are gonna say, 229 00:13:07,780 --> 00:13:11,820 sure, I'm much better in lobbying 230 00:13:11,820 --> 00:13:13,700 than before this project started. 231 00:13:14,960 --> 00:13:17,520 It's not a valid measure of technical capacity. 232 00:13:18,620 --> 00:13:22,800 And even if it was, even if we keep it, 233 00:13:22,800 --> 00:13:27,800 and their perceived levels of technical capacity 234 00:13:27,900 --> 00:13:31,540 to engage with these duty bearers has gone up, 235 00:13:31,540 --> 00:13:34,180 we still cannot say it was because of the project, 236 00:13:34,180 --> 00:13:36,620 because there's no comparison group. 237 00:13:36,620 --> 00:13:39,980 So what they should do is find other women's organizations 238 00:13:39,980 --> 00:13:43,800 in those countries that are not part of the project 239 00:13:43,800 --> 00:13:47,940 and ask them the same question over time. 240 00:13:47,940 --> 00:13:50,460 Then, and only then can they show 241 00:13:50,460 --> 00:13:54,780 that the perceived capacity has gone up more 242 00:13:54,780 --> 00:13:58,300 in their project compared to other women's organizations 243 00:13:58,300 --> 00:14:00,700 who answer the same question 244 00:14:00,700 --> 00:14:03,540 but have nothing to do with the project. 245 00:14:03,540 --> 00:14:06,100 So that's the major problem there. 246 00:14:06,100 --> 00:14:10,380 So thank you for listening to part one featuring Equitas. 247 00:14:11,680 --> 00:14:13,320 Stay tuned for part two, 248 00:14:13,320 --> 00:14:16,240 where I'm going to send them part one here, 249 00:14:17,120 --> 00:14:18,740 this episode, part one. 250 00:14:18,740 --> 00:14:21,820 And also I'm going to send Equitas 251 00:14:21,820 --> 00:14:23,820 the performance measurement framework, 252 00:14:23,820 --> 00:14:25,620 which they already have, 253 00:14:25,620 --> 00:14:28,060 as well as the Excel summary sheet 254 00:14:28,060 --> 00:14:31,900 that shows for each of the 20 outcome indicators 255 00:14:31,900 --> 00:14:36,180 why they are not properly designed to measure the outcomes. 256 00:14:37,100 --> 00:14:41,540 And an evaluation expert will also be exchanging, 257 00:14:42,660 --> 00:14:45,740 taking turns, going through the outcome indicators, 258 00:14:45,740 --> 00:14:49,080 explaining why we have issues with their outcome indicators. 259 00:14:49,080 --> 00:14:52,420 But we hope that Equitas will accept our invitation 260 00:14:52,420 --> 00:14:57,420 and provide an opportunity to respond to our critique. 261 00:14:58,340 --> 00:15:00,820 Thank you for listening and stay tuned 262 00:15:00,820 --> 00:15:17,820 for episode seven, part two featuring Equitas.