1 00:00:00,000 --> 00:00:05,000 Welcome to the Improving Development Evaluation Podcast. 2 00:00:05,000 --> 00:00:11,060 I'm your host David Wand and welcome back to Episode 7, Part 2, where we continue our 3 00:00:11,060 --> 00:00:17,280 discussion of the international development organization called Equitas that has their 4 00:00:17,280 --> 00:00:23,640 headquarters in Montreal, Canada, and we are talking about their project worth about 18 5 00:00:23,640 --> 00:00:29,720 million Canadian dollars, courtesy of the Canadian taxpayer, and this is a human rights 6 00:00:29,720 --> 00:00:38,120 project where they're promoting women, gender equality in five different developing countries, 7 00:00:38,120 --> 00:00:44,360 and we're going to go into more detail about their performance measurement framework for 8 00:00:44,360 --> 00:00:51,480 those five developing countries, and they are Burkina Faso, Haiti, Kenya, Senegal, and 9 00:00:51,480 --> 00:00:58,120 Tanzania, but they also use some of the 18 million dollars for promoting gender equality 10 00:00:58,120 --> 00:01:04,280 and human rights in certain regions such as West Africa, and we won't go into detail about 11 00:01:04,280 --> 00:01:06,280 Part 1, we've already done that. 12 00:01:06,280 --> 00:01:10,200 We're going to focus on the outcomes as we usually do in Part 2. 13 00:01:10,200 --> 00:01:18,240 Just to give you some background, on April 23rd, I sent Equitas their performance measurement 14 00:01:18,240 --> 00:01:23,560 framework along with our critique of their performance measurement framework that outlines 15 00:01:23,560 --> 00:01:31,280 why each of the 20 outcome indicators do not adequately measure properly the 10 outcomes 16 00:01:31,280 --> 00:01:33,880 in the project. 17 00:01:33,880 --> 00:01:42,120 Then I also sent them the Part 1 of the podcast along with the PMF and the Excel summary sheet, 18 00:01:42,120 --> 00:01:48,120 and then on April 29th, I received an email back from Equitas where they declined to attend 19 00:01:48,120 --> 00:01:54,040 Part 2, which is today, of the podcast to respond to our critique of their performance 20 00:01:54,040 --> 00:02:00,320 measurement framework, and they also declined to provide a written response as to what their 21 00:02:00,320 --> 00:02:06,260 response would be to our critique, which is unfortunate, but we will proceed as usual 22 00:02:06,260 --> 00:02:11,620 with Part 2, going through each of the 20 outcomes, explaining why they fail to properly 23 00:02:11,620 --> 00:02:13,240 measure the outcomes. 24 00:02:13,240 --> 00:02:21,600 And with me today is an impact evaluation specialist from London, Ontario, Canada. 25 00:02:21,600 --> 00:02:24,560 Her name is Yvonne Okéké. 26 00:02:24,560 --> 00:02:29,960 She has a master's degree in impact evaluation for international development from the University 27 00:02:29,960 --> 00:02:32,880 of East Anglia in the United Kingdom. 28 00:02:32,880 --> 00:02:34,560 Welcome, Yvonne. 29 00:02:34,560 --> 00:02:35,560 How are you? 30 00:02:35,560 --> 00:02:37,440 Thank you, David. 31 00:02:37,440 --> 00:02:40,000 I'm doing absolutely fine. 32 00:02:40,000 --> 00:02:41,600 How are you doing? 33 00:02:41,600 --> 00:02:49,600 I'm fine, as usual, continuing our quest to let people know that we've got some issues 34 00:02:49,600 --> 00:02:52,880 with some international development projects. 35 00:02:52,880 --> 00:03:00,760 So we will start, as usual, with the 20 outcome indicators covering the 10 outcomes, and I'll 36 00:03:00,760 --> 00:03:06,680 start with the first outcome of this performance measurement framework, which is increased 37 00:03:06,680 --> 00:03:15,120 commitment of select women's organizations to engage duty bearers in dialogue on advancing 38 00:03:15,120 --> 00:03:18,840 gender equality in target countries. 39 00:03:18,840 --> 00:03:24,760 And the outcome indicator I'm looking at is, and I quote, number of plans implemented by 40 00:03:24,760 --> 00:03:33,400 select women's organizations to engage duty bearers in dialogue on advancing gender equality 41 00:03:33,400 --> 00:03:35,920 in target countries. 42 00:03:35,920 --> 00:03:41,320 The problem with this outcome indicator, it doesn't adequately measure increased commitment 43 00:03:41,320 --> 00:03:46,640 to engage with the duty bearers because it's just reporting on the number of plans. 44 00:03:46,640 --> 00:03:53,320 And the problem here is the plan has to be of sufficient quality to engage with these 45 00:03:53,320 --> 00:03:54,600 duty bearers. 46 00:03:54,600 --> 00:03:58,400 Just reporting on the number of plans is not adequate. 47 00:03:58,400 --> 00:03:59,560 The plans could be garbage. 48 00:03:59,560 --> 00:04:01,100 We just don't know. 49 00:04:01,100 --> 00:04:05,640 And to be fair to the Equitas, maybe somewhere in the performance measurement framework, 50 00:04:05,640 --> 00:04:07,580 but I cannot find it. 51 00:04:07,580 --> 00:04:11,960 And I want to repeat to the listeners out there, if you want a copy of the performance 52 00:04:11,960 --> 00:04:17,680 measurement framework from Equitas, as well as our critique, feel free to email me at 53 00:04:17,680 --> 00:04:23,180 evaluatecanadaaid at gmail.com, and I would be happy to send it to you. 54 00:04:23,180 --> 00:04:27,400 And you're also welcome to ask for any of the other performance measurement frameworks 55 00:04:27,400 --> 00:04:31,520 and any of the other critiques of those performance measurement frameworks that we've covered 56 00:04:31,520 --> 00:04:34,260 in the podcast so far. 57 00:04:34,260 --> 00:04:35,300 So that's the issue here. 58 00:04:35,300 --> 00:04:41,080 The number of plans is not adequate as a measure of increased commitment. 59 00:04:41,080 --> 00:04:46,740 What they should do instead is measure the percentage of plans that achieve, quote, quality 60 00:04:46,740 --> 00:04:50,080 criteria to engage effectively. 61 00:04:50,080 --> 00:04:55,680 In fact, another indicator in the performance measurement framework that we'll discuss 62 00:04:55,680 --> 00:05:02,360 later actually talks about quality criteria, but they don't seem how to do it here. 63 00:05:02,360 --> 00:05:07,440 Also, a thing to note in the performance measurement framework for this indicator is they also 64 00:05:07,440 --> 00:05:13,040 interview the duty bearers, the ones who receive these women who are lobbying them to promote 65 00:05:13,040 --> 00:05:15,080 gender equality. 66 00:05:15,080 --> 00:05:20,660 And another recommendation here would be to ask them, not the women's organizations, 67 00:05:20,660 --> 00:05:26,940 but go to the duty bearers and ask them, do you think these women who are engaging with 68 00:05:26,940 --> 00:05:35,560 you on advancing dialogue on advancing gender equality, are they doing it adequately? 69 00:05:35,560 --> 00:05:39,920 Because the whole project's based on training them on making sure that they can effectively 70 00:05:39,920 --> 00:05:42,760 lobby these duty bearers. 71 00:05:42,760 --> 00:05:44,580 So that's the end of that one. 72 00:05:44,580 --> 00:05:49,760 So what I'm going to do now is read the next outcome statement and Yvonne's going to look 73 00:05:49,760 --> 00:05:51,420 at the next outcome indicator. 74 00:05:51,420 --> 00:05:59,360 So here the next outcome statement is, and I quote, increased capacity of select women's 75 00:05:59,360 --> 00:06:06,540 organizations to engage duty bearers in better fulfilling their human rights obligations 76 00:06:06,540 --> 00:06:10,580 to advance gender equality in target countries. 77 00:06:10,580 --> 00:06:14,760 And the outcome indicator that you're going to look at, Yvonne? 78 00:06:14,760 --> 00:06:21,220 That is the percentage of intermediaries, men and women, trained in transferring knowledge 79 00:06:21,220 --> 00:06:28,100 to select women's organizations on engaging duty bearers in fulfilling their human rights 80 00:06:28,100 --> 00:06:34,000 obligations to advance gender equality in target countries. 81 00:06:34,000 --> 00:06:38,340 That is the outcome indicator that has been specified. 82 00:06:38,340 --> 00:06:47,240 And this indicator is flawed in that it has the disadvantage of self-reporting bias. 83 00:06:47,240 --> 00:06:54,760 The women's organizations are expected to have increased technical capacity to engage 84 00:06:54,760 --> 00:07:02,640 or lobby the duty bearers based on the training already provided in the previous output. 85 00:07:02,640 --> 00:07:10,360 Hence, they should be tested on this technical ability, just knowing that the knowledge has 86 00:07:10,360 --> 00:07:17,480 been transferred is not really good enough for this particular outcome. 87 00:07:17,480 --> 00:07:18,480 That's right. 88 00:07:18,480 --> 00:07:24,200 And there's even a weird, strange intermediary of the women and men who are trained that 89 00:07:24,200 --> 00:07:29,400 then transfer the knowledge to select women's organizations who then engage with the duty 90 00:07:29,400 --> 00:07:30,520 bearers. 91 00:07:30,520 --> 00:07:32,280 So it's kind of strange. 92 00:07:32,280 --> 00:07:34,940 They're just looking at the percent who are able to transfer. 93 00:07:34,940 --> 00:07:38,200 Now again, at the performance measurement framework, maybe they should have a little 94 00:07:38,200 --> 00:07:42,740 sentence where they say, see the following measurement tool, but they don't. 95 00:07:42,740 --> 00:07:50,400 So we can only assume that they've missed the mark here on properly measuring increased 96 00:07:50,400 --> 00:07:51,400 capacity. 97 00:07:51,400 --> 00:07:52,400 Yeah. 98 00:07:52,400 --> 00:07:53,400 So. 99 00:07:53,400 --> 00:07:54,400 Absolutely. 100 00:07:54,400 --> 00:07:55,400 Yeah. 101 00:07:55,400 --> 00:08:05,640 And the next outcome is increased opportunities for civil society organizations to collaborate 102 00:08:05,640 --> 00:08:10,920 on advancing gender equality in target countries. 103 00:08:10,920 --> 00:08:14,240 And the indicator for this is interesting. 104 00:08:14,240 --> 00:08:21,640 It's quote, type of opportunities available for civil society organizations to collaborate 105 00:08:21,640 --> 00:08:25,520 on advancing gender equality in target countries. 106 00:08:25,520 --> 00:08:31,760 Now initially when you read that, you think, well, if civil society organizations have 107 00:08:31,760 --> 00:08:36,400 increased opportunities to collaborate, that's a good measure, I would think. 108 00:08:36,400 --> 00:08:43,240 You know, when you first read it, you think that's a good measure because they can expand 109 00:08:43,240 --> 00:08:50,640 their opportunities to collaborate, which will over time be a benefit for the country 110 00:08:50,640 --> 00:08:52,720 in advancing gender equality. 111 00:08:52,720 --> 00:08:54,240 Makes sense. 112 00:08:54,240 --> 00:09:01,080 The problem is it's not really defined and it should be connected to the technical capacity 113 00:09:01,080 --> 00:09:04,420 to engage and promote gender equality. 114 00:09:04,420 --> 00:09:06,480 So maybe they do have more opportunities. 115 00:09:06,480 --> 00:09:14,120 So yes, you could argue that's pretty good, but it should be connected to their ability 116 00:09:14,120 --> 00:09:16,940 technically to engage and promote gender equality. 117 00:09:16,940 --> 00:09:24,160 So what exactly will these civil society organizations do when they get the opportunity to engage 118 00:09:24,160 --> 00:09:26,240 to advance gender equality? 119 00:09:26,240 --> 00:09:31,880 They can have all these wonderful opportunities running around, collaborating with other civil 120 00:09:31,880 --> 00:09:36,480 society organizations, but don't forget they have to engage. 121 00:09:36,480 --> 00:09:38,460 They have to promote gender equality. 122 00:09:38,460 --> 00:09:45,520 So that's the problem there is on the surface, it looks good, but we need that type of opportunity 123 00:09:45,520 --> 00:09:52,980 to be clearly defined and assume that that type of opportunity is equated with better 124 00:09:52,980 --> 00:10:01,020 performance on promoting gender equality and then also making sure that that in turn is 125 00:10:01,020 --> 00:10:08,000 connected to a better technical ability to engage and promote gender equality. 126 00:10:08,000 --> 00:10:14,240 So I'm criticizing it, but it's still not a bad indicator to be honest, but it doesn't 127 00:10:14,240 --> 00:10:17,280 go far enough and they don't define it. 128 00:10:17,280 --> 00:10:18,540 Right. 129 00:10:18,540 --> 00:10:24,280 So now we're going to move on to the next outcome, which is, and I quote, increased 130 00:10:24,280 --> 00:10:32,640 capacity of women's and human rights organizations to carry out human rights education to advance 131 00:10:32,640 --> 00:10:34,240 gender equality. 132 00:10:34,240 --> 00:10:38,400 And you're going to be looking at the outcome indicator. 133 00:10:38,400 --> 00:10:45,560 Level of perceived capacity on a four point scale on women's and human rights organizations 134 00:10:45,560 --> 00:10:51,600 to carry out human rights education to advance gender equality. 135 00:10:51,600 --> 00:11:01,640 Now, a measuring capacity is not a bad idea, but level of perceived capacity of those you 136 00:11:01,640 --> 00:11:07,920 train is not a valid measure of technical capacity. 137 00:11:07,920 --> 00:11:16,600 If people have been trained on how to carry out human rights education to advance gender 138 00:11:16,600 --> 00:11:24,800 equality, what should be done to measure the particular outcome is to test them. 139 00:11:24,800 --> 00:11:30,920 They should undergo tests because the perceived measures would need a comparison group, which 140 00:11:30,920 --> 00:11:35,480 is not also provided for in this PMF. 141 00:11:35,480 --> 00:11:42,840 There is no comparison group and also measuring level of perceived capacity for this particular 142 00:11:42,840 --> 00:11:49,560 outcome would not technically tell us whether they did a good job or not in increasing the 143 00:11:49,560 --> 00:11:55,160 capacity of women's and human rights organizations to carry out this education, human rights 144 00:11:55,160 --> 00:11:57,640 education to advance gender equality. 145 00:11:57,640 --> 00:12:00,160 So we've got two points there. 146 00:12:00,160 --> 00:12:04,760 First of all, the perceived capacity, measuring perceived capacity is not a valid measure 147 00:12:04,760 --> 00:12:06,160 of technical capacity. 148 00:12:06,160 --> 00:12:11,280 And if they truly want to test, if a good job has been done in training these women's 149 00:12:11,280 --> 00:12:16,200 and human rights organizations, we need a comparison group. 150 00:12:16,200 --> 00:12:17,200 Thank you. 151 00:12:17,200 --> 00:12:18,200 Sure. 152 00:12:18,200 --> 00:12:19,200 Thank you, Yvonne. 153 00:12:19,200 --> 00:12:20,200 Yeah, that's right. 154 00:12:20,200 --> 00:12:23,720 Even if we say, okay, look, we'll let you do perceived capacity. 155 00:12:23,720 --> 00:12:29,480 You've got to show that the project has improved or increased that perceived capacity, even 156 00:12:29,480 --> 00:12:33,840 though it's self-reporting bias and they're all going to say, oh yeah, much better thanks 157 00:12:33,840 --> 00:12:35,460 to Equitas. 158 00:12:35,460 --> 00:12:40,360 But maybe there's another group of women and human rights organizations out there that 159 00:12:40,360 --> 00:12:45,720 also are experiencing increased capacity to carry out human rights education to advance 160 00:12:45,720 --> 00:12:47,800 gender equality in the country. 161 00:12:47,800 --> 00:12:50,040 And they've had nothing to do with the project. 162 00:12:50,040 --> 00:12:56,880 So yes, need a comparison group if you want to show that your project of $18 million actually 163 00:12:56,880 --> 00:13:03,040 was involved and responsible for improving, increasing that perceived capacity, even though 164 00:13:03,040 --> 00:13:09,800 we know it's not a valid measure of the ability to carry out human rights education. 165 00:13:09,800 --> 00:13:11,240 Right. 166 00:13:11,240 --> 00:13:19,720 So the next outcome is increased opportunities for women and or girls to participate in decision-making 167 00:13:19,720 --> 00:13:25,080 process within local structures in target countries. 168 00:13:25,080 --> 00:13:31,820 Here the outcome indicator measure is level of confidence on a four-point scale of women 169 00:13:31,820 --> 00:13:38,200 and or girls in their ability to participate in decision-making process within local structures. 170 00:13:38,200 --> 00:13:45,080 Again, we have a self-reporting bias in their ability to participate. 171 00:13:45,080 --> 00:13:47,420 That's one issue. 172 00:13:47,420 --> 00:13:53,760 And the other issue is maybe Equitas is going to say, look, we're not necessarily interested 173 00:13:53,760 --> 00:13:57,680 in just increasing technical ability to abdicate. 174 00:13:57,680 --> 00:14:03,560 We also want to increase the confidence levels of women or girls in their ability or just 175 00:14:03,560 --> 00:14:06,640 their confidence to speak up and participate. 176 00:14:06,640 --> 00:14:08,820 And I think that's a fair point. 177 00:14:08,820 --> 00:14:11,000 Problem is they don't have a comparison group. 178 00:14:11,000 --> 00:14:16,000 So they can't even show if their project increased perceived levels of confidence reported by 179 00:14:16,000 --> 00:14:17,960 these women or girls. 180 00:14:17,960 --> 00:14:20,080 So they can't have it both ways. 181 00:14:20,080 --> 00:14:27,480 If they don't want to, if they want to measure technically increased opportunities, for example, 182 00:14:27,480 --> 00:14:32,360 on a decision-making body, do they really have the power to make decisions, which they're 183 00:14:32,360 --> 00:14:33,760 not doing here? 184 00:14:33,760 --> 00:14:38,320 Or if they want to stick with level of confidence, then again, they have to show their project 185 00:14:38,320 --> 00:14:43,040 is doing a better job of increasing levels of confidence than women and girls in the 186 00:14:43,040 --> 00:14:46,320 country that are not in the project. 187 00:14:46,320 --> 00:14:54,760 So next outcome is increased capacity of intermediaries that we mentioned earlier, women and girls, 188 00:14:54,760 --> 00:15:02,580 to mobilize communities to advance gender equality in target countries. 189 00:15:02,580 --> 00:15:05,880 And you're going to be looking at an outcome there, Yvonne. 190 00:15:05,880 --> 00:15:11,900 Yeah, the outcome indicator we're going to be focusing on here is the level of confidence 191 00:15:11,900 --> 00:15:19,680 on a four-point scale, I believe, of intermediaries, men and women, in their ability to mobilize 192 00:15:19,680 --> 00:15:26,080 communities to advance gender equality in target countries. 193 00:15:26,080 --> 00:15:27,160 Beautiful. 194 00:15:27,160 --> 00:15:34,120 We understand that the effort, the resources, the trainings that have involved these women 195 00:15:34,120 --> 00:15:39,800 and men has been targeted at improving their capacity to mobilize their communities, you 196 00:15:39,800 --> 00:15:41,960 know, to advance gender equality. 197 00:15:41,960 --> 00:15:42,960 Beautiful. 198 00:15:42,960 --> 00:15:50,400 However, this particular indicator is suffering from self-reporting bias again. 199 00:15:50,400 --> 00:15:52,600 Here we're talking about perceived capacity. 200 00:15:52,600 --> 00:15:59,000 If you ask people, did your level of confidence increase after our training? 201 00:15:59,000 --> 00:16:03,480 They could say yes, but that's really subjective. 202 00:16:03,480 --> 00:16:04,920 That's really subjective. 203 00:16:04,920 --> 00:16:10,880 And sadly, that might not be a valid measure of technical ability. 204 00:16:10,880 --> 00:16:11,880 It might not be. 205 00:16:11,880 --> 00:16:12,880 Right. 206 00:16:12,880 --> 00:16:13,880 David? 207 00:16:13,880 --> 00:16:19,440 No, I was just going to add, if you read the last part of the outcome indicator in their 208 00:16:19,440 --> 00:16:25,280 ability to mobilize communities to advance gender equality in target countries, ability 209 00:16:25,280 --> 00:16:32,880 to mobilize communities, you could also, as an alternative, just look in the future to 210 00:16:32,880 --> 00:16:34,440 see how many communities they mobilize. 211 00:16:34,440 --> 00:16:39,840 And you could have some sort of crude definition of what do you mean by a community being mobilized? 212 00:16:39,840 --> 00:16:43,800 Maybe a couple of meetings, a criteria that they have to meet. 213 00:16:43,800 --> 00:16:47,560 That's another way you could measure whether they've actually done it. 214 00:16:47,560 --> 00:16:48,560 Exactly. 215 00:16:48,560 --> 00:16:52,000 Forget about asking them if they feel they have the ability. 216 00:16:52,000 --> 00:16:53,000 Go and find out. 217 00:16:53,000 --> 00:16:54,000 Yes, I agree with you, David. 218 00:16:54,000 --> 00:16:59,880 Find out how many communities they actually mobilized and then compare that to another 219 00:16:59,880 --> 00:17:04,960 area in the country where the project is not operating and see how many communities they 220 00:17:04,960 --> 00:17:12,200 mobilized, and that would be a way of showing that communities are actually being mobilized. 221 00:17:12,200 --> 00:17:15,720 But they haven't done that here. 222 00:17:15,720 --> 00:17:21,240 Another way of looking at this outcome indicator is to just look at whether these women and 223 00:17:21,240 --> 00:17:26,020 men actually mobilize these communities, and that would show that they do have the ability 224 00:17:26,020 --> 00:17:31,280 to mobilize, and there could be some minimum criteria, how many communities, and exactly 225 00:17:31,280 --> 00:17:38,040 what is a minimum criteria to reach to say, da, da, da, da, we've mobilized this community. 226 00:17:38,040 --> 00:17:42,600 And then they could compare that with a group of men and women in another part of the country 227 00:17:42,600 --> 00:17:48,640 that are not even involved in the project and how they have mobilized their communities. 228 00:17:48,640 --> 00:17:52,200 And you would expect in this project to be a higher level. 229 00:17:52,200 --> 00:17:57,920 Or if you don't want to compare, just look at the number of communities that have been 230 00:17:57,920 --> 00:18:03,880 mobilized over time for the target men and women in the project. 231 00:18:03,880 --> 00:18:08,880 That would be another way of showing that our project has indeed succeeded. 232 00:18:08,880 --> 00:18:13,440 Either way, but this asking how you feel is unacceptable. 233 00:18:13,440 --> 00:18:19,520 Yes, because that's exactly, that's so, so, so subjective. 234 00:18:19,520 --> 00:18:27,720 And David, I could have, my confidence level could be improved about something, but that 235 00:18:27,720 --> 00:18:32,560 does not actually translate to me actually doing it. 236 00:18:32,560 --> 00:18:33,600 Do you understand? 237 00:18:33,600 --> 00:18:39,700 Because there are other contextual and environmental and socio-economic factors that actually play 238 00:18:39,700 --> 00:18:45,360 into the actual decision to mobilize communities to advance gender equality. 239 00:18:45,360 --> 00:18:51,080 So it's not just about, you know, we've done the training, we've trained 500 people, 240 00:18:51,080 --> 00:18:55,600 we expect them to do this because your confidence levels have improved. 241 00:18:55,600 --> 00:18:58,200 I don't think it's such a linear equation. 242 00:18:58,200 --> 00:18:59,200 Yeah. 243 00:18:59,200 --> 00:19:00,600 That's a good point. 244 00:19:00,600 --> 00:19:06,760 And it raises an issue for those outcome harvesters out there, still trying to get one on the 245 00:19:06,760 --> 00:19:13,920 show to talk about the fact that in outcome harvesting, you could go to the end, see how 246 00:19:13,920 --> 00:19:20,080 many communities have been mobilized, and then work your way backwards to see if the 247 00:19:20,080 --> 00:19:25,880 mobilization of those communities was actually due to the project. 248 00:19:25,880 --> 00:19:26,880 Absolutely. 249 00:19:26,880 --> 00:19:27,880 And that's what I think. 250 00:19:27,880 --> 00:19:34,120 I'm not an expert on this, but outcome harvesting, I think when I read the description of that 251 00:19:34,120 --> 00:19:39,880 method, that's what they try to do because they argue, you know, the whole world is complex 252 00:19:39,880 --> 00:19:43,720 and you can't do a linear connection, which is true. 253 00:19:43,720 --> 00:19:46,980 But there's not even an attempt here, as you can see. 254 00:19:46,980 --> 00:19:52,080 And some of the other projects we've talked about on the podcast, they do use most significant 255 00:19:52,080 --> 00:19:57,680 change outcome harvesting, but so far I haven't got them on the show. 256 00:19:57,680 --> 00:19:59,160 So I'm quite excited. 257 00:19:59,160 --> 00:20:05,280 Hopefully someday we'll get somebody on the show that can explain it and actually show 258 00:20:05,280 --> 00:20:10,000 that in their performance measurement framework, we can show that our outputs are achieving 259 00:20:10,000 --> 00:20:11,000 the outcomes. 260 00:20:11,000 --> 00:20:13,400 But right now we're not there yet. 261 00:20:13,400 --> 00:20:15,120 Good point. 262 00:20:15,120 --> 00:20:18,080 Yeah, yeah, absolutely. 263 00:20:18,080 --> 00:20:19,080 Yeah. 264 00:20:19,080 --> 00:20:20,080 Go ahead. 265 00:20:20,080 --> 00:20:21,080 Yeah. 266 00:20:21,080 --> 00:20:30,800 The next one is increased efforts by select women's organizations to influence duty bearers 267 00:20:30,800 --> 00:20:36,240 in fulfilling their human rights obligations related to gender equality in target countries. 268 00:20:36,240 --> 00:20:39,320 It's the same outcome I think we read before. 269 00:20:39,320 --> 00:20:45,160 So I'm talking about it again here and the outcome indicator is type of efforts made 270 00:20:45,160 --> 00:20:52,720 by select women's organizations to influence duty bearers in fulfilling their human rights 271 00:20:52,720 --> 00:20:57,040 obligations related to gender equality in target countries. 272 00:20:57,040 --> 00:21:02,320 And again, it's like type of opportunities I talked about earlier, type of efforts. 273 00:21:02,320 --> 00:21:04,040 True. 274 00:21:04,040 --> 00:21:09,040 It's in the PMF, if you look at it, it says duty bearers who receive these efforts going 275 00:21:09,040 --> 00:21:12,360 from none up to frequent efforts is not good enough. 276 00:21:12,360 --> 00:21:14,080 That's what I point out here. 277 00:21:14,080 --> 00:21:17,520 Duty bearers are interviewed, which is good. 278 00:21:17,520 --> 00:21:24,560 They not the women organizations who report their frequency effort should be asked not 279 00:21:24,560 --> 00:21:29,120 how many times the women came and engaged with them, but something like this. 280 00:21:29,120 --> 00:21:36,880 Do you believe those efforts were a sufficient quantity and quality to advance gender equality? 281 00:21:36,880 --> 00:21:41,280 So they should get right at it, go right to the chase, ask the duty bearers, we've got 282 00:21:41,280 --> 00:21:47,240 these women coming at you from this project, cost to the taxpayer of 18 million Canadian 283 00:21:47,240 --> 00:21:48,240 dollars. 284 00:21:48,240 --> 00:21:50,720 Have they done their homework? 285 00:21:50,720 --> 00:21:53,320 Do they look like they're ready? 286 00:21:53,320 --> 00:21:56,680 Do they know what they're talking about when they're advancing gender equality, when they 287 00:21:56,680 --> 00:21:58,960 engage with you? 288 00:21:58,960 --> 00:22:01,320 Forget about how many times they show up. 289 00:22:01,320 --> 00:22:03,480 What's the quality like? 290 00:22:03,480 --> 00:22:08,800 The irony here is they interviewed the duty bearers in the performance measurement framework, 291 00:22:08,800 --> 00:22:10,880 but they asked them the wrong question. 292 00:22:10,880 --> 00:22:16,240 They should really probe, are you really doing a good job here? 293 00:22:16,240 --> 00:22:17,240 And they don't. 294 00:22:17,240 --> 00:22:25,920 Now, to be fair to them, maybe it's buried in their measurement tool somewhere, but I 295 00:22:25,920 --> 00:22:29,280 don't see it in the performance measurement framework. 296 00:22:29,280 --> 00:22:33,400 All we see is the scale, none up to frequent efforts. 297 00:22:33,400 --> 00:22:40,200 And that's clearly not a good measure of their ability to influence these duty bearers. 298 00:22:40,200 --> 00:22:42,240 Yes, yes. 299 00:22:42,240 --> 00:22:48,200 We're supposed to be checking for the effectiveness, their effectiveness in influencing the duty 300 00:22:48,200 --> 00:22:51,880 bearers, not necessarily the frequency. 301 00:22:51,880 --> 00:22:54,800 And the irony is they're getting there. 302 00:22:54,800 --> 00:22:57,520 They've done what we've recommended in earlier indicators. 303 00:22:57,520 --> 00:23:00,840 They've said, forget the women, we're going to go straight to the duty bearers because 304 00:23:00,840 --> 00:23:03,120 that's where the impact is supposed to happen. 305 00:23:03,120 --> 00:23:05,240 We're going to ask them how well they're doing. 306 00:23:05,240 --> 00:23:08,720 But in this question, they just asked them how many times they knocked on the door. 307 00:23:08,720 --> 00:23:13,400 So yeah, it's ironic because they're kind of in the right direction, but they're not 308 00:23:13,400 --> 00:23:16,640 asking the obvious question, you know? 309 00:23:16,640 --> 00:23:19,080 So they're getting there. 310 00:23:19,080 --> 00:23:20,080 Next outcome. 311 00:23:20,080 --> 00:23:25,080 I, sorry, going back to, sorry, David. 312 00:23:25,080 --> 00:23:31,880 That's why it would have really been interesting to have a representative from Equitas here 313 00:23:31,880 --> 00:23:38,920 because maybe they could have provided a bit more exposure to their rationale behind just 314 00:23:38,920 --> 00:23:44,960 stating things like this, because it's possible in your brainstorming sessions, they might 315 00:23:44,960 --> 00:23:49,600 have thought, oh, we can't do this, but we can do this. 316 00:23:49,600 --> 00:23:53,320 Maybe they've thought about what we've just said, but canceled it out. 317 00:23:53,320 --> 00:23:54,680 And then they just put this in. 318 00:23:54,680 --> 00:23:56,400 Do you understand? 319 00:23:56,400 --> 00:24:03,080 It would have been really exciting to hear from them to know why they stated this indicators 320 00:24:03,080 --> 00:24:04,360 as they are. 321 00:24:04,360 --> 00:24:05,680 Yeah, absolutely. 322 00:24:05,680 --> 00:24:07,240 But go ahead. 323 00:24:07,240 --> 00:24:08,960 Yes, I agree. 324 00:24:08,960 --> 00:24:12,200 And I think I've alluded it to in my trailer. 325 00:24:12,200 --> 00:24:17,920 Yeah, I think I have, which is one of the problems is Global Affairs Canada requires 326 00:24:17,920 --> 00:24:22,400 this performance measurement framework before they get any funding, but they're not following 327 00:24:22,400 --> 00:24:24,160 their own guidelines. 328 00:24:24,160 --> 00:24:29,480 They could say, look, these outcome indicators are not valid measures of your outcomes. 329 00:24:29,480 --> 00:24:31,200 What's going on here? 330 00:24:31,200 --> 00:24:36,200 And that's one thing that is an issue here. 331 00:24:36,200 --> 00:24:42,920 And they may have another document buried somewhere that explains why they went ahead 332 00:24:42,920 --> 00:24:44,120 with this outcome indicator. 333 00:24:44,120 --> 00:24:50,400 But based on the performance measurement framework, it's still an issue, but you're right. 334 00:24:50,400 --> 00:24:56,480 It would be good to have a representative on to provide more detail about why they pick 335 00:24:56,480 --> 00:24:58,720 these outcome indicators. 336 00:24:58,720 --> 00:25:01,760 And sometimes it's an afterthought just to get the funding. 337 00:25:01,760 --> 00:25:02,760 I don't know. 338 00:25:02,760 --> 00:25:07,320 That may be an unfair statement, but that's the way it's happening these days. 339 00:25:07,320 --> 00:25:13,940 So I think we're on enhanced collaboration again among women and human rights organizations 340 00:25:13,940 --> 00:25:20,520 to advance human rights in their countries, and you have an outcome indicator for there. 341 00:25:20,520 --> 00:25:28,000 Yes, and the indicator in this case is the degree of collaboration on a four-point scale 342 00:25:28,000 --> 00:25:37,200 among women and human rights organizations to advance human rights in their countries. 343 00:25:37,200 --> 00:25:38,680 Beautiful. 344 00:25:38,680 --> 00:25:49,520 However, this still points towards a self-reporting bias because they need to measure externally 345 00:25:49,520 --> 00:25:55,980 and not ask the organization to report the frequency of their collaborations. 346 00:25:55,980 --> 00:26:01,240 We need to also measure more than just baseline and endline. 347 00:26:01,240 --> 00:26:04,920 So I feel that this is a good indicator. 348 00:26:04,920 --> 00:26:11,520 I don't think it's bad in and of itself, but however, the measurements for this indicator 349 00:26:11,520 --> 00:26:16,160 should have a broader scope than what they presented. 350 00:26:16,160 --> 00:26:23,080 Yes, and most important, well, I think importantly, again, they shouldn't be asking the women's 351 00:26:23,080 --> 00:26:30,540 and human rights organizations to report on their own efforts on how well they are collaborating. 352 00:26:30,540 --> 00:26:32,200 It's just a huge bias. 353 00:26:32,200 --> 00:26:40,480 They should have an external third-party monitoring going in and going in and using their own 354 00:26:40,480 --> 00:26:46,960 criteria figuring out if over time, here we've got just a baseline and an endline, are they 355 00:26:46,960 --> 00:26:53,400 actually increasing their degree of collaboration between themselves because it is an important 356 00:26:53,400 --> 00:27:02,000 activity, for lack of a better word, or outcome that you want to increase collaboration to 357 00:27:02,000 --> 00:27:09,040 get this critical mass so that you can advance human rights in the country. 358 00:27:09,040 --> 00:27:10,960 But don't ask them. 359 00:27:10,960 --> 00:27:17,120 Go out and do it independently and show that the project is actually in some way through 360 00:27:17,120 --> 00:27:20,640 networking or whatever, because there's a whole list. 361 00:27:20,640 --> 00:27:26,680 If you read part one of this podcast, describes all the, there's 10 target groups, all the 362 00:27:26,680 --> 00:27:32,960 activities that they're doing is in terms of outputs to achieve these outcomes. 363 00:27:32,960 --> 00:27:38,400 So yes, it has to be an external measure because it just defeats the whole purpose if they 364 00:27:38,400 --> 00:27:39,400 just ask themselves. 365 00:27:39,400 --> 00:27:40,400 Yes. 366 00:27:40,400 --> 00:27:41,400 Right. 367 00:27:41,400 --> 00:27:45,200 Yes, I agree with you. 368 00:27:45,200 --> 00:27:46,680 Okay. 369 00:27:46,680 --> 00:27:54,720 So the next one is increased leadership of women and men, community mobilization activities 370 00:27:54,720 --> 00:27:58,240 that advance gender equality in target countries. 371 00:27:58,240 --> 00:28:04,600 And the outcome indicator for that outcome is, quote, percent of total women and men 372 00:28:04,600 --> 00:28:10,980 trained demonstrating leadership in community mobilization activities that advance gender 373 00:28:10,980 --> 00:28:14,600 equality in target countries. 374 00:28:14,600 --> 00:28:16,160 This is just attendance. 375 00:28:16,160 --> 00:28:18,120 Everybody goes home. 376 00:28:18,120 --> 00:28:21,020 They say trained demonstrating leadership. 377 00:28:21,020 --> 00:28:25,840 So that implies that during the training, they demonstrated leadership. 378 00:28:25,840 --> 00:28:29,120 So how do they measure leadership skill? 379 00:28:29,120 --> 00:28:31,500 It's not clear in the performance measurement framework. 380 00:28:31,500 --> 00:28:33,720 So it looks like it's just attendance. 381 00:28:33,720 --> 00:28:37,820 I don't see in the performance measurement framework, again, the following measurement 382 00:28:37,820 --> 00:28:45,200 tool was measured, used to show that X percent actually showed leadership after the training 383 00:28:45,200 --> 00:28:49,080 that they didn't have before in terms of leadership skill, for example. 384 00:28:49,080 --> 00:28:50,080 Yeah. 385 00:28:50,080 --> 00:28:53,040 I don't see that in the performance measurement framework. 386 00:28:53,040 --> 00:28:58,780 Even if we did, and I missed it, they're not using a comparison group. 387 00:28:58,780 --> 00:29:05,440 So we can't show that the training was directly responsible for an increase in leadership 388 00:29:05,440 --> 00:29:07,980 skills. 389 00:29:07,980 --> 00:29:09,260 So that's the problem there. 390 00:29:09,260 --> 00:29:12,480 They only measure baseline and end line. 391 00:29:12,480 --> 00:29:17,960 That suggests only at the beginning of the project and at the end of the project, which 392 00:29:17,960 --> 00:29:23,280 means in the middle, they could easily, even if they have a proper measuring instrument, 393 00:29:23,280 --> 00:29:29,340 they could easily teach themselves these leadership skills that have nothing to do with the training 394 00:29:29,340 --> 00:29:31,840 on leadership. 395 00:29:31,840 --> 00:29:33,960 See what I'm getting at? 396 00:29:33,960 --> 00:29:34,960 Okay. 397 00:29:34,960 --> 00:29:35,960 Yeah. 398 00:29:35,960 --> 00:29:43,320 So even if they have a measurement tool, I can't find it in the PMF, but even if they 399 00:29:43,320 --> 00:29:48,920 have a measurement tool that measures their level of demonstrating leadership and it's 400 00:29:48,920 --> 00:29:54,400 gone up and the percents are going up, they only do it twice, once at the beginning of 401 00:29:54,400 --> 00:29:58,120 the project and once at the end, which it's a four-year project. 402 00:29:58,120 --> 00:30:04,280 So there's just no way they can claim that the training has increased the leadership 403 00:30:04,280 --> 00:30:08,760 skill levels, even if they claim the percent of total women and men trained demonstrating 404 00:30:08,760 --> 00:30:15,200 leadership has gone up from the baseline to the end line, right? 405 00:30:15,200 --> 00:30:17,440 Just two measures is not good enough. 406 00:30:17,440 --> 00:30:24,040 You have to measure frequently along the scale to show that the project training is responsible. 407 00:30:24,040 --> 00:30:29,120 Like I say in my trailer, after the first training, it could go down the road and figure 408 00:30:29,120 --> 00:30:33,040 out how to do this leadership skills on their own with somebody else. 409 00:30:33,040 --> 00:30:37,480 And then they come back four years later, hello, percent goes up. 410 00:30:37,480 --> 00:30:38,480 Yeah. 411 00:30:38,480 --> 00:30:40,720 They have nothing to do with the project training. 412 00:30:40,720 --> 00:30:49,920 Yeah, but David, in a little bit to their defense, in the PMF, they specified for specific 413 00:30:49,920 --> 00:30:55,800 countries and now they're taking the example of Tanzania, they said they listed four specific 414 00:30:55,800 --> 00:31:02,520 activities, awareness raising campaigns, capacity building activities, engagement with duty 415 00:31:02,520 --> 00:31:08,920 bearers, support to service providers working with victims of gender-based violence. 416 00:31:08,920 --> 00:31:17,320 So I guess maybe they're trying to check amongst this cohort of people, of men and women that 417 00:31:17,320 --> 00:31:24,680 we trained at this specific point in time, how many of them from each cohort led awareness 418 00:31:24,680 --> 00:31:33,520 raising campaign programs for Veterans warmth day leader and community. 419 00:31:33,520 --> 00:31:48,560 That's a little off the top of my head.