1 00:00:00,000 --> 00:00:04,700 Welcome to the Improving Development Evaluation Podcast. 2 00:00:04,700 --> 00:00:08,100 I'm your host David Wand and welcome to episode six, 3 00:00:08,100 --> 00:00:11,380 part two, where we continue our discussion 4 00:00:11,380 --> 00:00:14,700 on colleges and institutes Canada 5 00:00:14,700 --> 00:00:17,060 and their $18 million project funded 6 00:00:17,060 --> 00:00:22,060 by the Canadian taxpayer that is operating in Senegal 7 00:00:22,280 --> 00:00:25,620 and is entitled, A Thousand Women, I Am Woman, 8 00:00:25,620 --> 00:00:27,360 I Exist, I Participate. 9 00:00:27,360 --> 00:00:31,480 And just to give you a brief recap, 10 00:00:31,480 --> 00:00:34,200 I contacted the colleges and institutes Canada 11 00:00:34,200 --> 00:00:37,600 on March 9th to invite them to this podcast 12 00:00:37,600 --> 00:00:39,260 to respond to our critique 13 00:00:39,260 --> 00:00:42,120 of their performance measurement framework. 14 00:00:42,120 --> 00:00:43,360 They did not reply. 15 00:00:43,360 --> 00:00:45,120 I then followed up on March 18th, 16 00:00:45,120 --> 00:00:48,120 along with a phone call and left a message. 17 00:00:48,120 --> 00:00:49,800 We still have not received a reply. 18 00:00:49,800 --> 00:00:52,840 Today is now March 28th 19 00:00:52,840 --> 00:00:55,520 and they have decided obviously 20 00:00:55,520 --> 00:00:59,040 not to attend this part two of the podcast. 21 00:00:59,040 --> 00:01:02,300 But fortunately, I do have with me an evaluation expert 22 00:01:02,300 --> 00:01:04,320 that I will introduce to you shortly. 23 00:01:04,320 --> 00:01:06,340 This $18 million project, 24 00:01:06,340 --> 00:01:08,880 looking at the performance measurement framework, 25 00:01:08,880 --> 00:01:13,880 had nine outcomes and used 17 outcome indicators 26 00:01:14,820 --> 00:01:17,580 to measure whether or not those outcomes 27 00:01:17,580 --> 00:01:19,800 had been achieved for the project. 28 00:01:19,800 --> 00:01:22,720 And I led to the conclusion that colleges 29 00:01:22,720 --> 00:01:25,280 and institutes Canada could not make the claim 30 00:01:25,280 --> 00:01:27,920 that its project was achieving its outcomes 31 00:01:27,920 --> 00:01:31,000 simply because the outcome indicators 32 00:01:31,000 --> 00:01:34,480 were not properly measuring the achievement of the outcomes 33 00:01:34,480 --> 00:01:38,160 with only one out of the 17 outcome indicators 34 00:01:38,160 --> 00:01:41,680 properly measuring whether the outcome was achieved. 35 00:01:41,680 --> 00:01:43,360 So with that being said, 36 00:01:43,360 --> 00:01:48,360 I'd like to introduce to you Dr. Jenny Yau Jorgensen, 37 00:01:48,960 --> 00:01:53,360 holds a PhD in system safety from Lund University 38 00:01:53,360 --> 00:01:57,200 and she is in Sweden and has been kind enough 39 00:01:57,200 --> 00:02:00,640 to participate in part two of this podcast. 40 00:02:00,640 --> 00:02:03,600 And she is also an evaluation practitioner. 41 00:02:03,600 --> 00:02:06,440 Welcome to the podcast, Jenny. 42 00:02:06,440 --> 00:02:08,800 Thank you for having me, David. 43 00:02:08,800 --> 00:02:09,840 Good, great. 44 00:02:09,840 --> 00:02:11,400 So what we're going to do 45 00:02:11,400 --> 00:02:14,280 is we're gonna go through each of the nine outcomes 46 00:02:14,280 --> 00:02:16,440 and select some indicators. 47 00:02:16,440 --> 00:02:18,160 And we're going to in detail, 48 00:02:18,160 --> 00:02:20,680 describe why they are flawed 49 00:02:20,680 --> 00:02:22,840 in properly measuring the outcome. 50 00:02:22,840 --> 00:02:25,320 And we will start with a few outcomes 51 00:02:25,320 --> 00:02:27,320 that are actually in the performance measurement 52 00:02:27,320 --> 00:02:28,560 framework as outputs, 53 00:02:28,560 --> 00:02:29,920 but we put them as outcomes 54 00:02:29,920 --> 00:02:34,120 because they expect outcomes to be achieved. 55 00:02:34,120 --> 00:02:36,920 So starting with the first one, 56 00:02:38,240 --> 00:02:41,480 the outcome was quote leadership training 57 00:02:41,480 --> 00:02:44,200 led by male leaders in the communities. 58 00:02:45,120 --> 00:02:47,880 And you would expect something to be learned 59 00:02:47,880 --> 00:02:49,240 from that training. 60 00:02:49,240 --> 00:02:52,480 And the outcome indicator that they stated was 61 00:02:52,480 --> 00:02:53,320 and I quote, 62 00:02:53,320 --> 00:02:56,920 number of people benefiting from leadership training 63 00:02:56,920 --> 00:02:58,640 by male leaders. 64 00:02:58,640 --> 00:03:02,160 And the problem with that indicator 65 00:03:02,160 --> 00:03:04,800 is that it's just assuming that people show up, 66 00:03:04,800 --> 00:03:07,320 they take attendance, number of people, 67 00:03:07,320 --> 00:03:09,760 but we don't know if they actually benefited 68 00:03:09,760 --> 00:03:11,360 from the leadership training. 69 00:03:11,360 --> 00:03:12,960 It's just taking attendance. 70 00:03:12,960 --> 00:03:15,440 And we would prefer obviously, 71 00:03:15,440 --> 00:03:17,120 that they would go a step further 72 00:03:17,120 --> 00:03:19,080 and actually measure whether or not 73 00:03:19,080 --> 00:03:22,280 they learned the content of the training 74 00:03:22,280 --> 00:03:24,360 with respect to leadership. 75 00:03:24,360 --> 00:03:27,960 So moving on, I will now state the second outcome 76 00:03:27,960 --> 00:03:32,240 and then Jenny, you can state the outcome indicator 77 00:03:32,240 --> 00:03:34,760 and make your comments on why you believe 78 00:03:34,760 --> 00:03:37,480 it may or may not be properly measuring the outcome. 79 00:03:37,480 --> 00:03:39,280 The outcome is and I quote, 80 00:03:39,280 --> 00:03:42,600 strengthening workshops on leadership 81 00:03:42,600 --> 00:03:45,520 organized with women and men leaders. 82 00:03:45,520 --> 00:03:46,920 And what would be the outcome indicator 83 00:03:46,920 --> 00:03:47,920 that you're looking at? 84 00:03:47,920 --> 00:03:50,800 Right, then the indicator that we analyze 85 00:03:50,800 --> 00:03:52,920 is the number of men and women leaders 86 00:03:52,920 --> 00:03:55,040 strengthened in leadership. 87 00:03:55,040 --> 00:03:58,760 So the issue that's sort of associated with this indicator 88 00:03:58,760 --> 00:04:00,840 can be looked at from different perspective. 89 00:04:00,840 --> 00:04:03,120 One is that strengthening, 90 00:04:03,120 --> 00:04:05,520 strengthening leadership is such a broad term. 91 00:04:05,520 --> 00:04:08,560 So that means the indicator itself would require 92 00:04:08,560 --> 00:04:11,480 different kinds of qualitative indicators. 93 00:04:11,480 --> 00:04:14,400 At the moment, the focus is mainly on the quantitative, 94 00:04:14,400 --> 00:04:18,360 the number of leaders being strengthened in leadership. 95 00:04:18,360 --> 00:04:19,960 So that's one aspect we thought 96 00:04:19,960 --> 00:04:22,120 the indicator could be enhanced. 97 00:04:23,440 --> 00:04:24,440 That's great. 98 00:04:24,440 --> 00:04:25,760 Okay, thank you, Jenny. 99 00:04:25,760 --> 00:04:27,840 And we'll move on to the next outcome. 100 00:04:29,080 --> 00:04:32,440 Consultation mechanism with women's groups, 101 00:04:32,440 --> 00:04:35,840 economic interest groups and organizations. 102 00:04:35,840 --> 00:04:38,320 And the outcome indicator that you're looking at? 103 00:04:38,320 --> 00:04:40,560 Is the number of consultation mechanisms 104 00:04:40,560 --> 00:04:42,360 established and functional. 105 00:04:42,360 --> 00:04:47,360 So again, the issues with the functionality, 106 00:04:47,360 --> 00:04:49,520 which also require indicators 107 00:04:49,520 --> 00:04:52,000 that goes beyond the quantitative. 108 00:04:52,000 --> 00:04:53,720 What kind of mechanism? 109 00:04:53,720 --> 00:04:56,560 Is it about addressing the effectiveness 110 00:04:56,560 --> 00:05:00,640 and different qualitative sort of matrix, for example? 111 00:05:00,640 --> 00:05:03,240 Is it really like regularity of meetings? 112 00:05:03,240 --> 00:05:04,320 Are we talking about that? 113 00:05:04,320 --> 00:05:07,200 Or is it talking about decision-making processes? 114 00:05:07,200 --> 00:05:09,120 And what are the follow-up actions? 115 00:05:09,120 --> 00:05:12,200 So these are sort of missing, missing matrix 116 00:05:12,200 --> 00:05:15,360 in this over quantitative indicator. 117 00:05:15,360 --> 00:05:18,040 Right, and I'm doing a similar one, 118 00:05:18,040 --> 00:05:21,960 the mechanism for consultation between government partners 119 00:05:21,960 --> 00:05:24,840 put in place integrating gender. 120 00:05:24,840 --> 00:05:26,280 And the outcome indicator there 121 00:05:26,280 --> 00:05:28,400 is number of consultation mechanisms 122 00:05:28,400 --> 00:05:31,000 established and functional. 123 00:05:31,000 --> 00:05:33,680 And the key thing I would add to what you've said is, 124 00:05:33,680 --> 00:05:37,120 they've written right in the outcome indicator functional. 125 00:05:37,120 --> 00:05:39,320 So you would think they would say, 126 00:05:39,320 --> 00:05:42,160 okay, let's get a tool, measurement tool, 127 00:05:42,160 --> 00:05:45,480 as how do we define and measure 128 00:05:45,480 --> 00:05:49,160 that that consultation mechanism is actually functional? 129 00:05:49,160 --> 00:05:50,000 And they don't. 130 00:05:50,000 --> 00:05:53,000 All they do is count the number of mechanisms 131 00:05:53,000 --> 00:05:56,240 they've produced on a piece of paper and maybe used, 132 00:05:56,240 --> 00:05:57,840 but they haven't gone the next step, 133 00:05:57,840 --> 00:06:00,640 which is what's the expected outcome of that mechanism? 134 00:06:00,640 --> 00:06:02,000 It is, it's functional. 135 00:06:02,000 --> 00:06:04,200 So what do you mean by it's working? 136 00:06:04,200 --> 00:06:05,240 It's functional. 137 00:06:05,240 --> 00:06:06,840 How are you gonna measure that? 138 00:06:06,840 --> 00:06:11,840 And that's missing in this performance measurement framework. 139 00:06:11,840 --> 00:06:13,880 So they can't claim it's functional 140 00:06:13,880 --> 00:06:15,800 because they're not even measuring it. 141 00:06:15,800 --> 00:06:16,640 Exactly. 142 00:06:16,640 --> 00:06:17,800 Maybe just to add a little bit, 143 00:06:17,800 --> 00:06:20,480 going back to my sort of similar indicator, 144 00:06:20,480 --> 00:06:22,760 because there's actually, in the outcome statement, 145 00:06:22,760 --> 00:06:25,200 is looking into the economic interest groups 146 00:06:25,200 --> 00:06:26,680 and organizations. 147 00:06:26,680 --> 00:06:28,800 So in terms of measuring the functionality, 148 00:06:28,800 --> 00:06:31,200 I think another aspect kind of missing 149 00:06:31,200 --> 00:06:34,000 is about the inclusivity and accessibility 150 00:06:34,000 --> 00:06:36,000 by underrepresented groups. 151 00:06:36,000 --> 00:06:39,200 So that is quite missing out in the indicators, 152 00:06:39,200 --> 00:06:41,400 the way it's being formulated at the moment. 153 00:06:41,400 --> 00:06:42,240 Right. 154 00:06:42,240 --> 00:06:44,200 And as we get further down, 155 00:06:44,200 --> 00:06:48,120 we'll return to your point about decision-making, 156 00:06:48,120 --> 00:06:51,400 because in particular with women in the project, 157 00:06:51,400 --> 00:06:55,800 and I should elaborate and recap that in part one, 158 00:06:55,800 --> 00:06:58,840 major part of these services of the project 159 00:06:58,840 --> 00:07:01,320 is for the vulnerable women to receive training 160 00:07:01,320 --> 00:07:03,000 in entrepreneurship and leadership, 161 00:07:03,000 --> 00:07:05,760 to the point where they have more power 162 00:07:05,760 --> 00:07:08,600 in decision-making bodies and they have more money 163 00:07:08,600 --> 00:07:10,800 from learning how to run a business. 164 00:07:10,800 --> 00:07:14,120 And that's basically what they call the empowerment. 165 00:07:14,120 --> 00:07:15,320 So we'll get to that. 166 00:07:15,320 --> 00:07:17,800 So that's a good point about the decision-making 167 00:07:17,800 --> 00:07:19,160 that you raised. 168 00:07:19,160 --> 00:07:24,160 So the next outcome is increased community awareness 169 00:07:24,160 --> 00:07:27,120 of the importance of women's participation 170 00:07:27,120 --> 00:07:28,720 in decision-making, 171 00:07:28,720 --> 00:07:31,560 and the outcome indicator you'll be looking at 172 00:07:31,560 --> 00:07:33,000 to measure that outcome? 173 00:07:33,000 --> 00:07:35,600 Yes, it's the number of people in communities 174 00:07:35,600 --> 00:07:37,960 directly sensitized to the importance 175 00:07:37,960 --> 00:07:40,720 of women's participation in decision-making. 176 00:07:40,720 --> 00:07:44,360 Desegregated by sex and occupation 177 00:07:44,360 --> 00:07:46,760 in practice leaders or not. 178 00:07:46,760 --> 00:07:51,280 So the issue that we find about the indicators 179 00:07:51,280 --> 00:07:53,840 that while the indicators track the reach, 180 00:07:53,840 --> 00:07:56,960 the outreach of the sensitization efforts, 181 00:07:56,960 --> 00:08:01,320 what is missing in terms is kind of related to the quality 182 00:08:01,320 --> 00:08:05,160 or the depth of the impact, for example, 183 00:08:05,160 --> 00:08:06,920 the attitudes, whether attitudes change 184 00:08:06,920 --> 00:08:08,640 after the sensitization. 185 00:08:08,640 --> 00:08:11,240 So that sort of aspect were not really captured 186 00:08:11,240 --> 00:08:12,560 in the indicator. 187 00:08:12,560 --> 00:08:16,240 Another bit is about, you know, in terms of who actually came 188 00:08:16,240 --> 00:08:19,680 forward to this, who are being reached out 189 00:08:19,680 --> 00:08:21,240 in terms of these activities, 190 00:08:21,240 --> 00:08:23,480 those were also not quite captured. 191 00:08:23,480 --> 00:08:26,840 So some other qualitative measures might be needed. 192 00:08:26,840 --> 00:08:31,640 Right, and I would add to that also the fact that they do this 193 00:08:31,640 --> 00:08:33,760 from the performance measurement framework. 194 00:08:33,760 --> 00:08:37,240 You can see it's an annual survey with targets 195 00:08:37,240 --> 00:08:39,160 to reach levels of awareness, 196 00:08:39,160 --> 00:08:41,480 but how do we know these targets were reached 197 00:08:41,480 --> 00:08:44,520 due to the project services of raising awareness? 198 00:08:44,520 --> 00:08:47,800 We don't because they only measure it once 199 00:08:47,800 --> 00:08:50,280 and they didn't use a comparison group, 200 00:08:50,280 --> 00:08:53,640 even though interestingly enough later on in this podcast, 201 00:08:53,640 --> 00:08:55,960 they actually do use a comparison group. 202 00:08:55,960 --> 00:08:58,920 So it begs the question, why did they not use 203 00:08:58,920 --> 00:09:04,400 a comparison group here to see if communities outside 204 00:09:04,400 --> 00:09:08,200 of where the project is operating also got sensitized, 205 00:09:08,200 --> 00:09:09,800 but at a lower level? 206 00:09:09,800 --> 00:09:11,800 And so that's not being done here. 207 00:09:11,800 --> 00:09:14,680 And also, did they ask the people surveyed 208 00:09:14,680 --> 00:09:17,000 if they knew about the project and if no, 209 00:09:17,000 --> 00:09:19,200 use them as a comparison group? 210 00:09:19,200 --> 00:09:23,680 So that was a key thing I would add to what you've said, yeah. 211 00:09:23,680 --> 00:09:26,360 Yeah, and then we have quite a number we will get back 212 00:09:26,360 --> 00:09:28,120 when it comes to other indicators, 213 00:09:28,120 --> 00:09:29,800 it's about the desegregation. 214 00:09:29,800 --> 00:09:32,720 I think there's more intersectionality aspects 215 00:09:32,720 --> 00:09:34,800 we can look at, but we will come back 216 00:09:34,800 --> 00:09:37,200 to the desegregation issue. 217 00:09:37,200 --> 00:09:38,600 Sure, yeah. 218 00:09:38,600 --> 00:09:42,560 The next outcome is increased capacity of men and women 219 00:09:42,560 --> 00:09:46,120 leaders to intervene in communities. 220 00:09:46,120 --> 00:09:47,320 That's the outcome. 221 00:09:47,320 --> 00:09:51,360 The outcome indicator that I'll be looking at is, quote, 222 00:09:51,360 --> 00:09:55,520 perception of community members on the effectiveness 223 00:09:55,520 --> 00:09:58,520 of the promotion of gender equality 224 00:09:58,520 --> 00:10:01,760 by the project's gender equality chance 225 00:10:01,760 --> 00:10:04,760 and gender equality champions. 226 00:10:04,760 --> 00:10:09,360 Now, the issue here is it's measured only once per year 227 00:10:09,360 --> 00:10:11,160 through focus groups. 228 00:10:11,160 --> 00:10:13,080 What about other communities that did not 229 00:10:13,080 --> 00:10:14,840 receive the leadership training? 230 00:10:14,840 --> 00:10:17,960 How are those women doing on leadership there? 231 00:10:17,960 --> 00:10:22,560 So we have this issue of capacity of men and women 232 00:10:22,560 --> 00:10:24,680 leaders to intervene, but they could 233 00:10:24,680 --> 00:10:29,200 be intervening in areas where the project is not operating. 234 00:10:29,200 --> 00:10:32,440 And so again, it's only measured once a year, 235 00:10:32,440 --> 00:10:34,880 and it could be very easy that they could have figured this out 236 00:10:34,880 --> 00:10:39,360 all on their own without the project services teaching them 237 00:10:39,360 --> 00:10:42,000 how to intervene in these communities. 238 00:10:42,000 --> 00:10:44,840 And even the perception of community members 239 00:10:44,840 --> 00:10:47,760 on the effectiveness of promotion of gender equality 240 00:10:47,760 --> 00:10:49,640 could be going up in these other communities 241 00:10:49,640 --> 00:10:51,100 where the project's not operating. 242 00:10:51,100 --> 00:10:54,360 So it's the same problem of where they're not measuring. 243 00:10:54,360 --> 00:10:57,360 If they don't want to use a comparison group, that's fine. 244 00:10:57,360 --> 00:10:59,920 But the other issue is if they're only measuring once 245 00:10:59,920 --> 00:11:01,160 a year, that's not enough. 246 00:11:01,160 --> 00:11:05,280 They need to measure frequently before they start the project, 247 00:11:05,280 --> 00:11:07,240 during, during, during the project, 248 00:11:07,240 --> 00:11:11,200 and after to show that there's a trend where the sensitization 249 00:11:11,200 --> 00:11:12,520 is going up. 250 00:11:12,520 --> 00:11:15,680 Yeah, otherwise there would be causality issue, right? 251 00:11:15,680 --> 00:11:18,480 How do we claim that is the contribution of the project 252 00:11:18,480 --> 00:11:19,560 intervention? 253 00:11:19,560 --> 00:11:21,940 Right, and you don't necessarily need a comparison group. 254 00:11:21,940 --> 00:11:24,800 You can just do proper quasi-experimental 255 00:11:24,800 --> 00:11:28,520 and measure several times rather than just once. 256 00:11:28,520 --> 00:11:29,400 You're exactly right. 257 00:11:29,400 --> 00:11:30,960 You can't make these claims. 258 00:11:30,960 --> 00:11:32,640 You can at least try to improve on it, 259 00:11:32,640 --> 00:11:36,400 that the perception levels are moving in the right direction. 260 00:11:36,400 --> 00:11:38,160 But if you want to claim that, then you've 261 00:11:38,160 --> 00:11:41,000 got to measure more frequently, right? 262 00:11:41,000 --> 00:11:43,920 Yeah, over time, tracking the over time, yeah. 263 00:11:43,920 --> 00:11:44,800 Exactly. 264 00:11:44,800 --> 00:11:45,920 Thank you. 265 00:11:45,920 --> 00:11:49,520 The next outcome is increased influence 266 00:11:49,520 --> 00:11:54,000 of women as active citizens in their communities. 267 00:11:54,000 --> 00:11:56,800 And you're looking at an outcome indicator for that outcome. 268 00:11:56,800 --> 00:11:58,960 Yeah, we have a couple of indicators here. 269 00:11:58,960 --> 00:12:01,080 So the first one is the percentage 270 00:12:01,080 --> 00:12:03,800 of women targeted by the project who 271 00:12:03,800 --> 00:12:07,960 participate in decision-making bodies in their community. 272 00:12:07,960 --> 00:12:13,040 So this indicator measures the participation rate of women, 273 00:12:13,040 --> 00:12:15,800 again, quantitative in decision-making bodies, 274 00:12:15,800 --> 00:12:16,760 which is a good start. 275 00:12:16,760 --> 00:12:19,600 But it really doesn't reflect or capture 276 00:12:19,600 --> 00:12:22,400 the level of influence on the effectiveness 277 00:12:22,400 --> 00:12:24,080 of their participation. 278 00:12:24,080 --> 00:12:27,000 So again, this participation does not always 279 00:12:27,000 --> 00:12:30,000 equate to having an active role or voices. 280 00:12:30,000 --> 00:12:31,880 We have seen even from our experiences, 281 00:12:31,880 --> 00:12:36,520 sometimes the typical gender role, taking notes, 282 00:12:36,520 --> 00:12:40,240 are happening in many of these decision-making bodies. 283 00:12:40,240 --> 00:12:42,360 So it's important to look at what kind of roles, what kind 284 00:12:42,360 --> 00:12:43,800 of influence do they have. 285 00:12:43,800 --> 00:12:45,840 Can they set the agenda, for example? 286 00:12:45,840 --> 00:12:47,320 And does it really make a difference 287 00:12:47,320 --> 00:12:50,280 if those decisions are being initiated by women? 288 00:12:50,280 --> 00:12:53,520 So these are some missing aspects. 289 00:12:53,520 --> 00:12:55,880 Yes, and I would add to that, again, it's 290 00:12:55,880 --> 00:12:57,960 only measured once a year. 291 00:12:57,960 --> 00:13:02,000 But it may not be a problem if it's only once a year, 292 00:13:02,000 --> 00:13:04,440 as long as they look at the decision-making bodies, 293 00:13:04,440 --> 00:13:07,160 like you said, and see what exactly do they 294 00:13:07,160 --> 00:13:09,560 have any power in making decisions. 295 00:13:09,560 --> 00:13:14,320 And has their representation as women gone up over time? 296 00:13:14,320 --> 00:13:16,080 And we just don't know. 297 00:13:16,080 --> 00:13:18,280 We just know the percent of women targeted, 298 00:13:18,280 --> 00:13:21,080 but they don't go into detail about. 299 00:13:21,080 --> 00:13:25,920 It's another issue of defining and measuring actual influence. 300 00:13:25,920 --> 00:13:29,760 And just being a member of the decision-making body 301 00:13:29,760 --> 00:13:31,200 is not enough, is what you point to. 302 00:13:31,200 --> 00:13:33,280 Absolutely, yeah. 303 00:13:33,280 --> 00:13:35,600 So we'll come back to some of the other two indicators 304 00:13:35,600 --> 00:13:36,760 later on. 305 00:13:36,760 --> 00:13:39,280 Right. 306 00:13:39,280 --> 00:13:43,000 And I think we're going to the next one, which 307 00:13:43,000 --> 00:13:46,960 is increased inclusion of women as actors 308 00:13:46,960 --> 00:13:49,680 in social and economic development. 309 00:13:49,680 --> 00:13:52,640 And you have an indicator there that you want to look at. 310 00:13:52,640 --> 00:13:53,600 Yes. 311 00:13:53,600 --> 00:13:55,000 So that's the percentage of women 312 00:13:55,000 --> 00:13:57,720 targeted by the project who have improved 313 00:13:57,720 --> 00:14:02,040 the management of their income-generating activity. 314 00:14:02,040 --> 00:14:05,320 So this, again, is what do we mean by improved management? 315 00:14:05,320 --> 00:14:09,200 That's a broad term that requires a specific matrix. 316 00:14:09,200 --> 00:14:13,880 But again, it's also about why not measure the actual improved 317 00:14:13,880 --> 00:14:17,080 income generated because of their participation, 318 00:14:17,080 --> 00:14:19,920 their empowerment through the project. 319 00:14:19,920 --> 00:14:25,320 So these are kind of the missing aspects in the indicator. 320 00:14:25,320 --> 00:14:26,240 Absolutely, yeah. 321 00:14:26,240 --> 00:14:28,800 And I've noticed this in another episode I did, 322 00:14:28,800 --> 00:14:32,320 where they just seem to get scared of using 323 00:14:32,320 --> 00:14:33,880 the I-word, income. 324 00:14:33,880 --> 00:14:36,680 They will talk about savings plans, 325 00:14:36,680 --> 00:14:38,680 but they won't say, show me the money. 326 00:14:38,680 --> 00:14:41,080 And this is a good example where if you just show me 327 00:14:41,080 --> 00:14:45,640 the money, please, and this will show that women are actually 328 00:14:45,640 --> 00:14:48,840 inclusive, being included in economic development 329 00:14:48,840 --> 00:14:50,920 because their incomes are rising. 330 00:14:50,920 --> 00:14:53,160 And you would think it would just look at that. 331 00:14:53,160 --> 00:14:53,960 Exactly. 332 00:14:53,960 --> 00:14:56,120 Again, you can categorize it almost 333 00:14:56,120 --> 00:15:00,640 as an output rather than an outcome indicator. 334 00:15:00,640 --> 00:15:03,520 Yes, yes, exactly. 335 00:15:03,520 --> 00:15:06,400 Because again, they're not measuring increased. 336 00:15:06,400 --> 00:15:08,760 And the key word is increased inclusion. 337 00:15:08,760 --> 00:15:10,600 It just says percent of women targeted 338 00:15:10,600 --> 00:15:13,000 who have improved the management. 339 00:15:13,000 --> 00:15:15,040 So maybe their management has improved, 340 00:15:15,040 --> 00:15:19,200 but that's evading the actual inclusion of economics. 341 00:15:19,200 --> 00:15:22,760 Exactly, exactly. 342 00:15:22,760 --> 00:15:25,080 And then the last outcome of the nine 343 00:15:25,080 --> 00:15:28,640 is strengthening the empowerment of women 344 00:15:28,640 --> 00:15:31,520 among the most vulnerable across Senegal. 345 00:15:31,520 --> 00:15:33,520 And this is where the irony comes 346 00:15:33,520 --> 00:15:35,520 because this is the one indicator 347 00:15:35,520 --> 00:15:39,080 at the very top of the food chain, results chain, 348 00:15:39,080 --> 00:15:41,920 where they are actually using what's 349 00:15:41,920 --> 00:15:45,640 called the Female Empowerment Index, FEMI. 350 00:15:45,640 --> 00:15:49,240 And they probably named it after the well-known musician 351 00:15:49,240 --> 00:15:51,680 from Nigeria, which is kind of cool. 352 00:15:51,680 --> 00:15:54,080 Give them credit for that, even though it's Senegal. 353 00:15:54,080 --> 00:15:54,800 And it's good. 354 00:15:54,800 --> 00:15:56,800 They've used a comparison group. 355 00:15:56,800 --> 00:16:00,400 So they've actually set in the performance measurement 356 00:16:00,400 --> 00:16:04,240 framework, we're going to use a group of women that are not 357 00:16:04,240 --> 00:16:06,180 participating in the project. 358 00:16:06,180 --> 00:16:09,960 And we're going to measure their empowerment levels 359 00:16:09,960 --> 00:16:11,520 on this index. 360 00:16:11,520 --> 00:16:14,040 But the problem is it's a bit late. 361 00:16:14,040 --> 00:16:18,120 So even if they can show, the first thing they need to do 362 00:16:18,120 --> 00:16:20,620 is even before the project starts, 363 00:16:20,620 --> 00:16:22,440 hopefully they've done this, they've 364 00:16:22,440 --> 00:16:24,640 shown that there's no statistical difference 365 00:16:24,640 --> 00:16:27,200 between the two groups to begin with. 366 00:16:27,200 --> 00:16:29,360 That's how you start before the project. 367 00:16:29,360 --> 00:16:32,680 And then you can show over time that the group that they 368 00:16:32,680 --> 00:16:35,360 are engaging with in the project is actually 369 00:16:35,360 --> 00:16:39,200 their index is going up in a much greater rate 370 00:16:39,200 --> 00:16:40,680 than the control group. 371 00:16:40,680 --> 00:16:43,560 And that could be happening. 372 00:16:43,560 --> 00:16:44,480 We don't know. 373 00:16:44,480 --> 00:16:46,840 Because the other issue is we don't have the data. 374 00:16:46,840 --> 00:16:49,040 They only provide a blank PMF. 375 00:16:49,040 --> 00:16:52,520 And if you go to the project browser, Global Affairs Canada, 376 00:16:52,520 --> 00:16:54,760 they don't give the data for this indicator, which 377 00:16:54,760 --> 00:16:57,400 is ironically one of the few good ones. 378 00:16:57,400 --> 00:16:59,680 And there's no data available. 379 00:16:59,680 --> 00:17:01,600 So we don't know. 380 00:17:01,600 --> 00:17:02,120 Yeah. 381 00:17:02,120 --> 00:17:03,200 Sorry, go ahead. 382 00:17:03,200 --> 00:17:05,840 I quite like what they use in the indicator, 383 00:17:05,840 --> 00:17:07,880 like in brackets, adaptive version. 384 00:17:07,880 --> 00:17:09,720 That means they are quite sensitive to adapt 385 00:17:09,720 --> 00:17:11,840 to the local context. 386 00:17:11,840 --> 00:17:14,840 But in terms of improving the indicator, 387 00:17:14,840 --> 00:17:17,000 they could also measure a bit of what we call it 388 00:17:17,000 --> 00:17:18,640 the process indicator. 389 00:17:18,640 --> 00:17:22,160 It's like to what extent that the adaptation process 390 00:17:22,160 --> 00:17:24,840 to the local context is addressing 391 00:17:24,840 --> 00:17:27,840 the cultural diversity sensitivity, for example, 392 00:17:27,840 --> 00:17:31,600 to make those indicators relevant. 393 00:17:31,600 --> 00:17:36,680 So I feel that that could be an area for the organization 394 00:17:36,680 --> 00:17:41,120 to rethink about these kind of composite indicators. 395 00:17:41,120 --> 00:17:41,640 Yes. 396 00:17:41,640 --> 00:17:45,280 And to add to that, there are six components 397 00:17:45,280 --> 00:17:49,320 of the FEMI index, the Female Empowerment Index. 398 00:17:49,320 --> 00:17:51,600 And two of them are relevant to this project. 399 00:17:51,600 --> 00:17:54,760 One is employment, achievements, and achievements 400 00:17:54,760 --> 00:17:56,480 on decision making. 401 00:17:56,480 --> 00:17:58,880 And those two, they obviously I think 402 00:17:58,880 --> 00:18:00,320 have adapted so that they're going 403 00:18:00,320 --> 00:18:03,160 to be asking the women specifically on how are you 404 00:18:03,160 --> 00:18:06,640 doing on employment and how are you doing on decision making. 405 00:18:06,640 --> 00:18:10,200 But the irony is, as we've just covered, 406 00:18:10,200 --> 00:18:13,200 is that for employment, they're not measuring 407 00:18:13,200 --> 00:18:14,360 incomes in the project. 408 00:18:14,360 --> 00:18:17,640 So even if they say yes, we have no data 409 00:18:17,640 --> 00:18:20,120 to support their claim because it's biased, 410 00:18:20,120 --> 00:18:21,640 because they're self-reporting it. 411 00:18:21,640 --> 00:18:24,640 To what extent the income has actually increased. 412 00:18:24,640 --> 00:18:26,000 Right. 413 00:18:26,000 --> 00:18:27,960 They could say yes, but we just don't know. 414 00:18:27,960 --> 00:18:32,440 Because the indicators underneath the FEMI empowerment, 415 00:18:32,440 --> 00:18:34,440 like employment, they're not measuring income. 416 00:18:34,440 --> 00:18:36,440 And on decision making, as we just 417 00:18:36,440 --> 00:18:39,480 discussed, which is the other dimension of the six 418 00:18:39,480 --> 00:18:42,560 they could use in the index, they're not looking at, 419 00:18:42,560 --> 00:18:46,600 did they increase their power on these decision making bodies? 420 00:18:46,600 --> 00:18:47,880 We just don't know. 421 00:18:47,880 --> 00:18:49,840 So that's the irony in all of this 422 00:18:49,840 --> 00:18:52,880 is they've got this wonderful indicator at the end 423 00:18:52,880 --> 00:18:56,640 where they can wave the flag and claim that their project women 424 00:18:56,640 --> 00:19:00,360 compared to the comparison group have become more empowered. 425 00:19:00,360 --> 00:19:03,200 But then when you ask them, what do you mean by more empowered, 426 00:19:03,200 --> 00:19:05,600 you go to the performance measurement framework 427 00:19:05,600 --> 00:19:08,000 and there's no support for whether or not 428 00:19:08,000 --> 00:19:10,320 their incomes went up or whether or not 429 00:19:10,320 --> 00:19:12,440 their decision making power went up 430 00:19:12,440 --> 00:19:16,240 because they're not asking them in the performance measurement 431 00:19:16,240 --> 00:19:18,720 framework for those indicators. 432 00:19:18,720 --> 00:19:20,680 So that's what's really interesting about it 433 00:19:20,680 --> 00:19:23,600 is it's kind of like a smoke and mirrors, as they say. 434 00:19:23,600 --> 00:19:26,880 But that's why we're here is to bring that out. 435 00:19:26,880 --> 00:19:31,160 And if they were to improve the indicators the way 436 00:19:31,160 --> 00:19:33,440 we have discussed them, then they 437 00:19:33,440 --> 00:19:36,680 could make the claim that the reason their empowerment has 438 00:19:36,680 --> 00:19:39,360 gone up, or assuming it is, we don't have the data, 439 00:19:39,360 --> 00:19:41,520 but if we get the data, it would show that. 440 00:19:41,520 --> 00:19:44,280 And it's because we can show their incomes went up 441 00:19:44,280 --> 00:19:46,440 compared to the comparison group, 442 00:19:46,440 --> 00:19:48,360 their decision making power went up 443 00:19:48,360 --> 00:19:51,120 compared to the comparison group, et cetera. 444 00:19:51,120 --> 00:19:54,160 So we just don't know, but at least they're 445 00:19:54,160 --> 00:19:55,680 trying to move in the right direction. 446 00:19:55,680 --> 00:20:00,600 So what we can do now is go back to the outcomes 447 00:20:00,600 --> 00:20:06,600 and look at the other indicators where we could cover all 17 448 00:20:06,600 --> 00:20:08,720 because I think we still have a bit of time. 449 00:20:08,720 --> 00:20:12,800 So we'll go back to the outcome strengthening workshops 450 00:20:12,800 --> 00:20:16,800 on leadership organized with women and men leaders. 451 00:20:16,800 --> 00:20:19,000 The other outcome indicator for that 452 00:20:19,000 --> 00:20:22,080 was number of leadership workshops held. 453 00:20:22,080 --> 00:20:25,160 And it clearly falls into that problem category 454 00:20:25,160 --> 00:20:27,880 of just attendance and everybody goes home, 455 00:20:27,880 --> 00:20:29,440 as I say in my trailer. 456 00:20:29,440 --> 00:20:32,360 They don't even measure whether or not 457 00:20:32,360 --> 00:20:36,960 they've strengthened their leadership skills or knowledge 458 00:20:36,960 --> 00:20:39,440 with respect to leadership after the trading. 459 00:20:39,440 --> 00:20:41,720 So that's a problem there. 460 00:20:41,720 --> 00:20:47,120 The next indicator was for, sorry, 461 00:20:47,120 --> 00:20:48,040 you wanted to make a comment? 462 00:20:48,040 --> 00:20:48,920 Go ahead. 463 00:20:48,920 --> 00:20:52,520 Yeah, I would just like to add to in terms of, 464 00:20:52,520 --> 00:20:55,520 they could also track in terms of the active engagement 465 00:20:55,520 --> 00:20:58,280 of both women and men participants, 466 00:20:58,280 --> 00:21:01,080 but maybe even organizers because those are the women 467 00:21:01,080 --> 00:21:03,960 and men leaders who will organize this training. 468 00:21:03,960 --> 00:21:07,280 So in terms of the engagement, what kind of roles 469 00:21:07,280 --> 00:21:11,120 they are also playing out in running these workshops, 470 00:21:11,120 --> 00:21:13,760 that could also be an interesting aspects 471 00:21:13,760 --> 00:21:16,040 in setting the indicator. 472 00:21:16,040 --> 00:21:18,560 Oh, yes, because one of the target groups 473 00:21:18,560 --> 00:21:20,680 is male champions. 474 00:21:20,680 --> 00:21:25,680 And that's a key, very important point is disaggregation 475 00:21:26,720 --> 00:21:31,720 and seeing if the men are actually championing 476 00:21:31,840 --> 00:21:35,680 and moving in the right direction in that area. 477 00:21:35,680 --> 00:21:38,800 Because right now all we have is we're assuming 478 00:21:38,800 --> 00:21:40,560 that their leadership has been strengthened 479 00:21:40,560 --> 00:21:44,000 from the training, but again, there's really no measurement. 480 00:21:44,000 --> 00:21:44,920 The pre-post, right? 481 00:21:44,920 --> 00:21:48,760 The pre-post workshop assessment is missing out. 482 00:21:48,760 --> 00:21:51,560 Yeah, even a pre-post is limited, 483 00:21:51,560 --> 00:21:53,280 but at least it's better than nothing. 484 00:21:53,280 --> 00:21:54,920 You could do in the same day, 485 00:21:54,920 --> 00:21:56,920 figure out if they've actually learned 486 00:21:56,920 --> 00:21:59,200 and their levels have gone up, 487 00:21:59,200 --> 00:22:02,000 even for that just workshop on training 488 00:22:02,000 --> 00:22:04,360 with respect to leadership, yeah. 489 00:22:04,360 --> 00:22:09,360 So the next one is increased community awareness 490 00:22:09,800 --> 00:22:12,280 of the importance of women's participation. 491 00:22:12,280 --> 00:22:15,920 And what I was looking at there is the other indicator was, 492 00:22:15,920 --> 00:22:19,560 quote, direct beneficiaries in the project 493 00:22:19,560 --> 00:22:23,160 through focus groups also measured on importance 494 00:22:23,160 --> 00:22:26,240 of women's participation in decision-making. 495 00:22:26,240 --> 00:22:28,800 Well, of course there's self-reporting bias there. 496 00:22:28,800 --> 00:22:30,720 They're holding a focus group. 497 00:22:30,720 --> 00:22:34,040 It's of the people that are participating in the project, 498 00:22:34,040 --> 00:22:35,040 the women. 499 00:22:35,040 --> 00:22:38,480 So they're obviously a bias for them to say, 500 00:22:38,480 --> 00:22:43,320 yes, I've become increased aware of the importance 501 00:22:43,320 --> 00:22:45,760 of women's participation in decision-making. 502 00:22:45,760 --> 00:22:48,800 So this is a case where I believe focus groups 503 00:22:48,800 --> 00:22:52,720 are improperly being used, and this is a common problem, 504 00:22:52,720 --> 00:22:55,080 where instead of using the focus group 505 00:22:55,080 --> 00:22:58,560 to compliment a problem on an indicator, 506 00:22:58,560 --> 00:23:01,280 to say, why are you not achieving your target 507 00:23:01,280 --> 00:23:04,040 on this indicator, going in and having a focus group 508 00:23:04,040 --> 00:23:07,320 and asking them, which is why you use focus groups, 509 00:23:07,320 --> 00:23:10,280 to compliment a quantitative indicator. 510 00:23:10,280 --> 00:23:12,920 They're instead using it to replace 511 00:23:12,920 --> 00:23:16,320 the quantitative indicator measure on community awareness, 512 00:23:16,320 --> 00:23:17,360 for example. 513 00:23:17,360 --> 00:23:19,480 So this is a problem that I often see. 514 00:23:19,480 --> 00:23:20,400 Yeah. 515 00:23:20,400 --> 00:23:22,200 Totally agree, yeah. 516 00:23:22,200 --> 00:23:23,400 The next one's for you. 517 00:23:23,400 --> 00:23:24,240 I'll read the outcome. 518 00:23:24,240 --> 00:23:26,160 Next one, yes. 519 00:23:26,160 --> 00:23:28,480 Increased capacity of men and women leaders 520 00:23:28,480 --> 00:23:31,080 to intervene in communities, 521 00:23:31,080 --> 00:23:32,920 and you have an outcome indicator for that. 522 00:23:32,920 --> 00:23:35,840 Yes, the number of men and women trained in leadership 523 00:23:35,840 --> 00:23:39,280 who carry out gender equality sensitization activities 524 00:23:39,280 --> 00:23:43,360 in communities desegregated by gender. 525 00:23:43,360 --> 00:23:48,360 This indicator measures the quantity of activities 526 00:23:48,360 --> 00:23:52,240 and people trained, yeah, but really does not necessarily 527 00:23:52,240 --> 00:23:56,120 reflect the quality of public awareness activities 528 00:23:56,120 --> 00:23:58,320 and what came out of it. 529 00:23:58,320 --> 00:24:02,040 So what they could actually improve is to really 530 00:24:02,040 --> 00:24:05,000 maybe have a follow-up study to actually see 531 00:24:05,000 --> 00:24:07,560 does these kind of activities generate 532 00:24:07,560 --> 00:24:10,400 some sort of behavioral change or attitude change 533 00:24:10,400 --> 00:24:11,880 in the communities. 534 00:24:11,880 --> 00:24:14,400 But again, I think that desegregation by gender 535 00:24:14,400 --> 00:24:18,760 is good to have, but depending on the communities, 536 00:24:18,760 --> 00:24:22,560 you might need also other sort of different matrix 537 00:24:22,560 --> 00:24:25,200 to understand the social economic background 538 00:24:25,200 --> 00:24:28,760 of the community members who actually come forward 539 00:24:28,760 --> 00:24:31,000 to participate in these activities. 540 00:24:31,880 --> 00:24:33,600 Yes, it seems quite complicated, 541 00:24:33,600 --> 00:24:36,720 like when you look at it, how are we gonna say 542 00:24:36,720 --> 00:24:39,040 that these men and women who showed up 543 00:24:39,040 --> 00:24:42,800 for these project trainings actually increase their capacity 544 00:24:42,800 --> 00:24:47,080 to quote intervene in communities effectively? 545 00:24:47,080 --> 00:24:50,120 I mean, the first part is do they have the technical capacity 546 00:24:50,120 --> 00:24:51,920 to go into these communities? 547 00:24:51,920 --> 00:24:55,560 And that's the immediate outcome that they put here. 548 00:24:56,640 --> 00:24:59,080 But you've made a good point about even if they get 549 00:24:59,080 --> 00:25:02,960 to that level, what's gonna happen next is actually 550 00:25:02,960 --> 00:25:07,800 gonna lead to the awareness and changes at the, 551 00:25:07,800 --> 00:25:09,600 downstream as they say. 552 00:25:09,600 --> 00:25:12,880 So yeah, attendance simply is just not good enough. 553 00:25:12,880 --> 00:25:14,440 Yeah, absolutely. 554 00:25:15,400 --> 00:25:19,160 And the next one you're gonna be doing is for the outcome, 555 00:25:19,160 --> 00:25:23,800 increased influence of women as active citizens 556 00:25:23,800 --> 00:25:25,080 in their communities. 557 00:25:26,440 --> 00:25:29,920 Yeah, and the indicator that I was looking at 558 00:25:29,920 --> 00:25:33,720 is the perception of members of community organizations, 559 00:25:33,720 --> 00:25:37,480 men and women on women's participation in decision making 560 00:25:37,480 --> 00:25:41,640 within communities desegregated by sex. 561 00:25:42,800 --> 00:25:46,760 So this is again reliance on the subjective perceptions 562 00:25:46,760 --> 00:25:49,760 which have already we discussed earlier on, 563 00:25:49,760 --> 00:25:52,840 which can be influenced by personal biases, cultural norms. 564 00:25:52,840 --> 00:25:57,000 So we might need other sort of qualitative approaches 565 00:25:57,000 --> 00:26:00,160 or other sources of data to substantiate 566 00:26:00,160 --> 00:26:02,800 or at least to unpack some of these perceptions, 567 00:26:02,800 --> 00:26:04,360 subjective perceptions. 568 00:26:05,600 --> 00:26:08,560 Yeah, and then another aspect just before going back to you 569 00:26:08,560 --> 00:26:11,560 is again, the desegregation just by sex. 570 00:26:11,560 --> 00:26:13,800 I think it also kind of overlooking 571 00:26:13,800 --> 00:26:17,000 on other social economic dimensions, for example, 572 00:26:17,000 --> 00:26:21,000 age, ethnicity, disability, social economic status. 573 00:26:21,000 --> 00:26:23,560 So I think that desegregation is good, 574 00:26:23,560 --> 00:26:26,880 is a good start by gender, but a lot of indicators, 575 00:26:26,880 --> 00:26:29,280 you know, kind of miss out on collecting data 576 00:26:29,280 --> 00:26:34,000 that desegregate on these other important dimensions. 577 00:26:35,000 --> 00:26:38,520 Right, and it's also only measured once a year. 578 00:26:38,520 --> 00:26:41,920 So it's very difficult for the project to make the claim 579 00:26:41,920 --> 00:26:45,040 that their trainings with these women 580 00:26:45,040 --> 00:26:49,000 have actually led to the achievement of the outcome, 581 00:26:49,000 --> 00:26:52,760 which remember is quote, increased influence of women 582 00:26:52,760 --> 00:26:54,880 as active citizens in their communities. 583 00:26:54,880 --> 00:26:57,200 And the way they're gonna try to show that 584 00:26:57,200 --> 00:26:59,320 is by asking, I think it makes sense. 585 00:26:59,320 --> 00:27:01,920 They wanna go to the people on the other end, 586 00:27:01,920 --> 00:27:05,000 the communities that have received this engagement 587 00:27:05,000 --> 00:27:07,800 from these women and say, hey, what do you think? 588 00:27:07,800 --> 00:27:10,360 Do you think these women actually, you know, 589 00:27:10,360 --> 00:27:14,520 influenced you in the desired direction? 590 00:27:14,520 --> 00:27:18,480 Women's participation in decision-making in communities. 591 00:27:18,480 --> 00:27:20,040 That's basically what they're looking at 592 00:27:20,040 --> 00:27:22,640 in terms of the increased influence. 593 00:27:22,640 --> 00:27:27,640 But it seems to be quite a challenge to actually show that. 594 00:27:27,720 --> 00:27:30,920 But only measuring once a year is not good enough. 595 00:27:30,920 --> 00:27:34,280 They have to do it more frequently or also, 596 00:27:34,280 --> 00:27:37,800 or pick a comparison group and ask another group of women 597 00:27:37,800 --> 00:27:42,800 outside of the project, sorry, other communities, 598 00:27:42,920 --> 00:27:46,400 what they think of women's participation in decision-making 599 00:27:46,400 --> 00:27:49,200 and show that there's a major difference 600 00:27:49,200 --> 00:27:51,760 or measure more than just once a year. 601 00:27:51,760 --> 00:27:52,600 Exactly. 602 00:27:53,760 --> 00:27:55,720 The next one. 603 00:27:55,720 --> 00:27:59,440 Next one is also mine, right? 604 00:28:00,360 --> 00:28:01,520 Yeah, I'll read the outcome. 605 00:28:01,520 --> 00:28:05,120 Increased influence of women as active citizens 606 00:28:05,120 --> 00:28:05,960 in their communities. 607 00:28:05,960 --> 00:28:07,520 That's the same outcome. 608 00:28:07,520 --> 00:28:11,600 But I think the outcome indicator is slightly different. 609 00:28:11,600 --> 00:28:12,440 Yes. 610 00:28:12,440 --> 00:28:15,560 Yes, this is about the perception of women 611 00:28:15,560 --> 00:28:17,960 targeted by the project on the participation 612 00:28:17,960 --> 00:28:22,160 in decision-making within communities and in the family. 613 00:28:22,160 --> 00:28:24,600 So, but again, it's kind of similar 614 00:28:24,600 --> 00:28:26,680 to what we talked about previously 615 00:28:26,680 --> 00:28:30,440 is about the perception, again, that can be sort of affected 616 00:28:30,440 --> 00:28:34,520 by other sort of internalized gender roles, 617 00:28:34,520 --> 00:28:37,560 self-esteem and societal expectations. 618 00:28:37,560 --> 00:28:39,560 So this may not align with the actual. 619 00:28:39,560 --> 00:28:41,400 So instead of measuring the actual, 620 00:28:41,400 --> 00:28:45,680 which we highly recommend in terms of the influence 621 00:28:45,680 --> 00:28:48,800 of their participation, purely relying on perception 622 00:28:48,800 --> 00:28:52,560 may not be sufficient to really be an indicator 623 00:28:52,560 --> 00:28:55,760 for the outcomes, the increased influence of women 624 00:28:55,760 --> 00:28:58,800 in the, as active citizens in their communities. 625 00:28:58,800 --> 00:29:00,840 And in this case, it's in the family. 626 00:29:00,840 --> 00:29:04,000 So, you know, what we could sort of an improved indicator 627 00:29:04,000 --> 00:29:07,920 would could, for example, is maybe, you know, 628 00:29:07,920 --> 00:29:11,160 getting some information from the family members, 629 00:29:11,160 --> 00:29:14,400 like community members, for example. 630 00:29:14,400 --> 00:29:16,560 So this again, self-reporting bias, 631 00:29:16,560 --> 00:29:18,840 I think David, you already talk about 632 00:29:18,840 --> 00:29:23,080 on some other indicators that also comes into this indicator. 633 00:29:24,000 --> 00:29:25,400 Yeah, exactly. 634 00:29:25,400 --> 00:29:28,560 We preferred that they use the other approach 635 00:29:28,560 --> 00:29:30,120 that they used in the previous indicator, 636 00:29:30,120 --> 00:29:32,960 go to community members, not the women themselves 637 00:29:32,960 --> 00:29:35,640 in the project, or like you said, quite rightly, 638 00:29:35,640 --> 00:29:39,680 other family members, because I think they're claiming here 639 00:29:39,680 --> 00:29:42,600 that their decision-making power is going to go up 640 00:29:42,600 --> 00:29:45,280 even within the family for these women, 641 00:29:45,280 --> 00:29:47,480 not just in the community at large. 642 00:29:47,480 --> 00:29:52,480 And this is in other episodes, sorry to bore you, Jenny, 643 00:29:52,480 --> 00:29:55,480 I'll bring it in, in Vietnam with Care Canada, 644 00:29:55,480 --> 00:29:58,320 we raised this same problem where, 645 00:29:58,320 --> 00:30:02,200 how do you get at measuring increased women's power 646 00:30:02,200 --> 00:30:03,880 in the household? 647 00:30:03,880 --> 00:30:06,600 It's quite tricky, but you could go and ask 648 00:30:06,600 --> 00:30:09,520 other members in the household, but that's not even done. 649 00:30:09,520 --> 00:30:12,400 Self-efficacy is important, yeah, 650 00:30:12,400 --> 00:30:14,400 but at the same time, I think for, 651 00:30:14,400 --> 00:30:17,600 in order to generate robust evidence base, 652 00:30:17,600 --> 00:30:21,960 other type of data collection methods would also be needed 653 00:30:21,960 --> 00:30:24,680 to strengthen the evidence base. 654 00:30:25,600 --> 00:30:27,720 Right, exactly. 655 00:30:27,720 --> 00:30:32,240 The same outcome, increased inclusion of women as actors 656 00:30:32,240 --> 00:30:33,880 in social and economic development, 657 00:30:33,880 --> 00:30:36,240 sorry, that's not the same outcome, it's the next one. 658 00:30:36,240 --> 00:30:39,880 There was another indicator they used called, 659 00:30:39,880 --> 00:30:43,520 and I quote, percent of women targeted by the project 660 00:30:43,520 --> 00:30:46,360 who are business leaders or entrepreneurs, 661 00:30:46,360 --> 00:30:48,400 that it just doesn't go far enough. 662 00:30:48,400 --> 00:30:51,680 They could tick off the box, but are they earning any money? 663 00:30:51,680 --> 00:30:55,040 And are they earning more money than those in the, 664 00:30:55,040 --> 00:30:57,160 that are not participating in the project 665 00:30:57,160 --> 00:30:59,960 and not receiving business training, 666 00:30:59,960 --> 00:31:01,880 leadership training, et cetera? 667 00:31:01,880 --> 00:31:05,720 So you definitely need to have a comparison group, 668 00:31:05,720 --> 00:31:09,000 or even without a comparison group, it's the same problem. 669 00:31:09,000 --> 00:31:11,400 They can't just tick off the box and say, 670 00:31:11,400 --> 00:31:13,360 I'm a business leader and raise their hand. 671 00:31:13,360 --> 00:31:17,560 They've got to show that they've actually earned income 672 00:31:17,560 --> 00:31:20,680 as a better measure of increased inclusion 673 00:31:20,680 --> 00:31:22,480 as part of economic development. 674 00:31:23,440 --> 00:31:28,040 And the next one, indicator for that same outcome, 675 00:31:28,040 --> 00:31:30,640 increased inclusion of women as actors 676 00:31:30,640 --> 00:31:32,680 in social and economic development. 677 00:31:32,680 --> 00:31:34,360 Another indicator they used 678 00:31:34,360 --> 00:31:36,360 in the performance measurement framework 679 00:31:36,360 --> 00:31:39,040 was percent of women in employment 680 00:31:39,040 --> 00:31:41,360 or self-employment after training. 681 00:31:43,040 --> 00:31:45,800 So again, the problem there is the same, 682 00:31:45,800 --> 00:31:49,000 is it's a good idea that they've been employed, 683 00:31:49,000 --> 00:31:53,040 but they need to show that the training from the project 684 00:31:53,040 --> 00:31:55,440 has made the higher percentage of these women 685 00:31:55,440 --> 00:31:59,440 become employed or self-employed, specifically with income, 686 00:31:59,440 --> 00:32:03,520 compared to women who didn't benefit from the project, 687 00:32:03,520 --> 00:32:07,920 didn't receive the services such as leadership training 688 00:32:07,920 --> 00:32:10,960 and entrepreneurship. 689 00:32:10,960 --> 00:32:13,880 Maybe another aspect is also the nature of the jobs, 690 00:32:13,880 --> 00:32:17,440 the employment, is it related to the training or not? 691 00:32:18,320 --> 00:32:19,680 Exactly. 692 00:32:19,680 --> 00:32:21,480 Disaggregation, please. 693 00:32:21,480 --> 00:32:22,800 Absolutely, yeah. 694 00:32:24,280 --> 00:32:27,000 And I think we've got one more here, 695 00:32:27,000 --> 00:32:29,200 increased inclusion of women as actors 696 00:32:29,200 --> 00:32:31,320 in social and economic development. 697 00:32:31,320 --> 00:32:35,120 And again, one of the indicators is perception of women 698 00:32:35,120 --> 00:32:39,240 targeted by the project and men 699 00:32:39,240 --> 00:32:42,080 guarding their socioeconomic situation. 700 00:32:42,080 --> 00:32:44,280 So your point's a good one, Jenny, 701 00:32:44,280 --> 00:32:48,280 that they're disaggregating the men and the women, 702 00:32:48,280 --> 00:32:50,960 which is good, to see if both groups 703 00:32:50,960 --> 00:32:52,760 are moving in the right direction. 704 00:32:52,760 --> 00:32:56,240 The problem is, is it's the targeted by the project. 705 00:32:56,240 --> 00:32:59,520 So again, you're gonna have some self-reporting bias 706 00:32:59,520 --> 00:33:01,760 where they're gonna say, oh yeah, my measure, 707 00:33:01,760 --> 00:33:04,520 or however they define it, of being included 708 00:33:04,520 --> 00:33:07,240 in the economy has increased. 709 00:33:08,440 --> 00:33:10,320 But they need to use other measures 710 00:33:10,320 --> 00:33:14,080 that we've suggested before to get at this increased 711 00:33:14,080 --> 00:33:18,760 inclusion rather than asking the project participants 712 00:33:18,760 --> 00:33:22,360 themselves because of this self-reporting bias. 713 00:33:22,360 --> 00:33:24,320 Yeah, you also have an earlier point 714 00:33:24,320 --> 00:33:25,880 about the targeted group. 715 00:33:25,880 --> 00:33:28,000 Are they reaching out to the targeted group? 716 00:33:28,000 --> 00:33:30,880 Because it's all about the underrepresented 717 00:33:30,880 --> 00:33:34,600 or marginalized women and men. 718 00:33:34,600 --> 00:33:38,320 But again, it's that, are they, are the target group, 719 00:33:38,320 --> 00:33:42,520 have improved their socioeconomic status or situation? 720 00:33:42,520 --> 00:33:45,360 That is also of interest to measure the outcome. 721 00:33:46,400 --> 00:33:48,680 Oh, absolutely, that's one of the issues is, 722 00:33:49,720 --> 00:33:52,160 I've found with many projects, 723 00:33:52,160 --> 00:33:54,720 the vulnerable women as they identify them here, 724 00:33:54,720 --> 00:33:56,520 sometimes they're not even reached. 725 00:33:56,520 --> 00:33:57,360 Exactly. 726 00:33:57,360 --> 00:34:01,840 And it's the women who run the NGOs 727 00:34:01,840 --> 00:34:04,640 who benefit the most from the project 728 00:34:04,640 --> 00:34:06,800 in terms of salary, et cetera, 729 00:34:06,800 --> 00:34:11,440 and the actual delivery to the vulnerable women is minimal. 730 00:34:11,440 --> 00:34:15,400 And in this case, the measurement is certainly minimal. 731 00:34:15,400 --> 00:34:20,400 So it's very difficult to reach. 732 00:34:20,720 --> 00:34:24,200 So I think we can conclude that, 733 00:34:24,200 --> 00:34:25,920 at least from my point of view, 734 00:34:25,920 --> 00:34:29,080 that this project in its current design 735 00:34:29,080 --> 00:34:32,000 from performance measurement is not adequate 736 00:34:32,000 --> 00:34:35,040 to support the project in colleges 737 00:34:35,040 --> 00:34:37,040 and institutes Canada claiming 738 00:34:37,040 --> 00:34:38,680 that they're achieving their outcomes. 739 00:34:38,680 --> 00:34:43,680 That is, that women are becoming more empowered economically 740 00:34:43,680 --> 00:34:45,200 and politically. 741 00:34:45,200 --> 00:34:46,840 But that's my opinion. 742 00:34:46,840 --> 00:34:48,960 What do you think, Jenny? 743 00:34:48,960 --> 00:34:51,600 Now, I agree with you with your conclusion, 744 00:34:51,600 --> 00:34:53,800 but I also want to applaud the organization 745 00:34:53,800 --> 00:34:57,320 for setting actually very good outcome statements. 746 00:34:57,320 --> 00:34:59,960 But I think there's really a space as we, 747 00:34:59,960 --> 00:35:03,120 the objective of our podcast is actually 748 00:35:03,120 --> 00:35:05,400 try to improve practice when it comes 749 00:35:05,400 --> 00:35:07,720 to formulating the indicators. 750 00:35:07,720 --> 00:35:10,040 So if there's already a lot of investment 751 00:35:10,040 --> 00:35:12,320 in setting out very good outcomes, 752 00:35:12,320 --> 00:35:15,840 with suggestion then why not just also improve the indicators? 753 00:35:15,840 --> 00:35:18,240 So there is actually a very clear, 754 00:35:18,240 --> 00:35:21,080 more clarity and specific matrix 755 00:35:21,080 --> 00:35:23,000 that actually generate evidence base 756 00:35:23,000 --> 00:35:24,520 to support the argument, 757 00:35:24,520 --> 00:35:26,960 whether the outcomes have been achieved or not. 758 00:35:28,080 --> 00:35:29,200 Well said. 759 00:35:29,200 --> 00:35:31,640 Of course, I have a bias myself, 760 00:35:31,640 --> 00:35:36,400 setting up this podcast for that whole intent and purpose. 761 00:35:36,400 --> 00:35:38,240 So thank you, Jenny. 762 00:35:38,240 --> 00:35:40,440 Thank you so much, David. 763 00:35:40,440 --> 00:35:41,280 Yeah. 764 00:35:41,280 --> 00:35:43,240 Thank you for having me. 765 00:35:43,240 --> 00:35:44,080 You're welcome. 766 00:35:44,080 --> 00:35:48,640 And so what we'll do now is the last sentence in this podcast 767 00:35:48,640 --> 00:35:51,520 or a few summary remarks is that, 768 00:35:51,520 --> 00:35:54,160 we will send these part one and part two, 769 00:35:54,160 --> 00:35:55,960 two colleges and institutes Canada, 770 00:35:55,960 --> 00:35:57,520 but more importantly, 771 00:35:57,520 --> 00:36:00,040 we'll send it to the honorable minister 772 00:36:00,040 --> 00:36:01,480 for international development, 773 00:36:01,480 --> 00:36:04,760 as well as the shadow critics for international development 774 00:36:04,760 --> 00:36:06,280 in the conservative party, 775 00:36:06,280 --> 00:36:08,160 the new democratic party, 776 00:36:08,160 --> 00:36:10,120 the Bloc Québécois party, 777 00:36:10,120 --> 00:36:14,680 as well as I think maybe even the leader of the Green Party. 778 00:36:14,680 --> 00:36:17,960 And we'll leave it at that and continue to recommend 779 00:36:17,960 --> 00:36:20,840 that one, they make the performance measurement frameworks 780 00:36:20,840 --> 00:36:22,960 available to the public 781 00:36:22,960 --> 00:36:25,000 on the Global Affairs Canada website. 782 00:36:25,000 --> 00:36:27,920 Number two, they actually provide the data 783 00:36:27,920 --> 00:36:29,200 from the indicators. 784 00:36:29,200 --> 00:36:30,640 Here, we don't even have the data. 785 00:36:30,640 --> 00:36:33,880 It's just a blank performance measurement framework. 786 00:36:33,880 --> 00:36:36,760 And also finally, as we've discussed, 787 00:36:36,760 --> 00:36:40,240 the Global Affairs Canada should follow their own guidelines. 788 00:36:40,240 --> 00:36:41,840 And that is when they approve 789 00:36:41,840 --> 00:36:43,760 a performance measurement framework, 790 00:36:43,760 --> 00:36:45,240 they should be saying, 791 00:36:45,240 --> 00:36:46,880 why are your outcome indicators 792 00:36:46,880 --> 00:36:49,680 not properly measuring your outcomes? 793 00:36:49,680 --> 00:36:54,680 Please revise before we disperse $18 million. 794 00:36:54,800 --> 00:36:58,400 Finally, if you're gonna be a charity, 795 00:36:58,400 --> 00:37:01,000 become a charity and don't make these claims 796 00:37:01,000 --> 00:37:02,480 of these outcomes that you're achieving. 797 00:37:02,480 --> 00:37:03,760 There's nothing wrong with that, 798 00:37:03,760 --> 00:37:05,120 nothing to be ashamed of. 799 00:37:05,960 --> 00:37:07,920 And that's the other option they can have, 800 00:37:07,920 --> 00:37:09,360 these organizations, 801 00:37:09,360 --> 00:37:11,440 is don't have any outcomes at all. 802 00:37:11,440 --> 00:37:15,040 Just deliver the services because everybody needs them 803 00:37:15,040 --> 00:37:16,600 and everybody goes home. 804 00:37:16,600 --> 00:37:19,280 There's nothing wrong with that as another option. 805 00:37:19,280 --> 00:37:20,520 What do you think, Jenny? 806 00:37:21,720 --> 00:37:24,800 No, that's a very good remarks to sum up. 807 00:37:24,800 --> 00:37:26,680 But as a practitioner, 808 00:37:26,680 --> 00:37:28,800 I also suspect that there's actually maybe 809 00:37:28,800 --> 00:37:31,160 updated performance measurement framework 810 00:37:31,160 --> 00:37:32,560 that we're not really shared. 811 00:37:32,560 --> 00:37:35,320 So maybe there are some ongoing substantive work 812 00:37:35,320 --> 00:37:37,400 to improve the monitoring evaluation, 813 00:37:37,400 --> 00:37:39,040 but maybe these kind of documents 814 00:37:39,040 --> 00:37:41,600 are not always shared externally. 815 00:37:41,600 --> 00:37:44,400 So I also encourage implementing agency, 816 00:37:44,400 --> 00:37:46,080 particularly with such a huge budget, 817 00:37:46,080 --> 00:37:48,600 to really make this kind of information 818 00:37:48,600 --> 00:37:49,760 also available. 819 00:37:50,680 --> 00:37:52,400 That's a good point. 820 00:37:52,400 --> 00:37:56,320 And also some of these NGOs have claimed 821 00:37:56,320 --> 00:37:59,440 they don't have the evaluation expertise 822 00:37:59,440 --> 00:38:01,280 to design proper indicators 823 00:38:01,280 --> 00:38:04,160 or they don't have the resources or money 824 00:38:04,160 --> 00:38:07,400 to have proper comparison groups 825 00:38:07,400 --> 00:38:08,320 because it's expensive, 826 00:38:08,320 --> 00:38:10,520 you have to measure another control group. 827 00:38:10,520 --> 00:38:13,720 Or as we've advocated, 828 00:38:13,720 --> 00:38:15,160 you have to measure more frequently, 829 00:38:15,160 --> 00:38:16,400 which costs more money. 830 00:38:16,400 --> 00:38:19,320 But my position is the government of Canada 831 00:38:19,320 --> 00:38:21,400 should say, here's your 18 million, 832 00:38:21,400 --> 00:38:24,600 but by the way, 10 to 15% of it 833 00:38:24,600 --> 00:38:27,200 has to go to monitoring and evaluation. 834 00:38:27,200 --> 00:38:28,040 Right there. 835 00:38:28,040 --> 00:38:31,920 I think what, yeah, I totally agree with you, David. 836 00:38:31,920 --> 00:38:33,680 There's always the budget concern, 837 00:38:33,680 --> 00:38:35,120 the expertise concern, 838 00:38:35,120 --> 00:38:37,360 but even though there's a wide range of resources 839 00:38:37,360 --> 00:38:41,240 out there to improve monitoring evaluation practices, 840 00:38:41,240 --> 00:38:43,120 the issue is I think what we're advocating 841 00:38:43,120 --> 00:38:47,360 is that maybe steer a little bit away from the desire 842 00:38:47,360 --> 00:38:49,360 to just quantitative indicators 843 00:38:49,360 --> 00:38:51,160 could maybe perhaps use a mixture 844 00:38:51,160 --> 00:38:53,880 of quantitative and qualitative indicators. 845 00:38:53,880 --> 00:38:57,360 So at least you can have more substantive, 846 00:38:57,360 --> 00:39:01,600 more robust sort of measurement to measure results. 847 00:39:03,080 --> 00:39:03,920 Yeah, I agree. 848 00:39:03,920 --> 00:39:05,440 As I mentioned earlier, 849 00:39:05,440 --> 00:39:07,880 some people may suspect I have a quantitative bias, 850 00:39:07,880 --> 00:39:09,440 even if that's the case, 851 00:39:09,440 --> 00:39:10,800 they have to be complimentary. 852 00:39:10,800 --> 00:39:13,800 You can have increased inclusion. 853 00:39:13,800 --> 00:39:16,120 You can have a quantitative indicator for that, 854 00:39:16,120 --> 00:39:19,560 but you can also have a focus group qualitative 855 00:39:19,560 --> 00:39:21,680 to compliment that by saying, 856 00:39:21,680 --> 00:39:25,000 okay, we have statistical significance here. 857 00:39:25,000 --> 00:39:28,000 It's clearly shown that your incomes have gone up 858 00:39:28,000 --> 00:39:29,760 greater than the control group, 859 00:39:29,760 --> 00:39:33,440 but we'd also like to know what worked, what didn't 860 00:39:33,440 --> 00:39:35,600 in terms of increasing your incomes. 861 00:39:35,600 --> 00:39:38,800 And that could compliment the actual achievement 862 00:39:38,800 --> 00:39:41,920 in the right direction of the quantitative indicator. 863 00:39:41,920 --> 00:39:43,560 And you certainly can use both. 864 00:39:43,560 --> 00:39:46,000 The problem is they often don't, 865 00:39:46,000 --> 00:39:49,360 and they often miss the mark on the quantitative 866 00:39:49,360 --> 00:39:51,400 as we've I think highlighted here, 867 00:39:51,400 --> 00:39:55,280 or they abuse the qualitative by using it just by itself. 868 00:39:55,280 --> 00:39:58,800 So I think we're moving in the right direction, yes. 869 00:39:58,800 --> 00:40:03,360 Yeah, maybe just one last comment, if I may, David. 870 00:40:03,360 --> 00:40:05,720 Of course, we don't know the actual implementation 871 00:40:05,720 --> 00:40:07,200 of the monitoring evaluation 872 00:40:07,200 --> 00:40:10,640 or performance management framework, as you call it, 873 00:40:10,640 --> 00:40:13,480 that we also encourage, at least from my point of view, 874 00:40:13,480 --> 00:40:15,840 to encourage, if that wasn't already done, 875 00:40:15,840 --> 00:40:18,400 collaboration with the local research institutes 876 00:40:18,400 --> 00:40:21,200 to collect this kind of qualitative data 877 00:40:21,200 --> 00:40:24,840 or quantitative data, so to improve actual measurement 878 00:40:24,840 --> 00:40:26,280 of outcomes. 879 00:40:27,320 --> 00:40:28,760 Absolutely, and in fact, 880 00:40:29,960 --> 00:40:32,760 I think a lot of the organizations do that, 881 00:40:32,760 --> 00:40:35,080 use local research institutes. 882 00:40:35,080 --> 00:40:37,800 I've been involved in that when I was in Nigeria 883 00:40:38,840 --> 00:40:42,960 doing third-party oversight on local Nigerian survey firms 884 00:40:42,960 --> 00:40:46,040 that were collecting the data for the indicators. 885 00:40:46,040 --> 00:40:48,600 Absolutely, yeah, that's a very good point. 886 00:40:48,600 --> 00:40:51,920 In my trailer, I talk about that's important, 887 00:40:51,920 --> 00:40:54,400 but what's really important is the initial design, 888 00:40:54,400 --> 00:40:57,160 where if you don't design the indicators properly, 889 00:40:57,160 --> 00:41:00,000 you have garbage in, garbage out. 890 00:41:00,000 --> 00:41:03,920 And that's the focal point of my podcast, 891 00:41:03,920 --> 00:41:07,800 but yes, absolutely, you have to use local firms, 892 00:41:07,800 --> 00:41:11,680 and it's quite fascinating how many local firms 893 00:41:11,680 --> 00:41:15,000 in this global economy have expanded 894 00:41:15,000 --> 00:41:18,880 by relocating in developing countries. 895 00:41:18,880 --> 00:41:22,560 There's quite a few very impressive 896 00:41:22,560 --> 00:41:25,880 local data collection firms that have developed 897 00:41:25,880 --> 00:41:29,520 to address exactly what you've raised, yes. 898 00:41:29,520 --> 00:41:31,680 Yeah, maybe on that note, 899 00:41:31,680 --> 00:41:33,760 because I remember in your part one podcast, 900 00:41:33,760 --> 00:41:36,760 you talk about one of the target group of the project 901 00:41:36,760 --> 00:41:38,360 is the project staff itself. 902 00:41:38,360 --> 00:41:40,000 So maybe that's also an avenue, 903 00:41:40,000 --> 00:41:42,880 if that is not already done, 904 00:41:42,880 --> 00:41:45,280 is to include performance management 905 00:41:45,280 --> 00:41:50,280 or measurement capacity as part of the key competencies 906 00:41:50,600 --> 00:41:54,080 to develop the local project team members, 907 00:41:54,080 --> 00:41:57,200 sort of part of their competence development framework. 908 00:41:58,400 --> 00:42:00,640 Oh, absolutely, and that's a big issue, 909 00:42:00,640 --> 00:42:02,960 is some of these NGOs will respond 910 00:42:02,960 --> 00:42:05,720 that they don't have the technical capacity 911 00:42:05,720 --> 00:42:09,480 to do proper measurement and evaluation, 912 00:42:09,480 --> 00:42:11,360 and it's always a challenge. 913 00:42:12,440 --> 00:42:14,880 Absolutely, yes, very good point. 914 00:42:14,880 --> 00:42:16,760 So I think we'll end it at that, 915 00:42:16,760 --> 00:42:18,880 and we'll send it off to the minister, 916 00:42:18,880 --> 00:42:23,240 and hopefully we'll build this slow body of evidence 917 00:42:23,240 --> 00:42:26,600 where maybe someday performance measurement frameworks 918 00:42:26,600 --> 00:42:29,360 will be on Government of Canada websites 919 00:42:29,360 --> 00:42:32,440 for the four to five billion dollars worth of projects 920 00:42:32,440 --> 00:42:34,080 that they have every year, 921 00:42:34,080 --> 00:42:38,480 and the indicators will properly measure their outcomes. 922 00:42:38,480 --> 00:42:40,640 Okay. Thank you so much, David. 923 00:42:40,640 --> 00:42:42,040 Okay, we'll keep in touch, 924 00:42:42,040 --> 00:42:44,880 and maybe you'll be on another podcast if you want. 925 00:42:44,880 --> 00:43:03,880 Thank you.