1 00:00:00,000 --> 00:00:03,240 Welcome to the Improving Development Evaluation Podcast. 2 00:00:03,240 --> 00:00:06,040 I'm your host David Wand and welcome to episode one, 3 00:00:06,040 --> 00:00:08,640 part two, where we feature Care Canada 4 00:00:08,640 --> 00:00:11,800 as our international development organization. 5 00:00:11,800 --> 00:00:15,420 Before I introduce you to our two evaluation experts, 6 00:00:15,420 --> 00:00:17,560 I'm gonna give you a brief synopsis 7 00:00:17,560 --> 00:00:19,680 of what I've done with part one. 8 00:00:19,680 --> 00:00:23,540 On October 19th, I sent to Care Canada by email 9 00:00:23,540 --> 00:00:26,120 part one of this episode, 10 00:00:26,120 --> 00:00:27,520 explaining my critique of their 11 00:00:27,520 --> 00:00:29,440 performance measurement framework. 12 00:00:29,440 --> 00:00:31,440 And then on October 20th, 13 00:00:31,440 --> 00:00:33,400 I followed up with a phone call 14 00:00:33,400 --> 00:00:35,280 to make sure they had received the email. 15 00:00:35,280 --> 00:00:39,120 And then I followed up again on October 30th 16 00:00:39,120 --> 00:00:41,700 with a repeat email reminding them 17 00:00:41,700 --> 00:00:44,720 of the invitation to attend the podcast. 18 00:00:44,720 --> 00:00:45,960 It's now November 6th. 19 00:00:45,960 --> 00:00:49,200 We have not received a reply from Care Canada 20 00:00:49,200 --> 00:00:50,940 regarding whether to attend 21 00:00:50,940 --> 00:00:53,080 and we have not received a written response, 22 00:00:53,080 --> 00:00:54,780 which was another option for them. 23 00:00:54,780 --> 00:00:56,260 If they were unable to attend, 24 00:00:56,260 --> 00:00:58,120 they could send a written response 25 00:00:58,120 --> 00:01:01,560 to the critique of their performance measurement framework. 26 00:01:01,560 --> 00:01:03,160 To be fair to Care Canada, 27 00:01:03,160 --> 00:01:05,560 today it's 7 a.m. Ottawa time, 28 00:01:05,560 --> 00:01:07,000 where Care Canada is located. 29 00:01:07,000 --> 00:01:08,760 That's a bit early in the morning, 30 00:01:08,760 --> 00:01:10,760 but they could have sent a written response, 31 00:01:10,760 --> 00:01:12,960 but we haven't heard anything from them. 32 00:01:12,960 --> 00:01:16,760 So now we're gonna proceed and go into more detail 33 00:01:16,760 --> 00:01:19,040 about the outcome indicators and why they're flawed 34 00:01:19,040 --> 00:01:21,180 in their performance measurement framework. 35 00:01:21,180 --> 00:01:24,960 From Georgetown, Guyana, I have Donald Cole. 36 00:01:24,960 --> 00:01:27,440 Donald Cole holds a master's of science degree 37 00:01:27,440 --> 00:01:30,760 in data engineering from Edinburgh Napier University 38 00:01:30,760 --> 00:01:33,440 and is a monitoring and evaluation specialist. 39 00:01:33,440 --> 00:01:37,440 And we also have from Kampala, Uganda, Dr. Mark Degner, 40 00:01:37,440 --> 00:01:40,280 who holds PhD in social and economic science 41 00:01:40,280 --> 00:01:43,560 from Karl Franzens University in Austria. 42 00:01:43,560 --> 00:01:45,800 And he is currently a technical advisor 43 00:01:45,800 --> 00:01:49,600 in advocacy, communication, and documentation. 44 00:01:49,600 --> 00:01:50,740 So what we're gonna do now 45 00:01:50,740 --> 00:01:52,280 is we're gonna go through the outcomes. 46 00:01:52,280 --> 00:01:54,880 So let's start, I'll read out the outcome, 47 00:01:54,880 --> 00:01:56,720 and then Donald, you can talk about 48 00:01:56,720 --> 00:01:58,680 the outcome indicator and why it's flawed, 49 00:01:58,680 --> 00:02:03,440 and even if you wish to provide an alternative indicator. 50 00:02:03,440 --> 00:02:07,400 So beginning with the first outcome in the project, 51 00:02:07,400 --> 00:02:10,660 increased entrepreneurship capacity 52 00:02:10,660 --> 00:02:14,440 of women-led grassroots social enterprises 53 00:02:14,440 --> 00:02:18,100 to pursue economic opportunities in Vietnam. 54 00:02:18,100 --> 00:02:20,400 That's the outcome and the outcome indicator? 55 00:02:22,440 --> 00:02:25,820 Yes, the outcome indicator for that particular outcome 56 00:02:25,820 --> 00:02:30,280 is number of women-led grassroots social enterprises 57 00:02:30,280 --> 00:02:33,880 reported increased capacity to engage 58 00:02:33,880 --> 00:02:37,600 in entrepreneurship opportunities in Vietnam. 59 00:02:38,520 --> 00:02:40,840 So this particular indicator, 60 00:02:40,840 --> 00:02:44,920 there's like three main points I would like to highlight 61 00:02:44,920 --> 00:02:46,280 for this indicator. 62 00:02:46,280 --> 00:02:49,980 The first one is the vagueness in increased capacity. 63 00:02:49,980 --> 00:02:52,440 The term increased capacity is vague 64 00:02:52,440 --> 00:02:56,200 and it does not really specify what aspects of capacity 65 00:02:56,200 --> 00:02:57,840 are being measured. 66 00:02:57,840 --> 00:03:01,840 The second point is its reliance on self-reporting. 67 00:03:02,760 --> 00:03:05,160 And that data can be very biased 68 00:03:05,160 --> 00:03:10,160 because self-reporting data persons tend to be very biased 69 00:03:11,400 --> 00:03:15,240 in terms of how they report and provide feedback. 70 00:03:15,240 --> 00:03:18,000 And the third key point on this indicator 71 00:03:18,000 --> 00:03:19,800 is the lack of objective measures. 72 00:03:19,800 --> 00:03:24,160 There are no objective measures for entrepreneurship capacity 73 00:03:24,160 --> 00:03:27,080 or success. 74 00:03:28,280 --> 00:03:31,100 In terms of improving this particular indicator, 75 00:03:32,800 --> 00:03:35,280 the first thing would need to be done 76 00:03:35,280 --> 00:03:37,860 is to have a clear definition, obviously, 77 00:03:37,860 --> 00:03:39,600 of what capacity entails, 78 00:03:39,600 --> 00:03:43,380 whether it entails business skills, access to capital, 79 00:03:43,380 --> 00:03:45,920 market access, et cetera. 80 00:03:45,920 --> 00:03:48,000 The second thing that would be needed to address 81 00:03:48,000 --> 00:03:51,900 is the objective matrix of the indicator. 82 00:03:51,900 --> 00:03:54,560 What are the objective matrix are we going to measure? 83 00:03:54,560 --> 00:03:57,800 Are we measuring revenue growth, employment rates, 84 00:03:57,800 --> 00:04:01,480 number of new businesses started, and so forth. 85 00:04:01,480 --> 00:04:03,960 The third thing that would need to be addressed 86 00:04:03,960 --> 00:04:07,480 is periodic and independent assessments. 87 00:04:07,480 --> 00:04:09,280 Conduct periodic assessments. 88 00:04:09,280 --> 00:04:10,760 And these assessments should be done 89 00:04:10,760 --> 00:04:14,780 by independent evaluators to ensure 90 00:04:14,780 --> 00:04:17,060 that they measure the growth and the sustainability 91 00:04:17,060 --> 00:04:18,880 of the enterprises. 92 00:04:18,880 --> 00:04:23,040 And then there's an issue with confidence versus competence. 93 00:04:23,040 --> 00:04:24,600 While confidence is important, 94 00:04:24,600 --> 00:04:27,480 it should be measured alongside competence, 95 00:04:27,480 --> 00:04:31,320 especially when it comes to entrepreneurship skills. 96 00:04:31,320 --> 00:04:34,760 That would be my brief critique of this particular indicator 97 00:04:34,760 --> 00:04:38,000 and ways in which this indicator can be improved. 98 00:04:38,000 --> 00:04:41,680 I didn't come up with a rewarded indicator 99 00:04:41,680 --> 00:04:43,720 because I think I would probably need 100 00:04:43,720 --> 00:04:44,920 a bit more information, 101 00:04:44,920 --> 00:04:48,280 but those key points would definitely address the issues 102 00:04:48,280 --> 00:04:49,840 with this particular indicator. 103 00:04:51,080 --> 00:04:52,100 That's excellent, Donald. 104 00:04:52,100 --> 00:04:53,240 Thank you. 105 00:04:53,240 --> 00:04:55,680 So we'll move on to the next outcome statement 106 00:04:55,680 --> 00:04:57,880 in the performance measurement framework 107 00:04:57,880 --> 00:05:01,400 for this project by Care Canada delivered in Vietnam. 108 00:05:01,400 --> 00:05:04,560 That's increased gender responsive capacity 109 00:05:04,560 --> 00:05:07,620 of poor, rural, and ethnic minorities, 110 00:05:07,620 --> 00:05:10,960 especially women, to save, borrow money, 111 00:05:10,960 --> 00:05:14,840 to support productive and sustainable economic activities 112 00:05:14,840 --> 00:05:16,160 in Vietnam. 113 00:05:16,160 --> 00:05:18,280 The indicator I'll be looking at there is 114 00:05:19,240 --> 00:05:20,400 percent and number 115 00:05:21,840 --> 00:05:25,880 targeted poor, rural, and ethnic minority women and men 116 00:05:25,880 --> 00:05:29,700 who report an improvement in capacity to save money. 117 00:05:30,840 --> 00:05:34,400 And I have the same issue that Donald's explained 118 00:05:34,400 --> 00:05:38,080 is increased gender responsive capacity 119 00:05:39,080 --> 00:05:40,560 to save and borrow money. 120 00:05:41,640 --> 00:05:44,480 They're depending here again on self-reporting. 121 00:05:44,480 --> 00:05:46,940 So you have this self-reporting bias 122 00:05:46,940 --> 00:05:50,400 or what some people call social desirability bias 123 00:05:50,400 --> 00:05:54,080 where they will have a tendency to want to say, 124 00:05:54,080 --> 00:05:56,600 yes, we've increased our capacity. 125 00:05:56,600 --> 00:05:58,600 And another point is, 126 00:05:58,600 --> 00:06:02,400 if they have the capacity to save money and borrow money, 127 00:06:02,400 --> 00:06:04,580 why not just check out their bank accounts? 128 00:06:04,580 --> 00:06:06,280 Why not just go right to the chase? 129 00:06:07,640 --> 00:06:09,740 Some people may say that's intrusive, 130 00:06:09,740 --> 00:06:11,460 but if they really wanna show 131 00:06:11,460 --> 00:06:14,140 that they've been able to save and borrow money, 132 00:06:14,140 --> 00:06:16,360 they should look at their savings accounts. 133 00:06:16,360 --> 00:06:18,800 Their ability to borrow implies the ability 134 00:06:18,800 --> 00:06:20,640 to pay back a loan on time. 135 00:06:21,520 --> 00:06:23,640 So they could also use that as an alternate way 136 00:06:23,640 --> 00:06:25,660 of measuring their capacity. 137 00:06:26,800 --> 00:06:30,920 And also as we allude to in the trailer of this podcast, 138 00:06:30,920 --> 00:06:33,880 even if their capacity is properly measured, 139 00:06:33,880 --> 00:06:36,800 and it is not here, but let's assume it is, 140 00:06:36,800 --> 00:06:38,760 how do we know it's the project training 141 00:06:38,760 --> 00:06:41,320 that was responsible for this increase 142 00:06:41,320 --> 00:06:44,520 in gender responsive capacity to save and borrow money? 143 00:06:44,520 --> 00:06:46,040 There could be another group of women 144 00:06:46,040 --> 00:06:47,820 in another part of Vietnam 145 00:06:47,820 --> 00:06:50,220 who are also doing this just fine, 146 00:06:50,220 --> 00:06:52,800 and there's no project training going on. 147 00:06:52,800 --> 00:06:55,240 They figured it out all by themselves. 148 00:06:55,240 --> 00:06:57,640 So that's what my points would be for that. 149 00:06:57,640 --> 00:07:01,400 Now, the next indicator is 1210.1. 150 00:07:01,400 --> 00:07:04,000 So that would be Donald again. 151 00:07:04,000 --> 00:07:05,280 This is where we have the outcome, 152 00:07:05,280 --> 00:07:08,220 increased gender responsive capacity 153 00:07:08,220 --> 00:07:10,120 of poor rural and ethnic minorities, 154 00:07:10,120 --> 00:07:12,520 especially women, to earn income 155 00:07:13,400 --> 00:07:16,580 from climate smart and sustainable on-farm 156 00:07:16,580 --> 00:07:19,540 and off-farm livelihoods in Vietnam. 157 00:07:20,700 --> 00:07:21,720 Donald? 158 00:07:21,720 --> 00:07:23,200 Sure, thanks, David. 159 00:07:23,200 --> 00:07:26,920 This particular indicator that supports that outcome 160 00:07:26,920 --> 00:07:28,940 is percent and normal. 161 00:07:28,940 --> 00:07:32,240 Targeted poor rural and ethnic minority women 162 00:07:32,240 --> 00:07:35,320 and men who report an improvement 163 00:07:35,320 --> 00:07:37,460 in their capacity to earn income. 164 00:07:37,460 --> 00:07:40,240 And this indicator is disaggregated by sex, 165 00:07:40,240 --> 00:07:42,920 ethnicity, age, disability status. 166 00:07:42,920 --> 00:07:45,000 Three main issues again with this indicator. 167 00:07:45,000 --> 00:07:48,820 The first one, again, same issue 168 00:07:48,820 --> 00:07:50,680 like the last two indicators. 169 00:07:50,680 --> 00:07:53,080 It's based on self-reported improvements, 170 00:07:53,080 --> 00:07:56,760 which may not reflect actual income changes. 171 00:07:57,680 --> 00:08:00,080 The second issue I have with this indicator 172 00:08:00,080 --> 00:08:01,680 is the frequency of measurement. 173 00:08:01,680 --> 00:08:05,080 It's unclear how often this measurement is taken, 174 00:08:05,080 --> 00:08:10,080 which can affect the reliability of the data being collected. 175 00:08:10,240 --> 00:08:12,060 And then there's no comparison group. 176 00:08:12,060 --> 00:08:14,380 It's difficult to attribute any improvements 177 00:08:14,380 --> 00:08:17,800 to the project, specifically if you don't have 178 00:08:17,800 --> 00:08:19,680 a comparison group. 179 00:08:21,640 --> 00:08:24,880 How this indicator would need to be improved 180 00:08:24,880 --> 00:08:28,440 to measure the impact project is, 181 00:08:28,440 --> 00:08:31,200 the first thing that would be needed to address is, 182 00:08:31,200 --> 00:08:34,400 you know, having that control group being selected. 183 00:08:34,400 --> 00:08:36,560 And that control group should be matched 184 00:08:36,560 --> 00:08:40,040 on key demographic and soco-economic variables 185 00:08:40,040 --> 00:08:45,040 to ensure compatibility and implements physical techniques 186 00:08:45,880 --> 00:08:50,880 such as propensity score matching can be used, for example, 187 00:08:50,880 --> 00:08:54,240 to select a control group that mirrors the intervention group 188 00:08:54,240 --> 00:08:56,480 as closely as possible. 189 00:08:56,480 --> 00:08:59,640 The second thing is the frequency of the data. 190 00:08:59,640 --> 00:09:02,520 Determine an optimal data collection frequency 191 00:09:02,520 --> 00:09:05,600 that balances the need for the timely data 192 00:09:05,600 --> 00:09:09,120 with the logistical constraints of data collection. 193 00:09:09,120 --> 00:09:11,360 This may involve seasonal assessment 194 00:09:11,360 --> 00:09:14,800 to account for agricultural or market cycles 195 00:09:14,800 --> 00:09:17,160 that affect income. 196 00:09:18,080 --> 00:09:20,680 And then there's income measurement. 197 00:09:20,680 --> 00:09:23,000 A comprehensive income measurement tool 198 00:09:23,000 --> 00:09:26,440 would need to capture not only the amount of income, 199 00:09:26,440 --> 00:09:29,480 but also the sources and stability of that income. 200 00:09:29,480 --> 00:09:31,400 And this tool should be sensitive enough 201 00:09:31,400 --> 00:09:33,160 to detect changes over time 202 00:09:33,160 --> 00:09:38,160 and attributable to specific project interventions. 203 00:09:38,280 --> 00:09:40,360 And the fourth thing I would suggest 204 00:09:40,360 --> 00:09:44,240 that would need to be considered for this indicator 205 00:09:44,240 --> 00:09:48,320 for improvement is link the specific activities 206 00:09:48,320 --> 00:09:51,840 to income improvement, to specify project activities, 207 00:09:51,840 --> 00:09:55,880 to identify what exactly is really working. 208 00:09:55,880 --> 00:09:58,160 So the next outcome is increased willingness 209 00:09:58,160 --> 00:10:02,640 of poor rural and ethnic minority women and men, 210 00:10:02,640 --> 00:10:06,760 duty bearers in particular, commune officers, 211 00:10:06,760 --> 00:10:11,760 to address the issue of women's unequal caregiving burden 212 00:10:13,080 --> 00:10:15,400 and economic rights in Vietnam. 213 00:10:16,400 --> 00:10:19,440 And I think Donald, you're looking at the indicator there. 214 00:10:19,440 --> 00:10:22,880 Yes, that particular indicator states 215 00:10:23,880 --> 00:10:27,120 average score of ethnic minority men and women, 216 00:10:27,120 --> 00:10:29,360 duty bearers in particular, 217 00:10:29,360 --> 00:10:34,240 commune officers who report gender equitably attitudes. 218 00:10:34,240 --> 00:10:38,240 And this is disaggregated by sex, ethnicity, age, 219 00:10:38,240 --> 00:10:41,440 disability status, type of participant. 220 00:10:42,520 --> 00:10:45,680 I have three main issues with this particular indicator. 221 00:10:45,680 --> 00:10:48,920 The first one is the subjectivity of it. 222 00:10:48,920 --> 00:10:51,600 Discard indicators relies again 223 00:10:51,600 --> 00:10:54,560 on self-reported measures of attitudes. 224 00:10:54,560 --> 00:10:57,360 And this can be highly, and in more cases 225 00:10:57,360 --> 00:10:59,800 are very highly subjective and influenced 226 00:10:59,800 --> 00:11:03,640 by social desirability biases. 227 00:11:03,640 --> 00:11:05,920 The second issue I have with this indicator 228 00:11:05,920 --> 00:11:07,960 is the lack of baseline and follow-up. 229 00:11:07,960 --> 00:11:11,280 Without the baseline measure or regular follow-up, 230 00:11:11,280 --> 00:11:13,920 it's very difficult to assess whether any changes 231 00:11:13,920 --> 00:11:18,520 in attitudes are due to the project 232 00:11:18,520 --> 00:11:22,440 or the activities coming out of the project. 233 00:11:22,440 --> 00:11:24,520 And there's no actionable outcome. 234 00:11:24,520 --> 00:11:27,680 Porting an attitude does not necessarily translate 235 00:11:27,680 --> 00:11:32,680 into actionable change or improvement in gender equality. 236 00:11:33,560 --> 00:11:35,840 Let me just ask you, Donald, that's correct. 237 00:11:35,840 --> 00:11:38,760 But in this outcome, to be fair to Care Canada, 238 00:11:38,760 --> 00:11:41,120 they just wanna make sure that the attitude 239 00:11:41,120 --> 00:11:44,840 has changed in the right direction, increased willingness. 240 00:11:44,840 --> 00:11:47,720 So you mentioned lack of frequency 241 00:11:47,720 --> 00:11:49,800 in measuring this attitude change. 242 00:11:49,800 --> 00:11:52,280 If we have a chance, we can look at the performance 243 00:11:52,280 --> 00:11:53,760 measurement framework. 244 00:11:53,760 --> 00:11:57,880 What exactly is the frequency they measure 245 00:11:57,880 --> 00:11:58,960 this attitude change? 246 00:11:58,960 --> 00:12:02,000 Does it say in the performance measurement framework? 247 00:12:02,000 --> 00:12:04,200 I think they mention outcome harvesting 248 00:12:04,200 --> 00:12:06,440 and it might be annual reporting. 249 00:12:06,440 --> 00:12:08,360 Let's check to make sure. 250 00:12:08,360 --> 00:12:11,000 I'm quite sure the frequency is not enough, 251 00:12:11,000 --> 00:12:12,120 but here it is. 252 00:12:13,040 --> 00:12:13,880 Yeah, you're right. 253 00:12:13,880 --> 00:12:17,120 It's outcome harvesting and it's only done annually. 254 00:12:17,120 --> 00:12:19,560 So that means you get an attitude change, 255 00:12:19,560 --> 00:12:21,560 you get it in the right direction, 256 00:12:21,560 --> 00:12:23,600 but they only measure it once a year. 257 00:12:23,600 --> 00:12:26,440 So there's all sorts of different things 258 00:12:26,440 --> 00:12:28,640 that could be going on that could change their attitude 259 00:12:28,640 --> 00:12:30,840 that's got nothing to do with the project. 260 00:12:30,840 --> 00:12:33,840 Yeah, so, yeah, that's a good point. 261 00:12:33,840 --> 00:12:35,080 Yeah. 262 00:12:35,080 --> 00:12:39,680 Yeah, so suggestions on how this could be improved 263 00:12:39,680 --> 00:12:42,840 is definitely, first of all, 264 00:12:42,840 --> 00:12:45,760 addressing the regular assessments, right? 265 00:12:45,760 --> 00:12:48,000 Implement more pre-project surveys 266 00:12:48,000 --> 00:12:49,560 to establish a baseline, 267 00:12:49,560 --> 00:12:52,680 conduct regular follow-up surveys 268 00:12:52,680 --> 00:12:54,280 to track changes over time. 269 00:12:54,280 --> 00:12:57,080 They could also use a comparison group 270 00:12:57,080 --> 00:12:59,440 where people that are not in the project, 271 00:12:59,440 --> 00:13:04,440 that are not doing anything to any of these women 272 00:13:04,520 --> 00:13:06,400 or this target group to change their attitudes, 273 00:13:06,400 --> 00:13:09,240 they could also measure their attitudes 274 00:13:09,240 --> 00:13:13,840 on the same variable, which is increased willingness 275 00:13:13,840 --> 00:13:16,800 to particularly these commune officers 276 00:13:16,800 --> 00:13:20,160 to address the issue of women's unequal caregiving burden. 277 00:13:20,160 --> 00:13:22,760 And they could go to other districts in Vietnam 278 00:13:22,760 --> 00:13:25,000 and ask commune officers there 279 00:13:25,000 --> 00:13:26,960 where Care Canada is not working 280 00:13:26,960 --> 00:13:29,480 and trying to change their attitudes. 281 00:13:29,480 --> 00:13:32,680 Their attitudes could also be going in the right direction. 282 00:13:32,680 --> 00:13:34,120 Right, right. 283 00:13:34,120 --> 00:13:36,840 Yeah, that's a key point. 284 00:13:36,840 --> 00:13:39,800 And one of the methodologies I have here to note 285 00:13:39,800 --> 00:13:44,320 that would expand on the only tactical assessments 286 00:13:44,320 --> 00:13:47,040 of this particular indicator is, 287 00:13:47,040 --> 00:13:49,480 one, they can also implement 288 00:13:49,480 --> 00:13:53,160 longitudinal study design methodology 289 00:13:53,160 --> 00:13:56,680 that measures attitude over multiple time points. 290 00:13:56,680 --> 00:13:59,640 And this would allow for assessments of changes 291 00:13:59,640 --> 00:14:02,240 in attitudes and the sustainability 292 00:14:02,240 --> 00:14:05,680 of these changes over time. 293 00:14:05,680 --> 00:14:07,000 Another thing that they can do 294 00:14:07,000 --> 00:14:08,960 is implement a mixed method approach 295 00:14:08,960 --> 00:14:12,080 where they incorporate qualitative methods 296 00:14:12,080 --> 00:14:13,880 such as focus group discussions 297 00:14:13,880 --> 00:14:16,080 or in-depth interviews. 298 00:14:16,080 --> 00:14:17,520 And that is complemented 299 00:14:17,520 --> 00:14:20,160 with the quantitative self-reporting data. 300 00:14:20,160 --> 00:14:22,520 And that would provide a bit more context 301 00:14:22,520 --> 00:14:24,400 to the attitudes expressed 302 00:14:24,400 --> 00:14:26,200 and the reason behind any changes 303 00:14:26,200 --> 00:14:29,800 or lack of any changes thereof. 304 00:14:29,800 --> 00:14:30,640 Yeah, that's right. 305 00:14:30,640 --> 00:14:33,080 There's a common problem. 306 00:14:34,440 --> 00:14:37,000 Our focus groups are used just to measure 307 00:14:37,000 --> 00:14:38,920 the outcome indicator 308 00:14:38,920 --> 00:14:42,160 rather than measuring the outcome indicator first 309 00:14:42,160 --> 00:14:44,320 and then seeing that the outcome indicator 310 00:14:44,320 --> 00:14:46,800 is not moving in the right direction. 311 00:14:46,800 --> 00:14:49,200 And then, and only then, 312 00:14:49,200 --> 00:14:50,320 should they have a focus group 313 00:14:50,320 --> 00:14:51,400 where they ask the question, 314 00:14:51,400 --> 00:14:55,240 hello, why did your increased willingness not go up? 315 00:14:55,240 --> 00:14:57,840 And then you hold a focus group to find out. 316 00:14:57,840 --> 00:15:01,360 But often focus groups are abused, misused, 317 00:15:01,360 --> 00:15:04,160 and they only use them like outcome harvesting 318 00:15:04,160 --> 00:15:08,000 to try to measure the outcome indicator in the first place. 319 00:15:08,000 --> 00:15:08,960 Right, correct. 320 00:15:08,960 --> 00:15:13,120 Okay, so let's move on to the next outcome, 321 00:15:13,120 --> 00:15:17,560 which is increased ability of ethnic minority women 322 00:15:17,560 --> 00:15:20,120 to meaningfully participate 323 00:15:20,120 --> 00:15:24,160 and make decisions related to economic activities 324 00:15:24,160 --> 00:15:28,120 at the household level and in their livelihoods. 325 00:15:28,120 --> 00:15:29,600 And Mark, I think you're gonna be dealing 326 00:15:29,600 --> 00:15:31,800 with 1120.1 indicator. 327 00:15:33,120 --> 00:15:34,840 Well, yes, I'm reading. 328 00:15:34,840 --> 00:15:38,680 Percentage and number of ethnic minority women 329 00:15:38,680 --> 00:15:41,000 who have meaningfully participated 330 00:15:41,000 --> 00:15:43,600 in economic decision-making at the household. 331 00:15:44,920 --> 00:15:49,680 Yes, I think general increased ability participation 332 00:15:49,680 --> 00:15:54,600 are difficult terms to evaluate and also to monitor 333 00:15:54,600 --> 00:15:58,040 because meaningfully participation, 334 00:15:58,040 --> 00:16:00,320 what does it mean in detail? 335 00:16:00,320 --> 00:16:01,680 Does it mean, 336 00:16:01,680 --> 00:16:03,920 that they have not the chance to participate 337 00:16:03,920 --> 00:16:05,640 in the household before? 338 00:16:05,640 --> 00:16:07,480 Is there, because of what? 339 00:16:07,480 --> 00:16:11,480 Because of cultural aspects or social aspects. 340 00:16:11,480 --> 00:16:14,440 And now if they are able to do this, 341 00:16:15,960 --> 00:16:19,080 it's, we can maybe, this is for me also a question 342 00:16:19,080 --> 00:16:22,080 of a qualitative approach necessary, 343 00:16:22,080 --> 00:16:24,480 a question of qualitative measurement. 344 00:16:24,480 --> 00:16:28,640 And general decision-making, 345 00:16:28,640 --> 00:16:31,960 you have to go back and ask them, 346 00:16:31,960 --> 00:16:35,240 maybe in interview situations, 347 00:16:35,240 --> 00:16:38,880 what has improved your participation at the household? 348 00:16:38,880 --> 00:16:41,520 Is the husband now allowing you, 349 00:16:41,520 --> 00:16:45,720 or is your role in the family has been stronger than before 350 00:16:47,240 --> 00:16:49,320 to explain this indicator? 351 00:16:49,320 --> 00:16:51,400 And the other way is, 352 00:16:52,280 --> 00:16:54,520 sometimes the stories of change can also help 353 00:16:54,520 --> 00:16:57,480 to get to the point where 354 00:16:57,480 --> 00:17:00,880 you get to verify 355 00:17:00,880 --> 00:17:05,680 and to find out what is now the improvement of the situation. 356 00:17:05,680 --> 00:17:08,280 And the second one is similar, 357 00:17:08,280 --> 00:17:12,160 it's the percentage of numbers of ethnic minority women 358 00:17:12,160 --> 00:17:15,840 who report participating and voicing out their concerns 359 00:17:15,840 --> 00:17:19,520 about pay economic activities to local authorities 360 00:17:19,520 --> 00:17:20,800 and DTPers here. 361 00:17:22,160 --> 00:17:24,240 Here we have a similar situation. 362 00:17:24,240 --> 00:17:26,200 There's one aspect, 363 00:17:26,200 --> 00:17:27,880 it was written women who report, 364 00:17:27,880 --> 00:17:30,680 they're reporting their participation 365 00:17:30,680 --> 00:17:33,480 and voicing out their concern. 366 00:17:33,480 --> 00:17:37,840 So it's a self-reporting or it's a reporting that happens 367 00:17:37,840 --> 00:17:40,720 via a survey, I don't know. 368 00:17:40,720 --> 00:17:43,760 But for me, when I look at this thing, 369 00:17:43,760 --> 00:17:46,800 what local authorities are they, 370 00:17:46,800 --> 00:17:51,800 what type of DTPers are these who are concerning about this? 371 00:17:51,800 --> 00:17:54,480 Are they from the district, from the communities? 372 00:17:56,120 --> 00:18:01,120 And how can they see improvement to the situation before? 373 00:18:02,240 --> 00:18:07,240 It's a little bit tricky to use such indicators 374 00:18:07,240 --> 00:18:11,200 who are, you can use your qualitative approaches 375 00:18:11,200 --> 00:18:12,880 to by the end of the day, 376 00:18:12,880 --> 00:18:17,880 it's to see the change from the baseline to the target. 377 00:18:17,880 --> 00:18:20,240 It's sometimes not so, we will not know 378 00:18:20,240 --> 00:18:25,240 it's not so visible and also hard to measure it, yeah. 379 00:18:26,320 --> 00:18:28,520 Right, and if you look at the outcome, 380 00:18:28,520 --> 00:18:31,800 it says we've bolded it for, 381 00:18:31,800 --> 00:18:33,240 if anybody's interested in getting 382 00:18:33,240 --> 00:18:34,920 the performance measurement framework, 383 00:18:34,920 --> 00:18:35,760 you just email me, 384 00:18:35,760 --> 00:18:40,760 it says increased ability to meaningfully participate. 385 00:18:41,120 --> 00:18:44,920 Now you could argue, you can measure that quantitatively. 386 00:18:44,920 --> 00:18:49,080 You could measure their ability to negotiate or whatever, 387 00:18:49,080 --> 00:18:52,200 you could test them, you could do some sort of assessment 388 00:18:52,200 --> 00:18:54,760 before, during, and after the training 389 00:18:54,760 --> 00:18:56,360 instead of self-reporting. 390 00:18:56,360 --> 00:18:59,120 So that's a really quick way of arguing 391 00:18:59,120 --> 00:19:02,600 you could actually objectively measure increased ability 392 00:19:02,600 --> 00:19:04,400 based on some sort of training 393 00:19:04,400 --> 00:19:06,600 where they're training them how to negotiate 394 00:19:06,600 --> 00:19:09,720 for better participation in the household. 395 00:19:09,720 --> 00:19:13,480 And that would be good enough because the outcome is clear. 396 00:19:13,480 --> 00:19:15,800 It just says increased ability. 397 00:19:15,800 --> 00:19:17,600 After that, good luck. 398 00:19:17,600 --> 00:19:20,640 We don't know if that's gonna lead to greater levels 399 00:19:20,640 --> 00:19:23,240 of satisfaction for these women in the household, 400 00:19:23,240 --> 00:19:26,440 but at least that piece, that short-term piece 401 00:19:26,440 --> 00:19:29,480 in the immediate outcome of increased ability, 402 00:19:29,480 --> 00:19:30,640 they could measure it. 403 00:19:30,640 --> 00:19:33,360 But they're going straight to stories of change. 404 00:19:33,360 --> 00:19:35,120 And to be fair to Care Canada, 405 00:19:35,120 --> 00:19:37,760 of course they're doing what Mark has explained, 406 00:19:37,760 --> 00:19:41,840 which is trying to find out how to measure this level 407 00:19:41,840 --> 00:19:44,800 of participation in a household, which is very difficult. 408 00:19:44,800 --> 00:19:46,000 It's intrusive. 409 00:19:46,000 --> 00:19:50,280 So we understand stories of change may be a way of doing it, 410 00:19:50,280 --> 00:19:53,360 but you've forgotten your outcome in the first place, 411 00:19:53,360 --> 00:19:55,840 which is increased ability. 412 00:19:55,840 --> 00:19:58,240 So they've kind of missed, in my opinion, 413 00:19:58,240 --> 00:20:00,320 that first step where they could actually measure 414 00:20:00,320 --> 00:20:03,640 their ability using some sort of psychometric test 415 00:20:03,640 --> 00:20:06,120 of their skills to be able to negotiate 416 00:20:06,920 --> 00:20:10,720 based on training they have received in the project. 417 00:20:10,720 --> 00:20:14,280 And then you could also do this with another group of women 418 00:20:14,280 --> 00:20:16,400 who are not even in the project 419 00:20:16,400 --> 00:20:18,240 or in some other part of the country, 420 00:20:18,240 --> 00:20:19,640 and that would be good enough. 421 00:20:19,640 --> 00:20:22,800 They could show that ability has actually increased. 422 00:20:24,360 --> 00:20:27,520 The other question is also when you talk about household, 423 00:20:27,520 --> 00:20:31,840 then you have also to ask other members of the household, 424 00:20:31,840 --> 00:20:35,240 the husband or children or people who live in this household 425 00:20:35,240 --> 00:20:39,960 if the economic decision-making has been improved 426 00:20:39,960 --> 00:20:41,240 because of the project. 427 00:20:41,240 --> 00:20:44,400 So you need more voices to understand 428 00:20:44,400 --> 00:20:45,960 what is really the improvement. 429 00:20:47,760 --> 00:20:50,800 Even though they've made it clear in that second indicator, 430 00:20:50,800 --> 00:20:53,080 they only are interested in the women, 431 00:20:53,080 --> 00:20:57,640 but it's a good point is that it's a more complex dynamic 432 00:20:57,640 --> 00:21:00,280 in the decision-making within the household. 433 00:21:00,280 --> 00:21:01,800 But they say right in the, 434 00:21:01,800 --> 00:21:04,400 at least I give them credit for specifying, 435 00:21:04,400 --> 00:21:06,800 it's the ethnic minority women. 436 00:21:06,800 --> 00:21:09,040 They want to increase their participation. 437 00:21:09,040 --> 00:21:12,880 But you're right, it may depend on the other members 438 00:21:12,880 --> 00:21:14,960 in the household, absolutely, yeah. 439 00:21:14,960 --> 00:21:18,480 Okay, so we're gonna move on to the next outcome, 440 00:21:18,480 --> 00:21:21,520 which is in the performance measure framework, 441 00:21:21,520 --> 00:21:24,440 increased capacity of civil society organizations, 442 00:21:24,440 --> 00:21:26,840 including women's rights organizations, 443 00:21:26,840 --> 00:21:30,000 private sector companies and media organizations 444 00:21:30,000 --> 00:21:33,400 to promote poor rural and ethnic minority women's 445 00:21:33,400 --> 00:21:35,200 economic rights in Vietnam. 446 00:21:35,200 --> 00:21:40,200 And the indicator, number of civil society organizations, 447 00:21:40,200 --> 00:21:42,560 media organizations and network members, 448 00:21:42,560 --> 00:21:46,120 private sector companies, who report an improvement 449 00:21:46,120 --> 00:21:50,480 in their perception, opinion related to 450 00:21:50,480 --> 00:21:53,000 ethnic minority women's economic rights. 451 00:21:54,440 --> 00:21:58,680 So again, outcome is clear, increased capacity. 452 00:21:58,680 --> 00:22:02,400 So that means we should be looking at their ability, 453 00:22:02,400 --> 00:22:07,400 increased ability to actually advocate for these women 454 00:22:07,880 --> 00:22:10,000 regarding their economic rights. 455 00:22:10,000 --> 00:22:13,640 So there's a huge bias in here because these organizations 456 00:22:13,640 --> 00:22:17,080 are paid for by Care Canada, by the project 457 00:22:17,080 --> 00:22:19,120 to advocate for these women. 458 00:22:19,120 --> 00:22:22,120 So they're expected to advocate on their behalf. 459 00:22:22,120 --> 00:22:26,320 So if you ask them, have you improved your ability 460 00:22:26,320 --> 00:22:29,440 or your perception or opinion related to 461 00:22:29,440 --> 00:22:32,640 women's economic rights that you happen to be paid for 462 00:22:32,640 --> 00:22:34,520 by the project to advocate? 463 00:22:34,520 --> 00:22:36,960 Of course, they're gonna say yes. 464 00:22:36,960 --> 00:22:38,640 So this is a huge bias. 465 00:22:38,640 --> 00:22:40,840 And again, they missed the point of the outcome, 466 00:22:40,840 --> 00:22:42,440 which is they should be just measuring 467 00:22:42,440 --> 00:22:45,080 their increased capacity to advocate. 468 00:22:45,080 --> 00:22:47,640 There's all sorts of projects where they train people 469 00:22:47,640 --> 00:22:49,920 on how to advocate for another group, 470 00:22:49,920 --> 00:22:52,720 which is what's going on here in this outcome. 471 00:22:52,720 --> 00:22:55,000 So that's the major problem I see with that. 472 00:22:55,000 --> 00:22:57,760 And I pointed it out, why not objectively measure 473 00:22:57,760 --> 00:23:00,840 and also the other point is it's not the institutions, 474 00:23:00,840 --> 00:23:03,240 it's the human beings inside of them 475 00:23:03,240 --> 00:23:04,680 that we need to look at. 476 00:23:04,680 --> 00:23:07,280 We need to measure their technical capacity 477 00:23:07,280 --> 00:23:09,040 to advocate for others, right? 478 00:23:09,040 --> 00:23:12,960 So it's not just the number, that's the real problem here. 479 00:23:12,960 --> 00:23:15,480 The indicator says the number of organizations. 480 00:23:15,480 --> 00:23:17,120 So you just tick a box off. 481 00:23:17,120 --> 00:23:19,400 How do you tick that box off that says 482 00:23:19,400 --> 00:23:22,680 this civil society organization in Vietnam 483 00:23:22,680 --> 00:23:27,240 has met all our criteria of equity 484 00:23:27,240 --> 00:23:31,760 of adequately and increasingly in better ways 485 00:23:31,760 --> 00:23:33,800 advocating for these women? 486 00:23:33,800 --> 00:23:34,640 How do we know? 487 00:23:35,680 --> 00:23:40,160 So what we need to do instead is go inside that organization 488 00:23:40,160 --> 00:23:43,120 and test a group of the individuals 489 00:23:43,120 --> 00:23:45,600 who are expected to advocate for those women. 490 00:23:45,600 --> 00:23:48,240 I think it could be duty bearers, I'm not sure. 491 00:23:49,280 --> 00:23:51,440 And make sure that their technical ability 492 00:23:51,440 --> 00:23:52,920 to advocate has gone up 493 00:23:52,920 --> 00:23:55,880 and maybe even their attitudes have gone up. 494 00:23:55,880 --> 00:23:58,160 And then the other problem of course is 495 00:23:58,160 --> 00:24:01,480 even if they have gone up, there may be another group 496 00:24:01,480 --> 00:24:04,320 of civil society organizations that are doing 497 00:24:04,320 --> 00:24:07,560 just as good a job or even better job 498 00:24:07,560 --> 00:24:11,320 at advocating for ethnic minority women's economic rights 499 00:24:11,320 --> 00:24:13,000 in some other part of the country 500 00:24:13,000 --> 00:24:16,400 where Care Canada is not involved at all. 501 00:24:16,400 --> 00:24:19,240 So that's the problem with that indicator there. 502 00:24:20,480 --> 00:24:23,000 Now we'll move on to the next outcome 503 00:24:23,000 --> 00:24:26,640 which is increased participation by poor 504 00:24:26,640 --> 00:24:31,160 and ethnic minority women in paid economic activities. 505 00:24:31,160 --> 00:24:34,120 I think Mark you're doing this indicator here. 506 00:24:34,120 --> 00:24:37,920 Yes, percentage and number targeted poor, rural 507 00:24:37,920 --> 00:24:40,480 and ethnic minority women and men 508 00:24:40,480 --> 00:24:45,480 who report access to and control over economic resources 509 00:24:46,000 --> 00:24:50,760 desegregated by sex, ethnicity, age, disability, 510 00:24:50,760 --> 00:24:53,640 status and commune. 511 00:24:53,640 --> 00:24:55,880 Well, here we have again the same question 512 00:24:55,880 --> 00:24:59,480 like we had before, it's about self-reporting. 513 00:24:59,480 --> 00:25:04,120 Because you want to know this ethnic minority women and men 514 00:25:04,120 --> 00:25:06,760 who are reporting their access and control 515 00:25:06,760 --> 00:25:08,840 of our economic resources. 516 00:25:08,840 --> 00:25:13,840 And it's somehow difficult to value it. 517 00:25:14,480 --> 00:25:17,840 The measurement is somehow you can maybe check them 518 00:25:17,840 --> 00:25:21,160 in a questionnaire to see what economic resources 519 00:25:21,160 --> 00:25:25,040 they had before the project, what do they have now? 520 00:25:25,040 --> 00:25:29,800 So you can see the change in the impact 521 00:25:29,800 --> 00:25:34,400 but there's always some bias about self-reporting 522 00:25:34,400 --> 00:25:37,240 but it's sometimes not bad. 523 00:25:37,240 --> 00:25:39,760 It can also be if you do it in a way with interviews 524 00:25:39,760 --> 00:25:44,040 and focus group discussions to try to get the view 525 00:25:44,040 --> 00:25:47,720 of the people and to put it in numbers 526 00:25:47,720 --> 00:25:51,280 and figures it's time to see the track changes 527 00:25:51,280 --> 00:25:55,240 that it's a little bit not appropriate I think. 528 00:25:57,240 --> 00:25:59,120 You can maybe track their incomes. 529 00:25:59,120 --> 00:26:03,520 You can track their incomes against a non-project group 530 00:26:03,520 --> 00:26:06,320 of women and men to see if they have really increased 531 00:26:06,320 --> 00:26:07,840 their incomes before. 532 00:26:09,760 --> 00:26:11,160 Exactly, yeah. 533 00:26:11,160 --> 00:26:13,200 Now to be fair to Care Canada, 534 00:26:13,200 --> 00:26:15,400 the outcome is increased participation 535 00:26:15,400 --> 00:26:20,240 but that's difficult to measure but they've said 536 00:26:20,240 --> 00:26:22,760 they're gonna claim that their project is going 537 00:26:22,760 --> 00:26:26,760 to increase the levels of participation for these women 538 00:26:26,760 --> 00:26:28,320 and they're gonna train them in some way 539 00:26:28,320 --> 00:26:31,320 that this is going to happen through a variety 540 00:26:31,320 --> 00:26:35,560 of project services that are described in part one. 541 00:26:35,560 --> 00:26:37,200 But yes, it's difficult but yeah, 542 00:26:37,200 --> 00:26:40,360 you could just cut to the chase and figure out 543 00:26:40,360 --> 00:26:44,480 because you want to achieve ultimately increased incomes, 544 00:26:44,480 --> 00:26:48,640 economic resources for the family although we can't assume 545 00:26:48,640 --> 00:26:51,200 that all that money, increased income goes 546 00:26:51,200 --> 00:26:56,120 into the women's pocket may still be held by the husband 547 00:26:56,120 --> 00:26:57,680 and that is understandable. 548 00:26:57,680 --> 00:26:59,280 That's why they talk about empowerment 549 00:26:59,280 --> 00:27:01,440 of the women specifically. 550 00:27:01,440 --> 00:27:05,360 So it's difficult but I think we've noted it's a challenge 551 00:27:05,360 --> 00:27:07,880 for sure in measuring increased participation. 552 00:27:07,880 --> 00:27:09,720 Absolutely, yeah. 553 00:27:09,720 --> 00:27:11,640 Yeah, it's a general hard aspect. 554 00:27:11,640 --> 00:27:15,040 Participation is generally very difficult in developing 555 00:27:15,040 --> 00:27:19,440 projects so very often you use quantitative approaches 556 00:27:19,440 --> 00:27:23,360 because quantitative, you can work with a questionnaire 557 00:27:23,360 --> 00:27:26,720 and make a survey if you have a reliable sample 558 00:27:26,720 --> 00:27:30,840 but it's not always giving you the right view. 559 00:27:30,840 --> 00:27:32,720 Right, okay good. 560 00:27:32,720 --> 00:27:35,760 So the next outcome is enhanced promotion 561 00:27:35,760 --> 00:27:38,360 of economic rights for poor rural women, 562 00:27:38,360 --> 00:27:41,520 especially ethnic minority women in Vietnam 563 00:27:41,520 --> 00:27:45,960 and the indicator I'm looking at here is average total number 564 00:27:45,960 --> 00:27:49,440 and proportion of daily hours spent 565 00:27:49,440 --> 00:27:52,240 on unpaid domestic and care work. 566 00:27:52,240 --> 00:27:56,600 Now this is an interesting one because they are trying 567 00:27:56,600 --> 00:28:00,840 to have these women, part of the project actually delivers 568 00:28:00,840 --> 00:28:05,360 childcare saving options and they don't describe the options 569 00:28:05,360 --> 00:28:09,640 but we can assume that just by delivering that service 570 00:28:09,640 --> 00:28:11,880 in the project which is very important, 571 00:28:11,880 --> 00:28:15,640 it will reduce the hours that these women spend 572 00:28:15,640 --> 00:28:18,120 on unpaid domestic care work just 573 00:28:18,120 --> 00:28:21,600 by providing these alternate options for them to choose, 574 00:28:21,600 --> 00:28:22,440 which is fine. 575 00:28:22,440 --> 00:28:26,560 Two problems again, the one is even if that happens, 576 00:28:26,560 --> 00:28:30,120 we need to see a comparison group where there's another group 577 00:28:30,120 --> 00:28:34,640 of women who don't receive these alternate options 578 00:28:34,640 --> 00:28:38,360 of other people taking care of their children for example 579 00:28:38,360 --> 00:28:39,840 in this project. 580 00:28:39,840 --> 00:28:42,320 In another part of the country where another group 581 00:28:42,320 --> 00:28:45,440 of women get together for example and set 582 00:28:45,440 --> 00:28:48,520 up their own community childcare center 583 00:28:48,520 --> 00:28:51,440 where one woman says all your children can come 584 00:28:51,440 --> 00:28:53,680 to my house this day and the rest 585 00:28:53,680 --> 00:28:56,920 of you can do what you have to do during the day and I'll take 586 00:28:56,920 --> 00:28:59,080 and they do that on a rotational basis. 587 00:28:59,080 --> 00:29:02,200 So that's the first problem. 588 00:29:02,200 --> 00:29:05,920 The other problem is when you look at the outcome, again, 589 00:29:05,920 --> 00:29:11,600 they're looking at enhanced promotion of economic rights 590 00:29:11,600 --> 00:29:13,920 for poor rural women. 591 00:29:13,920 --> 00:29:18,720 So this particular indicator isn't promoting their rights 592 00:29:18,720 --> 00:29:23,880 specifically so enhanced promotion is not really being 593 00:29:23,880 --> 00:29:26,680 measured here directly. 594 00:29:26,680 --> 00:29:29,280 So that's the only way I could look at this is saying 595 00:29:29,280 --> 00:29:31,800 that is there another group out there 596 00:29:31,800 --> 00:29:35,240 that are also promoting the economic rights of these women 597 00:29:35,240 --> 00:29:38,800 in a way that's having the same benefits. 598 00:29:38,800 --> 00:29:40,520 So there's no comparison group. 599 00:29:40,520 --> 00:29:44,640 So that's the problem there with that indicator. 600 00:29:44,640 --> 00:29:47,320 Now to be fair to them, they have other indicators 601 00:29:47,320 --> 00:29:51,360 where they are trying to measure enhanced promotion 602 00:29:51,360 --> 00:29:54,240 but we're not going to discuss that right now 603 00:29:54,240 --> 00:29:57,760 because we have limited time to cover all the indicators 604 00:29:57,760 --> 00:30:02,640 but the summary is there for people to look at as to why all 605 00:30:02,640 --> 00:30:05,600 of the outcome indicators do fall into one of those five 606 00:30:05,600 --> 00:30:08,200 problem areas that we're discussing. 607 00:30:08,200 --> 00:30:12,240 So the next one, the next outcome is improve economic 608 00:30:12,240 --> 00:30:16,000 well-being for poor rural women, especially ethnic minority 609 00:30:16,000 --> 00:30:19,080 women in Vietnam and the indicator there is percent 610 00:30:19,080 --> 00:30:23,040 and number of poor rural and ethnic minority women and men 611 00:30:23,040 --> 00:30:26,960 who report an improvement in their economic well-being. 612 00:30:26,960 --> 00:30:29,480 And it's the same issue we've talked about before, 613 00:30:29,480 --> 00:30:32,840 which is instead of having them self-report 614 00:30:32,840 --> 00:30:34,320 on their economic well-being, 615 00:30:34,320 --> 00:30:36,080 why not just measure their incomes? 616 00:30:37,240 --> 00:30:40,560 And more importantly, do it compared to another group 617 00:30:40,560 --> 00:30:43,600 of women in another part of Vietnam 618 00:30:43,600 --> 00:30:46,160 that aren't receiving all these project services 619 00:30:46,160 --> 00:30:50,880 and track their incomes before, during and after the project 620 00:30:50,880 --> 00:30:54,400 and see if there is a significant increase 621 00:30:54,400 --> 00:30:57,440 or greater increase in incomes for the project group. 622 00:30:57,440 --> 00:30:58,920 So now that we've done that, 623 00:30:58,920 --> 00:31:01,560 we've covered all nine outcomes of the project 624 00:31:01,560 --> 00:31:05,040 and picked at least nine outcome indicators. 625 00:31:05,040 --> 00:31:07,240 Let's, I think Mark, thank you. 626 00:31:07,240 --> 00:31:11,120 You've already covered the other indicator for that outcome, 627 00:31:11,120 --> 00:31:13,440 but we'll just do one more. 628 00:31:14,280 --> 00:31:17,600 And that is, we'll go back to increased entrepreneurship 629 00:31:17,600 --> 00:31:21,120 capacity of women-led grassroots social enterprises 630 00:31:21,120 --> 00:31:24,320 to pursue economic opportunities in Vietnam. 631 00:31:24,320 --> 00:31:27,600 And we'll look at another indicator. 632 00:31:27,600 --> 00:31:30,440 Donald's gonna look at 1230.2. 633 00:31:31,400 --> 00:31:32,920 Yes, thanks David. 634 00:31:32,920 --> 00:31:35,960 1230.2, this indicator states, 635 00:31:35,960 --> 00:31:39,440 percent a number of ethnic minority women-led 636 00:31:39,440 --> 00:31:43,320 grassroots enterprise members who report a change. 637 00:31:43,320 --> 00:31:45,640 And here they're using a five-point scale 638 00:31:46,640 --> 00:31:51,360 in their confidence level of their entrepreneurship capacity. 639 00:31:51,360 --> 00:31:56,360 This has the same issues as the 1230.1 640 00:31:58,440 --> 00:31:59,720 that we discussed, 641 00:31:59,720 --> 00:32:03,600 or issues that relate to increased capacity, 642 00:32:03,600 --> 00:32:05,560 how that's being defined. 643 00:32:07,760 --> 00:32:10,440 Issues with self-reporting, 644 00:32:10,440 --> 00:32:13,600 the reliance of it being self-reporting. 645 00:32:13,600 --> 00:32:17,040 And the major issue I think that I have with this indicator 646 00:32:17,920 --> 00:32:20,960 is confidence assessment. 647 00:32:20,960 --> 00:32:24,360 To measure confidence and to improve on this indicator, 648 00:32:24,360 --> 00:32:26,920 they should probably use a psychrometrically 649 00:32:26,920 --> 00:32:30,320 sound instrument that would differentiate 650 00:32:30,320 --> 00:32:34,800 between perceived self-efficacy and overconfidence. 651 00:32:34,800 --> 00:32:39,080 And these often do not correlate with actual competence. 652 00:32:39,080 --> 00:32:42,320 So there are some clear technical issues 653 00:32:42,320 --> 00:32:45,120 with this particular indicator. 654 00:32:45,120 --> 00:32:48,720 And to improve it, those issues will be need to improve. 655 00:32:48,720 --> 00:32:51,800 Define what is capacity. 656 00:32:51,800 --> 00:32:54,360 Is it financial, market assessment, 657 00:32:54,360 --> 00:32:56,520 operational, and so forth. 658 00:32:56,520 --> 00:32:59,440 Address the issue with confidence, competence, 659 00:32:59,440 --> 00:33:01,320 and that would clear up the major issues 660 00:33:01,320 --> 00:33:03,440 with this particular indicator. 661 00:33:03,440 --> 00:33:07,360 Before we make a decision on this performance measurement 662 00:33:07,360 --> 00:33:10,800 framework and its ability to support Care Canada 663 00:33:10,800 --> 00:33:14,720 in its claim that its project outcomes have been achieved 664 00:33:14,720 --> 00:33:16,480 by its project services, 665 00:33:16,480 --> 00:33:18,880 I'm gonna go again to the last indicator 666 00:33:18,880 --> 00:33:19,920 on the last outcome, 667 00:33:21,480 --> 00:33:23,680 which is improve economic well-being 668 00:33:23,680 --> 00:33:24,840 for poor rural women, 669 00:33:24,840 --> 00:33:27,320 especially ethnic minority women in Vietnam. 670 00:33:27,320 --> 00:33:28,920 They have one other indicator, 671 00:33:28,920 --> 00:33:32,840 which was level of improvement of economic rights 672 00:33:32,840 --> 00:33:36,480 as perceived by ethnic minority women. 673 00:33:36,480 --> 00:33:37,320 And that's fine. 674 00:33:37,320 --> 00:33:39,120 It's a good way to look at, 675 00:33:39,120 --> 00:33:40,960 measure their perception levels 676 00:33:40,960 --> 00:33:42,680 to see if their perception levels 677 00:33:42,680 --> 00:33:44,800 of how they're doing economically, 678 00:33:44,800 --> 00:33:46,880 it's obviously subjective, 679 00:33:46,880 --> 00:33:48,320 but there's nothing wrong with that. 680 00:33:48,320 --> 00:33:49,920 It's not a bad indicator. 681 00:33:49,920 --> 00:33:54,000 But again, the problem is there's no comparison group. 682 00:33:54,000 --> 00:33:56,320 So we need to get another group of women 683 00:33:56,320 --> 00:33:57,920 outside of the project, 684 00:33:57,920 --> 00:34:00,480 some other part of the country in Vietnam, 685 00:34:00,480 --> 00:34:03,320 and measure also at the same time 686 00:34:03,320 --> 00:34:07,600 their perceived levels of economic well-being 687 00:34:07,600 --> 00:34:09,480 where the project's not operating. 688 00:34:09,480 --> 00:34:12,560 And if this project wants to claim 689 00:34:12,560 --> 00:34:15,400 that they're more successful, then they have to show 690 00:34:15,400 --> 00:34:19,400 that their levels of perceived economic well-being 691 00:34:19,400 --> 00:34:21,400 have increased at a greater rate 692 00:34:21,400 --> 00:34:25,920 or are significantly greater than the non-project group. 693 00:34:25,920 --> 00:34:28,240 So now that that's done, 694 00:34:28,240 --> 00:34:29,720 what do you think guys, 695 00:34:29,720 --> 00:34:32,960 after we've reviewed this performance measurement framework, 696 00:34:32,960 --> 00:34:36,320 do you think that Care Canada can make the claim 697 00:34:36,320 --> 00:34:40,160 that its project services are achieving its project outcomes 698 00:34:40,160 --> 00:34:45,160 at an expense of $3.4 million to the Canadian taxpayer? 699 00:34:46,160 --> 00:34:47,000 Donald? 700 00:34:48,960 --> 00:34:53,160 Based on the analysis that we just provided 701 00:34:53,160 --> 00:34:57,560 and the limitations that we identified with the indicators, 702 00:34:57,560 --> 00:35:00,520 I think it would be extremely difficult for us 703 00:35:00,520 --> 00:35:04,360 to conclusively claim, or Care Canada 704 00:35:04,360 --> 00:35:06,120 to conclusively claim that the program 705 00:35:06,120 --> 00:35:08,440 has achieved its objectives. 706 00:35:08,440 --> 00:35:11,720 And those are based on key issues we discussed, 707 00:35:11,720 --> 00:35:16,320 self-reporting bias, lack of objective measures, 708 00:35:16,320 --> 00:35:18,400 no baseline or control groups, 709 00:35:18,400 --> 00:35:22,200 measurement frequency issues, vague definitions, 710 00:35:22,200 --> 00:35:25,320 and no link to specific activities at all. 711 00:35:25,320 --> 00:35:27,640 With all these issues, it's very difficult 712 00:35:27,640 --> 00:35:30,560 to conclusively claim that the program 713 00:35:30,560 --> 00:35:32,480 has achieved its objectives. 714 00:35:34,080 --> 00:35:35,240 Good, Mark? 715 00:35:35,240 --> 00:35:37,640 Well, I think in general, I have no... 716 00:35:38,960 --> 00:35:40,960 The issue is that for me, 717 00:35:40,960 --> 00:35:44,560 it is not stopping at economic rights. 718 00:35:44,560 --> 00:35:49,560 You also consider social and cultural rights, 719 00:35:49,560 --> 00:35:52,520 because in the end, you want to include them, 720 00:35:52,520 --> 00:35:56,080 the rights to have adequate food, adequate housing, 721 00:35:56,080 --> 00:35:59,480 adequate education for minority groups. 722 00:35:59,480 --> 00:36:03,440 And to measure that in this template, 723 00:36:03,440 --> 00:36:06,680 to measure that in this type of project, 724 00:36:06,680 --> 00:36:09,760 it seems for me complicated, 725 00:36:09,760 --> 00:36:14,320 and they should be interlinked also, 726 00:36:15,280 --> 00:36:17,080 like I said before, the economic, social, 727 00:36:17,080 --> 00:36:19,480 and cultural aspects should be interlinked. 728 00:36:19,480 --> 00:36:24,160 And one point is that if you want to see improvement, 729 00:36:24,160 --> 00:36:27,680 you have also to talk with the authorities 730 00:36:27,680 --> 00:36:31,720 without duty bearers, if they have measures, 731 00:36:31,720 --> 00:36:36,720 if they have laws or policies that can improve 732 00:36:36,920 --> 00:36:39,480 the level of ethnic minorities in this district 733 00:36:39,480 --> 00:36:40,840 or in these places, yeah. 734 00:36:42,320 --> 00:36:43,280 Yeah, that's great. 735 00:36:43,280 --> 00:36:45,600 I think I mentioned in part one, 736 00:36:45,600 --> 00:36:48,880 that issue where the duty bearers 737 00:36:48,880 --> 00:36:51,680 are supposed to advocate for the women, 738 00:36:51,680 --> 00:36:55,720 but the bias is they're asked how they believe themselves 739 00:36:55,720 --> 00:36:58,400 are doing in that role, 740 00:36:58,400 --> 00:37:01,560 and it's not objectively looked at by a third party. 741 00:37:02,720 --> 00:37:05,720 And you need to look at that, I think. 742 00:37:05,720 --> 00:37:06,960 So that's good, okay. 743 00:37:06,960 --> 00:37:11,040 So we basically summarize that CARE Canada 744 00:37:11,040 --> 00:37:13,560 needs to do some work on its outcome indicators 745 00:37:13,560 --> 00:37:16,720 for starters, along with a lot of other issues. 746 00:37:16,720 --> 00:37:20,200 So what we're gonna do now is send this part two 747 00:37:20,200 --> 00:37:22,840 to the Minister for International Development in Canada, 748 00:37:22,840 --> 00:37:25,280 as well as the shadow critics for the Conservative Party 749 00:37:25,280 --> 00:37:27,120 and the New Democratic Party, 750 00:37:27,120 --> 00:37:29,520 and we'll also copy CARE Canada. 751 00:37:29,520 --> 00:37:31,920 And for you listeners out there, 752 00:37:31,920 --> 00:37:36,520 send me an email at evaluatecanadaaid at gmail.com 753 00:37:36,520 --> 00:37:40,760 if you want a copy of the performance measurement framework 754 00:37:40,760 --> 00:37:43,680 and a copy of the Excel summary 755 00:37:43,680 --> 00:37:46,560 of all the outcome indicators and why they fall 756 00:37:46,560 --> 00:37:49,160 into one of these five problem areas. 757 00:37:49,160 --> 00:37:51,920 We're also looking for evaluation experts 758 00:37:51,920 --> 00:37:54,800 who may wanna be interested in doing similar work 759 00:37:54,800 --> 00:37:57,160 to what Donald and Mark have done today. 760 00:37:57,160 --> 00:38:00,080 Send me an email, and I'd be happy to send you 761 00:38:00,080 --> 00:38:02,040 other performance measurement frameworks 762 00:38:02,040 --> 00:38:05,760 that I'm looking at for future podcast episodes. 763 00:38:05,760 --> 00:38:08,800 And if you have any comments or suggestions 764 00:38:08,800 --> 00:38:12,360 on this particular episode regarding CARE Canada 765 00:38:12,360 --> 00:38:15,960 and our review of their performance measurement framework, 766 00:38:15,960 --> 00:38:18,840 send me an email and I'd be happy to respond. 767 00:38:18,840 --> 00:38:20,600 And at some point, we'll be putting 768 00:38:20,600 --> 00:38:22,840 the performance measurement frameworks 769 00:38:22,840 --> 00:38:27,600 and the summaries of their outcome indicators on a website. 770 00:38:27,600 --> 00:38:31,760 And finally, for you listeners, Canadian taxpayers, 771 00:38:31,760 --> 00:38:36,760 you may wanna email CARE Canada at info at care.ca 772 00:38:37,320 --> 00:38:39,640 and simply ask them are they gonna be improving 773 00:38:39,640 --> 00:38:42,600 their outcome indicators, or you could also ask them 774 00:38:42,600 --> 00:38:44,760 if they're gonna put all their other project 775 00:38:44,760 --> 00:38:47,440 performance measurement frameworks on their website 776 00:38:47,440 --> 00:38:49,120 so that we can all have a look, 777 00:38:49,120 --> 00:38:52,120 because right now they're not available to the public. 778 00:38:52,120 --> 00:38:55,680 So thank you for listening and stay tuned 779 00:38:55,680 --> 00:38:59,920 for an upcoming new international development organization, 780 00:38:59,920 --> 00:39:04,160 $37 million project, so it's getting a little heavier. 781 00:39:04,160 --> 00:39:06,640 Thank you, Donald, and thank you, Mark. 782 00:39:08,520 --> 00:39:09,360 Thanks, Evan. 783 00:39:09,360 --> 00:39:10,180 You're welcome, David. 784 00:39:10,180 --> 00:39:12,920 All right, thanks and stay tuned. 785 00:39:12,920 --> 00:39:20,920 Bye for now.