1 00:00:00,680 --> 00:00:05,060 Welcome to the Improving Development Evaluation Podcast. 2 00:00:05,060 --> 00:00:06,840 I'm your host David Wand, 3 00:00:06,840 --> 00:00:09,940 and welcome to episode four, part two, 4 00:00:09,940 --> 00:00:12,240 where we continue our review and critique 5 00:00:12,240 --> 00:00:15,120 of the Ghana Rural Integrated Development, 6 00:00:15,120 --> 00:00:17,240 that's GRID, G-R-I-D, 7 00:00:17,240 --> 00:00:19,600 International Development Organization, 8 00:00:19,600 --> 00:00:22,120 and their project in Ghana entitled, 9 00:00:22,120 --> 00:00:26,840 Improving the Well-Being of Rural Women and Girls in Ghana. 10 00:00:26,840 --> 00:00:29,320 The cost to the Canadian taxpayer for this project 11 00:00:29,320 --> 00:00:34,320 is $1,337,797, 12 00:00:35,640 --> 00:00:39,480 and it has 18 outcome indicators covering seven outcomes 13 00:00:39,480 --> 00:00:43,360 that we will review each of those 18 outcome indicators 14 00:00:43,360 --> 00:00:44,900 in this part two. 15 00:00:46,260 --> 00:00:48,340 On January 7th, I sent an email 16 00:00:48,340 --> 00:00:51,200 and also left a voicemail with GRID, 17 00:00:51,200 --> 00:00:53,680 inviting them to attend the podcast. 18 00:00:53,680 --> 00:00:55,560 I did not receive a reply. 19 00:00:55,560 --> 00:00:58,360 Then I followed up again inviting them also again 20 00:00:58,360 --> 00:01:02,280 on January 18th with an email and voice message, 21 00:01:02,280 --> 00:01:06,020 but so far have not heard a reply regarding my invitation 22 00:01:06,020 --> 00:01:09,020 for them to attend this podcast in part two, 23 00:01:09,020 --> 00:01:12,060 to respond to my critique. 24 00:01:12,060 --> 00:01:14,320 So I will go through the 18 outcome indicators 25 00:01:14,320 --> 00:01:15,960 covering the seven outcomes, 26 00:01:15,960 --> 00:01:18,260 starting with the first outcome, 27 00:01:20,080 --> 00:01:24,260 which is increased ability of responsibility bearers 28 00:01:24,260 --> 00:01:27,160 to protect the rights of women in Northern Ghana. 29 00:01:27,160 --> 00:01:29,680 For this outcome, they have two outcome indicators. 30 00:01:29,680 --> 00:01:33,840 The first outcome indicator is number of times strategies 31 00:01:33,840 --> 00:01:36,760 have been implemented by responsibility bearers 32 00:01:36,760 --> 00:01:38,600 to protect the rights of women. 33 00:01:39,480 --> 00:01:42,880 And the second indicator is the percent of women 34 00:01:42,880 --> 00:01:45,360 who are fairly or very confident 35 00:01:45,360 --> 00:01:48,360 that community leaders are protecting their rights. 36 00:01:49,580 --> 00:01:51,360 Now the problem with this indicator is 37 00:01:51,360 --> 00:01:53,960 the number of strategies that have been implemented 38 00:01:53,960 --> 00:01:57,040 by responsibility bearers to protect the rights of women 39 00:01:57,040 --> 00:01:59,280 is quite a good indicator. 40 00:01:59,280 --> 00:02:01,880 You would think that as the strategies go up, 41 00:02:01,880 --> 00:02:04,960 this shows an increased ability to advocate. 42 00:02:04,960 --> 00:02:08,260 The problem is there's no comparison group 43 00:02:08,260 --> 00:02:11,760 outside of the project of other responsibility bearers 44 00:02:11,760 --> 00:02:14,680 to see how well they are protecting the rights of women 45 00:02:14,680 --> 00:02:16,020 in Northern Ghana. 46 00:02:16,020 --> 00:02:18,820 So that was the problem with that indicator. 47 00:02:18,820 --> 00:02:21,940 The second indicator, percent of women who are fairly 48 00:02:21,940 --> 00:02:24,600 or very confident that community leaders 49 00:02:24,600 --> 00:02:27,760 are protecting their rights is also quite a good indicator 50 00:02:27,760 --> 00:02:31,240 because you would expect as confidence levels go up 51 00:02:31,240 --> 00:02:35,120 in these women, in their perception of the increased ability 52 00:02:35,120 --> 00:02:37,800 of these responsibility bearers, 53 00:02:37,800 --> 00:02:39,360 this would seem to be a good measure 54 00:02:39,360 --> 00:02:41,560 of their increased ability to protect them. 55 00:02:42,800 --> 00:02:46,240 But unfortunately again, there's no comparison group. 56 00:02:46,240 --> 00:02:50,300 So we don't know if responsibility bearers 57 00:02:50,300 --> 00:02:53,440 in another part of Ghana outside of the project 58 00:02:53,440 --> 00:02:56,520 could also be doing quite well. 59 00:02:56,520 --> 00:02:59,280 So that was the problem with these two indicators 60 00:02:59,280 --> 00:03:00,280 for that outcome. 61 00:03:00,280 --> 00:03:05,080 The next outcome is improved ability of women 62 00:03:05,080 --> 00:03:09,480 from 30 rural communities to advocate for their wellbeing. 63 00:03:09,480 --> 00:03:13,360 And there's three indicators for this outcome. 64 00:03:13,360 --> 00:03:16,420 The first indicator is the number of women farmers 65 00:03:16,420 --> 00:03:21,420 able to successfully advocate for long-term access to land. 66 00:03:21,420 --> 00:03:24,300 The second indicator is the percent of women 67 00:03:24,300 --> 00:03:28,580 who feel they are completely or mostly able to advocate 68 00:03:28,580 --> 00:03:30,980 on behalf of women's concerns. 69 00:03:30,980 --> 00:03:35,180 And the final indicator is percent of women co-op leaders 70 00:03:35,180 --> 00:03:39,100 able to identify one or more things they plan 71 00:03:39,100 --> 00:03:41,980 to lead their co-ops and or communities 72 00:03:41,980 --> 00:03:45,540 in doing to improve quality of life. 73 00:03:47,300 --> 00:03:50,140 The problem with this, the first indicator, 74 00:03:50,140 --> 00:03:53,820 able to successfully advocate for long-term access to land 75 00:03:55,380 --> 00:03:57,060 is that it doesn't directly measure 76 00:03:57,060 --> 00:03:58,840 the ability to advocate. 77 00:03:58,840 --> 00:04:01,100 A better measure would be just to look 78 00:04:01,100 --> 00:04:05,060 at the amount of land that they actually acquired 79 00:04:05,060 --> 00:04:06,200 for the women. 80 00:04:06,200 --> 00:04:08,180 If they're advocating for access to land, 81 00:04:08,180 --> 00:04:11,980 instead of advocating for access, successfully advocate, 82 00:04:11,980 --> 00:04:13,820 they should just look at how much land 83 00:04:13,820 --> 00:04:18,040 they were able to acquire for these women. 84 00:04:18,040 --> 00:04:20,300 You could even argue that it's not related, 85 00:04:20,300 --> 00:04:21,900 the indicator in its current form. 86 00:04:21,900 --> 00:04:24,540 So that would be a better way to improve the indicator 87 00:04:24,540 --> 00:04:27,580 to actually measure their improved ability to advocate. 88 00:04:27,580 --> 00:04:32,280 And that is actually getting the land for the women. 89 00:04:32,280 --> 00:04:35,980 The second indicator, percent of women who feel 90 00:04:35,980 --> 00:04:38,740 they're completely or mostly able to advocate 91 00:04:38,740 --> 00:04:41,740 on behalf of women's concerns, they're asking for feeling. 92 00:04:42,900 --> 00:04:45,640 And instead, as I've mentioned before, 93 00:04:45,640 --> 00:04:48,900 they should measure directly their ability to advocate, 94 00:04:48,900 --> 00:04:51,780 more objectively than just asking them if they feel 95 00:04:51,780 --> 00:04:54,580 they are completely or mostly able to advocate. 96 00:04:54,580 --> 00:04:56,360 It's not a proper objective measure 97 00:04:56,360 --> 00:04:58,500 of the ability to advocate. 98 00:04:58,500 --> 00:05:02,500 The third indicator, the percent of women co-op leaders 99 00:05:02,500 --> 00:05:06,140 able to identify one or more things they plan 100 00:05:06,140 --> 00:05:09,100 to lead their co-ops or communities in doing 101 00:05:09,100 --> 00:05:10,920 to improve quality of life. 102 00:05:11,880 --> 00:05:14,380 They measure this through focus group discussions 103 00:05:14,380 --> 00:05:16,260 and they only do it three times 104 00:05:16,260 --> 00:05:19,900 over four years of the project. 105 00:05:19,900 --> 00:05:24,900 This is not going to show that the training they received 106 00:05:25,340 --> 00:05:28,540 to advocate is actually responsible 107 00:05:28,540 --> 00:05:31,140 for their ability to advocate. 108 00:05:31,140 --> 00:05:32,460 There's no objective measure there. 109 00:05:32,460 --> 00:05:34,020 It's a focus group discussion 110 00:05:34,020 --> 00:05:35,940 and they don't do it frequently enough. 111 00:05:35,940 --> 00:05:39,620 They only do it three times over the four years. 112 00:05:39,620 --> 00:05:44,540 So again, a better measure of their ability to advocate 113 00:05:44,540 --> 00:05:48,500 would be to test them on their ability to advocate 114 00:05:48,500 --> 00:05:50,300 and maybe do that every six months 115 00:05:50,300 --> 00:05:54,660 and even better compare it against another group of women 116 00:05:55,800 --> 00:05:58,920 who are advocating outside of the project 117 00:05:58,920 --> 00:06:02,940 and see on the same measures, maybe a test 118 00:06:02,940 --> 00:06:06,480 or reviewing their advocacy products that they deliver, 119 00:06:06,480 --> 00:06:10,540 such as maybe a pamphlet or a radio announcement 120 00:06:10,540 --> 00:06:12,900 where they're advocating for these women's rights 121 00:06:12,900 --> 00:06:15,220 and they could compare the quality of the products 122 00:06:15,220 --> 00:06:18,660 they've produced against another group of women 123 00:06:18,660 --> 00:06:21,540 and rate and compare the quality 124 00:06:21,540 --> 00:06:23,100 of those products objectively. 125 00:06:23,100 --> 00:06:27,260 The next outcome is improved access to senior secondary 126 00:06:27,260 --> 00:06:29,340 and post-secondary education 127 00:06:29,340 --> 00:06:33,340 by young women in 30 rural communities. 128 00:06:33,340 --> 00:06:36,260 The two indicators for this outcome are 129 00:06:36,260 --> 00:06:39,340 the number of women less than 25 years of age 130 00:06:39,340 --> 00:06:43,060 attending secondary and post-secondary school 131 00:06:43,060 --> 00:06:46,180 and the number of women versus men 132 00:06:46,180 --> 00:06:50,020 completing senior secondary school by age 18. 133 00:06:51,180 --> 00:06:54,260 Now, these outcomes of improved access 134 00:06:56,180 --> 00:06:57,780 actually are quite straightforward 135 00:06:57,780 --> 00:07:01,260 because they're just trying to deliver more education 136 00:07:01,260 --> 00:07:06,220 to these women through cash to pay for their school fees 137 00:07:06,220 --> 00:07:10,220 or scholarships to pay for their post-secondary tuition. 138 00:07:10,220 --> 00:07:13,680 So this is pretty straightforward 139 00:07:13,680 --> 00:07:17,260 in terms of improving access to education. 140 00:07:18,140 --> 00:07:21,180 This would be considered to be good measures. 141 00:07:21,180 --> 00:07:25,460 The problem is it's more a service through cash 142 00:07:25,460 --> 00:07:28,680 and scholarships rather than expected outcomes 143 00:07:28,680 --> 00:07:30,500 after they've attended school. 144 00:07:30,500 --> 00:07:32,860 The next outcome is improved knowledge 145 00:07:32,860 --> 00:07:36,380 and skills in managing sustainable, resilient, 146 00:07:36,380 --> 00:07:39,380 and productive agricultural livelihoods 147 00:07:39,380 --> 00:07:42,340 by adult women in 30 rural communities. 148 00:07:43,380 --> 00:07:46,420 Here we have three outcome indicators for this outcome. 149 00:07:46,420 --> 00:07:49,420 The number of women able to produce nine bags of peanuts 150 00:07:49,420 --> 00:07:51,020 on one acre farms. 151 00:07:51,020 --> 00:07:52,980 Second indicator is the number of women 152 00:07:52,980 --> 00:07:56,780 owning one or more healthy female goats. 153 00:07:56,780 --> 00:07:59,180 And the third indicator is the percentage of women 154 00:07:59,180 --> 00:08:02,340 who are moderately or highly confident 155 00:08:02,340 --> 00:08:05,220 in their knowledge and skills in farm management 156 00:08:05,220 --> 00:08:07,740 for food and crop production. 157 00:08:07,740 --> 00:08:10,720 And the percentage of women who are moderately 158 00:08:10,720 --> 00:08:13,780 or highly confident in their knowledge and skills 159 00:08:13,780 --> 00:08:15,500 in livestock rearing. 160 00:08:15,500 --> 00:08:17,380 Now, I mentioned this in part one, 161 00:08:17,380 --> 00:08:19,060 they're getting close, 162 00:08:19,060 --> 00:08:21,780 number of bags of peanuts on one acre farms, 163 00:08:21,780 --> 00:08:23,900 it's pretty good, but they haven't weighed the bags, 164 00:08:23,900 --> 00:08:24,980 they could be empty. 165 00:08:24,980 --> 00:08:27,220 So we wanna get a third party to get in there 166 00:08:27,220 --> 00:08:30,100 and actually weigh the bags of peanuts. 167 00:08:30,100 --> 00:08:32,820 So we have the total weight of peanuts produced, 168 00:08:32,820 --> 00:08:35,860 which would be a good measure of their ability, 169 00:08:35,860 --> 00:08:38,680 skill and knowledge levels in growing peanuts, 170 00:08:38,680 --> 00:08:39,580 which is good. 171 00:08:39,580 --> 00:08:43,140 And even better, it should be the median weight of peanuts. 172 00:08:43,140 --> 00:08:45,760 And they should compare that against another group 173 00:08:45,760 --> 00:08:48,940 of women peanut farmers that did not get access 174 00:08:48,940 --> 00:08:51,380 to the training through this project. 175 00:08:51,380 --> 00:08:53,660 And hopefully you'll see the median weight 176 00:08:53,660 --> 00:08:55,100 of this group through the project 177 00:08:55,100 --> 00:08:57,620 having a higher median weight 178 00:08:57,620 --> 00:08:59,820 than the women outside of the project 179 00:08:59,820 --> 00:09:04,820 who did not receive any peanut farm training. 180 00:09:05,060 --> 00:09:07,380 And the same goes for the goats. 181 00:09:07,380 --> 00:09:09,980 You would wanna get the median number of goats 182 00:09:09,980 --> 00:09:12,900 and compare that with another group of women 183 00:09:12,900 --> 00:09:14,220 who are raising goats, 184 00:09:14,220 --> 00:09:17,900 but haven't had the opportunity to receive training 185 00:09:17,900 --> 00:09:19,220 on how to raise goats. 186 00:09:19,220 --> 00:09:21,680 And finally, the last outcome indicator 187 00:09:21,680 --> 00:09:23,260 is the confidence levels 188 00:09:23,260 --> 00:09:25,180 in their knowledge and skill levels. 189 00:09:25,180 --> 00:09:27,860 I would say you don't need that. 190 00:09:27,860 --> 00:09:29,700 If you're directly measuring 191 00:09:29,700 --> 00:09:33,100 their ability through how many peanuts 192 00:09:33,100 --> 00:09:34,260 in weight they produced 193 00:09:34,260 --> 00:09:37,060 and how many goats they have produced and sold, 194 00:09:37,060 --> 00:09:38,660 that would be sufficient. 195 00:09:38,660 --> 00:09:41,640 There's no need to ask them on their confidence levels 196 00:09:41,640 --> 00:09:43,720 regarding their knowledge and skills. 197 00:09:43,720 --> 00:09:46,140 So you wouldn't even need that third indicator. 198 00:09:46,140 --> 00:09:49,500 The next outcome is improved participation 199 00:09:49,500 --> 00:09:53,780 in equitable household and community decision-making 200 00:09:53,780 --> 00:09:58,300 by women in 30 communities in rural Northern Ghana. 201 00:09:58,300 --> 00:10:01,780 We have three indicators for that outcome, 202 00:10:01,780 --> 00:10:04,820 and they are number of community members 203 00:10:04,820 --> 00:10:07,660 reporting that women participate 204 00:10:07,660 --> 00:10:10,020 in key household decisions. 205 00:10:10,020 --> 00:10:13,000 Second indicator, number of community members 206 00:10:13,000 --> 00:10:16,180 who are satisfied with women's participation 207 00:10:16,180 --> 00:10:17,980 in community decision-making. 208 00:10:17,980 --> 00:10:19,620 And number three, the third indicator 209 00:10:19,620 --> 00:10:22,680 is the number of girls completing or planning 210 00:10:22,680 --> 00:10:26,820 to complete secondary school and post-secondary education. 211 00:10:26,820 --> 00:10:31,180 So in the first one, community members reporting 212 00:10:31,180 --> 00:10:34,580 that women participate in key household decisions, 213 00:10:34,580 --> 00:10:36,220 that's a bit of a tricky one 214 00:10:36,220 --> 00:10:38,280 because you're asking community members 215 00:10:38,280 --> 00:10:40,700 to report on another group of women. 216 00:10:40,700 --> 00:10:44,140 That's the women that actually are inside of the household. 217 00:10:44,140 --> 00:10:46,540 That is kind of questionable. 218 00:10:46,540 --> 00:10:49,100 I would rather survey the women themselves 219 00:10:49,100 --> 00:10:53,840 inside their households to see how they feel they're doing 220 00:10:53,840 --> 00:10:57,940 when it comes to participating in key household decisions. 221 00:10:57,940 --> 00:11:01,780 So there's the flaw there, there's not a direct measurement. 222 00:11:01,780 --> 00:11:05,780 The second indicator, community members who are satisfied 223 00:11:05,780 --> 00:11:09,020 with women's participation in community decision-making, 224 00:11:09,020 --> 00:11:10,480 that's a pretty good measure. 225 00:11:12,500 --> 00:11:15,540 They could improve it a bit by actually looking 226 00:11:15,540 --> 00:11:19,500 at the actual representation of women 227 00:11:19,500 --> 00:11:22,700 on these community decision-making bodies, 228 00:11:22,700 --> 00:11:26,380 such as boards, school boards, for example, 229 00:11:26,380 --> 00:11:29,700 or even municipal elected councils, 230 00:11:29,700 --> 00:11:32,940 to see the percentage of women that are on these boards. 231 00:11:32,940 --> 00:11:35,820 These are bodies, community decision-making bodies 232 00:11:35,820 --> 00:11:38,420 that actually have power to make decisions. 233 00:11:38,420 --> 00:11:42,940 As for the third one, I would take it out. 234 00:11:42,940 --> 00:11:46,060 Percent of girls completing secondary school 235 00:11:46,060 --> 00:11:47,940 or post-secondary school, yes, 236 00:11:47,940 --> 00:11:50,600 it's in a measure of improved participation, 237 00:11:50,600 --> 00:11:52,980 obviously through attendance, 238 00:11:52,980 --> 00:11:57,700 but it's not related, the completion of educational school 239 00:11:57,700 --> 00:12:01,180 with the equitable household decision-making. 240 00:12:01,180 --> 00:12:03,820 Although you could argue that the more girls 241 00:12:03,820 --> 00:12:05,480 that graduate from secondary school 242 00:12:05,480 --> 00:12:07,340 and post-secondary education, 243 00:12:07,340 --> 00:12:11,840 the more knowledge they have in terms of trying to make 244 00:12:11,840 --> 00:12:15,240 a decision in the household or at the community level, 245 00:12:15,240 --> 00:12:18,580 but the other two indicators, if improved upon, 246 00:12:18,580 --> 00:12:21,860 would be sufficient to measure improved participation. 247 00:12:23,340 --> 00:12:27,480 The next outcome is increased participation 248 00:12:27,480 --> 00:12:31,940 in sustainable productive work by women in 30 communities 249 00:12:31,940 --> 00:12:34,060 in rural northern Ghana. 250 00:12:34,980 --> 00:12:37,500 Here we have three outcome indicators for that outcome 251 00:12:37,500 --> 00:12:40,780 of increased participation in productive work. 252 00:12:40,780 --> 00:12:45,780 They are the percent of women earning $500 or more per year 253 00:12:46,120 --> 00:12:47,900 through productive work, 254 00:12:47,900 --> 00:12:50,420 the percent of women under 25 completing 255 00:12:50,420 --> 00:12:52,700 or expecting to complete secondary 256 00:12:52,700 --> 00:12:54,220 and post-secondary education, 257 00:12:54,220 --> 00:12:56,660 and the third outcome indicator for that outcome 258 00:12:56,660 --> 00:12:59,300 is the percent of young and adult women 259 00:12:59,300 --> 00:13:03,300 who think they are somewhat or very likely to be hired 260 00:13:03,300 --> 00:13:05,920 for a job outside their household. 261 00:13:08,200 --> 00:13:10,460 Now, the first indicator I've mentioned previously 262 00:13:10,460 --> 00:13:11,620 is excellent. 263 00:13:11,620 --> 00:13:15,620 They are trying to get at the incomes that are measured 264 00:13:15,620 --> 00:13:18,940 from learning how to grow and sell peanuts 265 00:13:18,940 --> 00:13:21,180 and learning how to raise and sell goats. 266 00:13:21,180 --> 00:13:22,780 So it's excellent. 267 00:13:22,780 --> 00:13:24,680 They're trying to get in the right direction. 268 00:13:24,680 --> 00:13:26,580 It just needs a slight correction, 269 00:13:26,580 --> 00:13:29,580 and that is instead of 500 per year, 270 00:13:29,580 --> 00:13:32,460 they should measure the median income 271 00:13:32,460 --> 00:13:35,540 of the 750 women in the project 272 00:13:35,540 --> 00:13:38,020 and watch their incomes go up. 273 00:13:38,020 --> 00:13:40,380 It also should be specific, 274 00:13:40,380 --> 00:13:43,160 like earlier in the performance measurement framework, 275 00:13:43,160 --> 00:13:45,660 they should be measuring the median incomes 276 00:13:45,660 --> 00:13:49,760 for those growing peanuts and those raising goats. 277 00:13:49,760 --> 00:13:51,740 So it's not just productive work, 278 00:13:51,740 --> 00:13:55,220 it's precise, specific to the project training, 279 00:13:55,220 --> 00:13:57,140 which is on how to grow peanuts 280 00:13:57,140 --> 00:13:59,900 and how to raise goats and sell them, right? 281 00:13:59,900 --> 00:14:01,760 The second indicator, again, 282 00:14:01,760 --> 00:14:05,060 is talking about completion rates for women 283 00:14:05,060 --> 00:14:08,240 in secondary school and post-secondary education. 284 00:14:09,500 --> 00:14:12,260 Yes, it's a good measure of increased participation 285 00:14:12,260 --> 00:14:15,940 in productive work because they will have the potential 286 00:14:15,940 --> 00:14:19,700 to increase their participation in earning a livelihood 287 00:14:19,700 --> 00:14:23,180 based on the benefit of completing their education. 288 00:14:23,180 --> 00:14:25,600 So yes, that is important. 289 00:14:26,460 --> 00:14:30,620 I think focusing on the incomes is sufficient enough, 290 00:14:30,620 --> 00:14:32,620 and I would eliminate that indicator. 291 00:14:32,620 --> 00:14:37,620 You can keep it into track how many women as an output 292 00:14:38,300 --> 00:14:40,740 completed their education after getting the cash 293 00:14:40,740 --> 00:14:43,100 from the project or getting the scholarship 294 00:14:43,100 --> 00:14:45,240 from the project, that's useful. 295 00:14:45,240 --> 00:14:47,260 But in terms of productive work, 296 00:14:47,260 --> 00:14:50,620 I would look at incomes as a better measure 297 00:14:50,620 --> 00:14:53,500 of whether or not they've increased their participation 298 00:14:53,500 --> 00:14:55,380 in productive work. 299 00:14:55,380 --> 00:14:57,980 The last outcome indicator there, 300 00:14:57,980 --> 00:15:01,060 dealing with the percent of women who think they are somewhat 301 00:15:01,060 --> 00:15:03,820 or very likely to be hired for a job 302 00:15:03,820 --> 00:15:05,660 outside their household, again, 303 00:15:06,540 --> 00:15:08,940 it's not a bad measure, but it's indirect. 304 00:15:08,940 --> 00:15:10,780 I would focus on the incomes. 305 00:15:12,980 --> 00:15:17,580 The next outcome is improved wellbeing of women and girls 306 00:15:17,580 --> 00:15:20,620 in 30 rural communities in Ghana's northern 307 00:15:20,620 --> 00:15:23,900 and Brong Ahafl regions. 308 00:15:23,900 --> 00:15:26,540 And the indicators there, they have, there's two indicators. 309 00:15:26,540 --> 00:15:31,540 The first one is percent of women living on $2 US 310 00:15:31,800 --> 00:15:33,420 or more per day. 311 00:15:33,420 --> 00:15:37,340 And the second indicator is level of perceived wellbeing 312 00:15:37,340 --> 00:15:40,800 on a scale from one to 10 of the participating women. 313 00:15:42,420 --> 00:15:44,260 Now the first indicator is a standard, 314 00:15:44,260 --> 00:15:46,100 I think, World Bank measure. 315 00:15:46,100 --> 00:15:48,040 They don't have to do that. 316 00:15:48,040 --> 00:15:53,040 They can just, again, focus on the incomes over time 317 00:15:53,780 --> 00:15:56,700 for the project, for the training they have received. 318 00:15:56,700 --> 00:15:59,660 They don't have to stretch it and try to claim 319 00:15:59,660 --> 00:16:03,020 that there's this percent of women in the larger community 320 00:16:03,020 --> 00:16:05,040 living on $2 US per day. 321 00:16:05,040 --> 00:16:06,460 There's no connection. 322 00:16:06,460 --> 00:16:07,660 It's too far stretched. 323 00:16:07,660 --> 00:16:10,620 They should just focus on the incomes of the women 324 00:16:10,620 --> 00:16:14,100 in the project to make sure that they have been going up. 325 00:16:14,100 --> 00:16:18,420 And ideally, they should compare it against women 326 00:16:19,220 --> 00:16:21,660 who are not receiving training on how to grow peanuts 327 00:16:21,660 --> 00:16:25,820 and how to raise goats and track their incomes also. 328 00:16:25,820 --> 00:16:29,900 And hopefully the project will have higher median incomes 329 00:16:29,900 --> 00:16:32,340 than those women not in the project. 330 00:16:32,340 --> 00:16:34,740 That will be a better way of evaluating 331 00:16:34,740 --> 00:16:38,780 the effectiveness of the training project 332 00:16:38,780 --> 00:16:40,420 that we're talking about today. 333 00:16:40,420 --> 00:16:44,500 Now the second indicator, the level of perceived wellbeing 334 00:16:44,500 --> 00:16:47,600 of participating women, that's a pretty good indicator. 335 00:16:48,620 --> 00:16:49,820 I think it's okay. 336 00:16:50,820 --> 00:16:53,780 But if we look at the performance measurement framework 337 00:16:53,780 --> 00:16:58,100 in more detail, you'll notice it's only measured once 338 00:16:58,100 --> 00:17:02,460 per year over this four years of this project. 339 00:17:02,460 --> 00:17:06,620 So it would be very difficult to claim 340 00:17:06,620 --> 00:17:10,700 that the perceived wellbeing is going up 341 00:17:10,700 --> 00:17:12,500 if it is in fact going up, 342 00:17:13,580 --> 00:17:16,100 that it's due to all this wonderful training 343 00:17:16,100 --> 00:17:18,460 that these women are receiving on how to grow 344 00:17:18,460 --> 00:17:22,340 and sell peanuts and how to raise and sell goats 345 00:17:22,340 --> 00:17:24,480 if it's only measured once a year. 346 00:17:24,480 --> 00:17:26,360 There could be other factors involved 347 00:17:26,360 --> 00:17:29,160 in why their perceived wellbeing is going up 348 00:17:29,160 --> 00:17:32,220 that has nothing to do with the training 349 00:17:32,220 --> 00:17:34,560 they received in the project. 350 00:17:35,660 --> 00:17:37,760 So ideally what you'd like to do 351 00:17:37,760 --> 00:17:42,020 is measure it more frequently in the project 352 00:17:42,020 --> 00:17:47,020 and again compare it, this perceived level of wellbeing 353 00:17:48,540 --> 00:17:52,620 with other women outside of the project. 354 00:17:54,420 --> 00:17:57,900 Ideally those also involved in peanut farming 355 00:17:57,900 --> 00:18:01,680 and goat farming and then you could show 356 00:18:01,680 --> 00:18:05,020 that the perceived wellbeing is going up greater 357 00:18:05,020 --> 00:18:08,660 in the project compared to the women outside of the project 358 00:18:08,660 --> 00:18:12,360 who are also measured on their perceived wellbeing. 359 00:18:13,660 --> 00:18:16,620 Now in addition to these seven outcomes 360 00:18:16,620 --> 00:18:19,620 and the 18 outcome indicators I've covered, 361 00:18:19,620 --> 00:18:23,220 there's also another eight outcome indicators 362 00:18:23,220 --> 00:18:24,940 that all have the same problem. 363 00:18:26,060 --> 00:18:30,320 They have pre-test before the training 364 00:18:30,320 --> 00:18:34,080 and a post-test just once before they get tested 365 00:18:34,080 --> 00:18:35,860 on the knowledge and skill levels 366 00:18:35,860 --> 00:18:37,380 of the content of the training 367 00:18:37,380 --> 00:18:41,380 and then they get tested once more afterwards. 368 00:18:41,380 --> 00:18:42,780 Now that's pretty good. 369 00:18:42,780 --> 00:18:45,220 They're at least doing some objective testing 370 00:18:45,220 --> 00:18:46,620 which is great. 371 00:18:46,620 --> 00:18:50,140 The problem is they only do it once before and once after 372 00:18:50,140 --> 00:18:52,820 and as I've explained in the trailer, 373 00:18:52,820 --> 00:18:54,820 they need to show more frequent testing 374 00:18:54,820 --> 00:18:57,720 unless it's done during the same day. 375 00:18:57,720 --> 00:19:00,680 Now that's interesting. 376 00:19:00,680 --> 00:19:05,240 750 women, they come in and do a test before the training 377 00:19:05,240 --> 00:19:07,760 and then are tested after the training 378 00:19:07,760 --> 00:19:11,600 and you could show that their skill levels have gone up 379 00:19:11,600 --> 00:19:13,520 from the test, that would be good. 380 00:19:13,520 --> 00:19:15,880 We don't know from the performance measurement framework 381 00:19:15,880 --> 00:19:18,440 if it's a same day training. 382 00:19:18,440 --> 00:19:22,840 Ideally what you also want to have is a comparison group 383 00:19:22,840 --> 00:19:23,960 where they're measuring women 384 00:19:23,960 --> 00:19:26,200 who don't attend the training at all 385 00:19:26,200 --> 00:19:29,320 but are also given the same test before and after. 386 00:19:29,320 --> 00:19:30,860 That's ideally what you want to do 387 00:19:30,860 --> 00:19:32,560 so you also want to have a comparison group 388 00:19:32,560 --> 00:19:36,660 and those content trainings are in the following areas. 389 00:19:36,660 --> 00:19:40,920 Ability to identify justice issues specific to women 390 00:19:40,920 --> 00:19:43,120 along with an appropriate solution. 391 00:19:43,120 --> 00:19:47,160 Ability to identify key actions to maintain health. 392 00:19:47,160 --> 00:19:49,920 Ability to identify best practices 393 00:19:49,920 --> 00:19:52,000 for safe food preparation. 394 00:19:52,000 --> 00:19:57,000 Ability to identify ways of successfully accessing markets 395 00:19:59,220 --> 00:20:04,220 and understanding of basics in sustainable peanut farming 396 00:20:04,260 --> 00:20:07,000 and small ruminant rearing. 397 00:20:07,000 --> 00:20:09,260 And knowledge of peanut farming 398 00:20:09,260 --> 00:20:11,500 and knowledge of small ruminant husbandry. 399 00:20:11,500 --> 00:20:12,780 I guess that's the goats. 400 00:20:12,780 --> 00:20:15,680 And knowledge of women's concerns. 401 00:20:15,680 --> 00:20:18,220 So they do pre and post testing 402 00:20:18,220 --> 00:20:20,420 for these various subject areas 403 00:20:20,420 --> 00:20:22,660 but I think if they had comparison groups 404 00:20:22,660 --> 00:20:24,960 that would be better because then they could show 405 00:20:24,960 --> 00:20:27,660 that the project training is more effective 406 00:20:27,660 --> 00:20:32,500 increasing knowledge and skill levels in these subject areas 407 00:20:32,500 --> 00:20:35,020 compared to a group of similar women 408 00:20:35,020 --> 00:20:36,480 who are not in the project. 409 00:20:37,700 --> 00:20:41,020 So now that I've finished my review and critique 410 00:20:41,020 --> 00:20:43,980 of all the outcome indicators for the project, 411 00:20:43,980 --> 00:20:48,980 if you want to email me at evaluatecanadaaid at gmail.com, 412 00:20:48,980 --> 00:20:50,740 I would be happy to send you 413 00:20:50,740 --> 00:20:53,820 the performance measurement framework for this project 414 00:20:53,820 --> 00:20:58,060 as well as the summary of all of the 18 outcome indicators. 415 00:20:59,140 --> 00:21:02,020 And once again, if there are any evaluation experts 416 00:21:02,020 --> 00:21:06,060 out there who would like to participate in a part two 417 00:21:06,060 --> 00:21:10,160 of a podcast where we go through all the outcome indicators 418 00:21:10,160 --> 00:21:15,160 explaining how they do or do not fit properly 419 00:21:15,620 --> 00:21:17,560 with the measurement of the outcome, 420 00:21:17,560 --> 00:21:20,020 feel free to email me and I'm happy to send you 421 00:21:20,020 --> 00:21:23,020 the remaining performance measurement frameworks 422 00:21:23,020 --> 00:21:24,780 along with my critique 423 00:21:24,780 --> 00:21:27,340 of the performance measurement framework 424 00:21:27,340 --> 00:21:29,180 so that you could participate 425 00:21:30,480 --> 00:21:34,100 as we have a lot more PMFs coming shortly. 426 00:21:34,980 --> 00:21:38,020 Now what I will do is I will email this part two 427 00:21:38,020 --> 00:21:41,940 and part one to the minister for international development 428 00:21:41,940 --> 00:21:46,340 along with the shadow critics for international development 429 00:21:46,340 --> 00:21:49,380 for the conservative party, the new democratic party 430 00:21:49,380 --> 00:21:51,340 and the Bloc Québécois. 431 00:21:51,340 --> 00:21:54,580 And I will also recommend in that email 432 00:21:54,580 --> 00:21:56,420 that performance measurement frameworks 433 00:21:56,420 --> 00:21:59,460 immediately be available to the public 434 00:21:59,460 --> 00:22:02,540 on the Government of Canada, 435 00:22:02,540 --> 00:22:06,780 Global Affairs Canada project browser website. 436 00:22:06,780 --> 00:22:10,860 And obviously also recommend that there be improvements 437 00:22:10,860 --> 00:22:12,640 in development evaluation 438 00:22:12,640 --> 00:22:16,660 by developing better outcome indicators 439 00:22:16,660 --> 00:22:19,260 to measure project outcomes. 440 00:22:20,140 --> 00:22:25,060 Thank you very much and stay tuned for episode five, 441 00:22:25,060 --> 00:22:29,380 part one where we will be introducing once again 442 00:22:29,380 --> 00:22:32,940 another project and the project that we will be doing 443 00:22:32,940 --> 00:22:37,780 in episode five, part one, cost to the Canadian taxpayer, 444 00:22:37,780 --> 00:22:40,780 $19 million. 445 00:22:40,780 --> 00:22:42,940 So stay tuned and thank you for listening.