1 00:00:00,000 --> 00:00:04,640 Welcome to the Improving Development Evaluation Podcast. 2 00:00:04,640 --> 00:00:07,560 I'm your host David Wand and welcome to season one, 3 00:00:07,560 --> 00:00:10,440 episode four, part one. 4 00:00:10,440 --> 00:00:12,640 The International Development Organization 5 00:00:12,640 --> 00:00:14,960 where you're going to talk about in this episode, 6 00:00:14,960 --> 00:00:19,240 part one is the Ghana Rural Integrated Development 7 00:00:19,240 --> 00:00:24,200 International Organization or SHORTGRID, G-R-I-D. 8 00:00:24,200 --> 00:00:25,400 You can learn more about this 9 00:00:25,400 --> 00:00:27,560 International Development Organization 10 00:00:27,560 --> 00:00:36,600 if you check out their website at www.GRID.org. 11 00:00:36,600 --> 00:00:40,200 Their headquarters is located in Milton, Canada 12 00:00:40,200 --> 00:00:43,320 and the Ghana Rural Integrated Development 13 00:00:43,320 --> 00:00:45,720 is doing a project called 14 00:00:45,720 --> 00:00:50,600 Improving the Well-Being of Rural Women and Girls in Ghana. 15 00:00:50,600 --> 00:00:53,600 And the cost to the Canadian taxpayer for this project 16 00:00:53,600 --> 00:00:59,960 is $1,337,797. 17 00:00:59,960 --> 00:01:02,960 Now before I briefly describe the services 18 00:01:02,960 --> 00:01:05,160 delivered for this project, 19 00:01:05,160 --> 00:01:10,560 I'm going to first talk about the performance measurement framework 20 00:01:10,560 --> 00:01:14,200 that I acquired through an access to information request 21 00:01:14,200 --> 00:01:15,800 for this project. 22 00:01:15,800 --> 00:01:19,960 The PMF or performance measurement framework for this project 23 00:01:19,960 --> 00:01:24,840 has 18 outcome indicators measuring seven outcomes. 24 00:01:24,840 --> 00:01:29,560 And based on my review of this performance measurement framework, 25 00:01:29,560 --> 00:01:35,760 I have come to the conclusion that only two out of these 18 outcome indicators 26 00:01:35,760 --> 00:01:38,040 properly measure the outcomes. 27 00:01:38,040 --> 00:01:43,440 And therefore I cannot support GRID's claim 28 00:01:43,440 --> 00:01:46,120 that they have achieved their project outcomes 29 00:01:46,120 --> 00:01:48,920 with their project services delivered. 30 00:01:48,920 --> 00:01:54,760 In a breakdown of this summary of the 18 outcome indicators, 31 00:01:54,760 --> 00:01:59,920 11 of the outcome indicators do not measure the outcome at all, 32 00:01:59,920 --> 00:02:03,400 but something else not related to the outcome. 33 00:02:03,400 --> 00:02:08,600 Another three outcome indicators do measure the outcome, 34 00:02:08,600 --> 00:02:11,920 but they have no comparison group outside of the project. 35 00:02:11,920 --> 00:02:14,640 So that means women and girls outside of the project 36 00:02:14,640 --> 00:02:18,000 could have also achieved the outcomes at the same amount 37 00:02:18,000 --> 00:02:20,960 or even better than the project itself. 38 00:02:20,960 --> 00:02:25,040 Another outcome indicator asks people how they feel 39 00:02:25,040 --> 00:02:30,440 about achieving the outcome rather than measuring objectively the outcome. 40 00:02:30,440 --> 00:02:34,800 And one other outcome indicator measures the outcome, 41 00:02:34,800 --> 00:02:40,480 but not frequently enough to claim that the services achieved the outcome 42 00:02:40,480 --> 00:02:43,640 rather than other factors. 43 00:02:43,640 --> 00:02:49,520 So that's my conclusion for the performance measurement framework. 44 00:02:49,520 --> 00:02:52,920 And if you want to receive the performance measurement framework 45 00:02:52,920 --> 00:03:00,400 for this project or and or the summary of all of the 18 outcome indicators, 46 00:03:00,400 --> 00:03:05,960 feel free to email me at evaluatecanadaaid at gmail.com 47 00:03:05,960 --> 00:03:09,440 and I would be happy to send those documents to you. 48 00:03:09,440 --> 00:03:12,720 And also if there's any evaluation experts out there 49 00:03:12,720 --> 00:03:16,640 that would like to be on my podcast to also critique 50 00:03:16,640 --> 00:03:20,520 the performance measurement framework for upcoming episodes, 51 00:03:20,520 --> 00:03:22,120 feel free to email me. 52 00:03:22,120 --> 00:03:26,160 I will be happy to send you those performance measurement frameworks 53 00:03:26,160 --> 00:03:30,880 that I have acquired through an access to information request. 54 00:03:30,880 --> 00:03:35,440 Now the services for this project, there were four target groups 55 00:03:35,440 --> 00:03:38,440 that these services were being delivered to. 56 00:03:38,440 --> 00:03:42,400 And those four target groups were number one, women. 57 00:03:42,400 --> 00:03:47,960 And the focus of the project was to train them on how to grow and sell peanuts 58 00:03:47,960 --> 00:03:50,600 and how to raise and sell goats. 59 00:03:50,600 --> 00:03:55,040 In addition to that, they were also trained on safe food preparation, 60 00:03:55,040 --> 00:04:00,160 healthy behaviors and how to access markets to sell those peanuts 61 00:04:00,160 --> 00:04:02,240 and sell those goats. 62 00:04:02,240 --> 00:04:06,240 A second target group were women co-op leaders 63 00:04:06,240 --> 00:04:11,920 and they were trained in leadership, also peanut farming and animal husbandry 64 00:04:11,920 --> 00:04:16,600 and also in acquiring knowledge on women's concerns. 65 00:04:16,600 --> 00:04:19,800 And a third target group were women and men 66 00:04:19,800 --> 00:04:23,360 and they were trained in household financial management. 67 00:04:23,360 --> 00:04:27,120 And a fourth target group were girls and young women 68 00:04:27,120 --> 00:04:32,480 who received cash and scholarships at the secondary school level 69 00:04:32,480 --> 00:04:35,320 and at the post-secondary school level. 70 00:04:35,320 --> 00:04:39,640 So those services for those four target groups when delivered 71 00:04:39,640 --> 00:04:44,200 were expected for the project to achieve one of seven outcomes. 72 00:04:44,200 --> 00:04:49,960 And those seven outcomes taken from the performance measurement framework are, 73 00:04:49,960 --> 00:04:54,920 outcome number one, increased ability of responsibility bearers 74 00:04:54,920 --> 00:04:59,240 to protect the rights of women in rural northern Ghana. 75 00:04:59,240 --> 00:05:04,800 The second outcome, improved ability of women from 30 rural communities 76 00:05:04,800 --> 00:05:08,280 to advocate for their well-being. 77 00:05:08,280 --> 00:05:12,600 Outcome number three, improved access to senior secondary 78 00:05:12,600 --> 00:05:18,520 and post-secondary education by young women in 30 rural communities. 79 00:05:18,520 --> 00:05:22,320 Outcome number four, improved knowledge and skills 80 00:05:22,320 --> 00:05:27,280 in managing sustainable, resilient and productive agricultural livelihoods 81 00:05:27,280 --> 00:05:30,920 by adult women in 30 rural communities. 82 00:05:30,920 --> 00:05:35,480 Number five, improved participation in equitable household 83 00:05:35,480 --> 00:05:41,240 and community decision-making by women in 30 communities in rural northern Ghana. 84 00:05:41,240 --> 00:05:47,040 Outcome number six, increased participation in sustainable productive work 85 00:05:47,040 --> 00:05:51,120 by women in 30 communities in rural northern Ghana. 86 00:05:51,120 --> 00:05:55,520 And outcome number seven, improved well-being of women and girls 87 00:05:55,520 --> 00:06:00,800 in 30 rural communities in Ghana's northern and Brongahafu regions. 88 00:06:00,800 --> 00:06:04,280 So there you have the outcomes and the services. 89 00:06:04,280 --> 00:06:10,120 Now, for this part one, I'm just going to give you three examples 90 00:06:10,120 --> 00:06:13,960 of some outcome indicators for some of the outcomes in the project 91 00:06:13,960 --> 00:06:18,080 where they run into these various problems of poor measurement 92 00:06:18,080 --> 00:06:20,120 or no measurement at all. 93 00:06:20,120 --> 00:06:24,920 And despite the small size of this project, only over a million dollars 94 00:06:24,920 --> 00:06:29,760 if you've been following the podcast, the other projects I've been reviewing go, 95 00:06:29,760 --> 00:06:35,360 you know, anywhere up from five million dollars up to 37 million dollars. 96 00:06:35,360 --> 00:06:39,560 But in this example, I want to give some credit where credit is due. 97 00:06:39,560 --> 00:06:42,320 And as I'll get into detail here, you'll notice that some 98 00:06:42,320 --> 00:06:48,960 of the indicators are quite good as they start to actually measure incomes 99 00:06:48,960 --> 00:06:56,960 for expected immediate outcomes or longer term outcomes for training on how to grow 100 00:06:56,960 --> 00:07:00,560 and sell peanuts and how to raise and sell goats. 101 00:07:00,560 --> 00:07:02,000 So we'll get right into it now. 102 00:07:02,000 --> 00:07:03,760 So let's take a look at the outcome. 103 00:07:03,760 --> 00:07:10,080 Number four, improved knowledge and skills in managing sustainable, resilient 104 00:07:10,080 --> 00:07:16,480 and productive agricultural livelihoods by adult women in 30 rural communities. 105 00:07:16,480 --> 00:07:21,640 One of the outcome indicators for this outcome number four is, and I quote, 106 00:07:21,640 --> 00:07:29,280 number of women able to produce nine bags of peanuts on one acre farms. 107 00:07:29,280 --> 00:07:29,960 Not bad. 108 00:07:29,960 --> 00:07:32,440 They're getting close. 109 00:07:32,440 --> 00:07:38,040 If you learn how to grow and sell peanuts, eventually you're going to make income 110 00:07:38,040 --> 00:07:42,120 from selling those peanuts, but before you make the income from selling the peanuts, 111 00:07:42,120 --> 00:07:45,960 you have to be able to know how to grow and sell the peanuts. 112 00:07:45,960 --> 00:07:50,760 And so they're actually getting close by saying, let's take a look at the bags 113 00:07:50,760 --> 00:07:53,800 of peanuts that you've produced, which is pretty good. 114 00:07:53,800 --> 00:08:00,440 Only problem is those nine bags of peanuts could be empty. 115 00:08:00,440 --> 00:08:05,080 So I would suggest it's going in the right direction, this outcome indicator, 116 00:08:05,080 --> 00:08:12,640 but ideally what you'd like to do is weigh the bags of peanuts to see the weight 117 00:08:12,640 --> 00:08:16,720 of the peanuts that they've actually been able to harvest. 118 00:08:16,720 --> 00:08:24,400 So I would change the number of bags to the weight in kilograms, and please change the number 119 00:08:24,400 --> 00:08:29,800 of women to the median weight of peanuts produced by these women, along with the number 120 00:08:29,800 --> 00:08:36,000 of women in the project that were producing the peanuts that are associated 121 00:08:36,000 --> 00:08:41,240 with the median weight in kilograms of peanuts produced. 122 00:08:41,240 --> 00:08:44,680 The interesting thing about the performance measurement framework is 123 00:08:44,680 --> 00:08:50,400 if you take a closer look at it, you'll notice that the project staff are responsible 124 00:08:50,400 --> 00:08:56,520 for coming up with how many women produced nine bags of peanuts. 125 00:08:56,520 --> 00:08:59,480 Slight conflict of interest there. 126 00:08:59,480 --> 00:09:05,760 They review the project harvest records, as it says in the performance measurement framework. 127 00:09:05,760 --> 00:09:09,240 That should ideally be done by a third party. 128 00:09:09,240 --> 00:09:14,240 They should just go in and go to the women and say, show me your peanuts. 129 00:09:14,240 --> 00:09:15,040 Let's weigh them. 130 00:09:15,040 --> 00:09:21,720 That's a more accountable and transparent way of coming up with this weight 131 00:09:21,720 --> 00:09:25,360 in kilograms of peanuts produced. 132 00:09:25,360 --> 00:09:32,560 So that's a good indication that this organization, GRID, is going in the right direction. 133 00:09:32,560 --> 00:09:37,400 They're saying, show me your peanuts, but you want to get the weight, ideally. 134 00:09:37,400 --> 00:09:38,800 That's what you'd like to do. 135 00:09:38,800 --> 00:09:46,120 So the second outcome, outcome number two, improved ability of women from 30 rural communities 136 00:09:46,120 --> 00:09:48,920 to advocate for their well-being. 137 00:09:48,920 --> 00:09:55,640 The indicator here, one of them, is the percent of women co-op leaders able to identify one 138 00:09:55,640 --> 00:10:03,200 or more things, quote, they plan to lead their co-ops and their communities in doing 139 00:10:03,200 --> 00:10:06,320 to improve quality of life. 140 00:10:06,320 --> 00:10:11,760 The problem with this indicator is the plan to lead does not measure improved ability 141 00:10:11,760 --> 00:10:13,120 to advocate. 142 00:10:13,120 --> 00:10:14,480 It's too soft. 143 00:10:14,480 --> 00:10:16,520 It's too indirect. 144 00:10:16,520 --> 00:10:20,560 Also, a plan to lead is not advocating. 145 00:10:20,560 --> 00:10:26,480 The performance measurement framework shows this is done through focus group discussions 146 00:10:26,480 --> 00:10:28,920 with these women co-op leaders. 147 00:10:28,920 --> 00:10:31,520 That's not good enough. 148 00:10:31,520 --> 00:10:35,720 What you could do is, since you're training these women co-op leaders to be technically 149 00:10:35,720 --> 00:10:42,520 able to advocate, you should test them objectively on their ability to advocate. 150 00:10:42,520 --> 00:10:48,000 So for example, you could look at some advocacy or communication documents that they've produced 151 00:10:48,000 --> 00:10:50,280 and score them. 152 00:10:50,280 --> 00:10:54,280 And you could also look at their presentation skills, because I presume that they have to 153 00:10:54,280 --> 00:11:00,280 advocate to somebody, maybe the government, local government, people, governments that 154 00:11:00,280 --> 00:11:04,280 are going to respond to the interests of these women. 155 00:11:04,280 --> 00:11:07,240 So you have to see what their presentation skills are also. 156 00:11:07,240 --> 00:11:08,700 These are just examples. 157 00:11:08,700 --> 00:11:13,640 But the point is, a plan to lead is not good enough. 158 00:11:13,640 --> 00:11:18,000 If you're training them on how to advocate, you're training them how to lead, then you 159 00:11:18,000 --> 00:11:23,940 should objectively measure their ability to lead and their ability to advocate. 160 00:11:23,940 --> 00:11:29,520 So you could check out their documents that they're producing for advocacy on their quality. 161 00:11:29,520 --> 00:11:36,060 And you could check out their communication presentation skills on advocating. 162 00:11:36,060 --> 00:11:37,380 And you could score those. 163 00:11:37,380 --> 00:11:44,240 In addition, even if their technical ability to advocate does go up after the training, 164 00:11:44,240 --> 00:11:52,400 you need to have either frequent measures to show that, or a comparison group of women 165 00:11:52,400 --> 00:11:57,100 who've been not trained by the project, but are advocating for women in some other part 166 00:11:57,100 --> 00:12:04,320 of the country that are not in the project, because you would expect women who've been 167 00:12:04,320 --> 00:12:09,400 trained in this project to have better levels of advocacy skills. 168 00:12:09,400 --> 00:12:14,000 The same goes for, back to the other outcome, for the weight of the peanuts. 169 00:12:14,000 --> 00:12:19,760 You would expect women peanut farmers in Ghana, for example, outside of the project, to have 170 00:12:19,760 --> 00:12:25,320 lower median levels of weight of peanuts compared to the project where they were actually training 171 00:12:25,320 --> 00:12:28,640 them on how to sell and grow peanuts. 172 00:12:28,640 --> 00:12:32,040 So it would be helpful also to have a comparison group there too. 173 00:12:32,040 --> 00:12:38,400 Another example is outcome number six, increased participation in sustainable, productive work 174 00:12:38,400 --> 00:12:43,340 by women in 30 communities in rural northern Ghana. 175 00:12:43,340 --> 00:12:50,240 The indicator here is the percent of women earning at least $500 per year through productive 176 00:12:50,240 --> 00:12:51,920 work. 177 00:12:51,920 --> 00:13:00,240 As you can tell, the indicator earning income has got nothing to do with increased participation. 178 00:13:00,240 --> 00:13:06,920 Now you could argue that if they are increasing their participation, eventually their incomes 179 00:13:06,920 --> 00:13:11,520 are going to go up from doing that agricultural work, which is fine. 180 00:13:11,520 --> 00:13:17,160 And in fact, it's an excellent start for this organization to start measuring incomes, rather 181 00:13:17,160 --> 00:13:22,680 than asking them how they feel about their incomes, actually starting to measure. 182 00:13:22,680 --> 00:13:28,920 The only thing I would suggest here is, once again, like the number of bags of peanuts, 183 00:13:28,920 --> 00:13:36,320 here instead of just measuring the percent of women who earn $500 or more per year through 184 00:13:36,320 --> 00:13:45,140 the selling of peanuts or the selling of goats, they could simply get the median income for 185 00:13:45,140 --> 00:13:49,960 each of these activities for selling peanuts and for selling goats. 186 00:13:49,960 --> 00:13:53,880 But it's still useful because they're moving in the right direction by looking at income. 187 00:13:53,880 --> 00:14:01,480 And also again, they should have a comparison group of women outside of the project who 188 00:14:01,480 --> 00:14:08,060 are also selling peanuts and raising, selling goats to see if their incomes, their median 189 00:14:08,060 --> 00:14:10,060 incomes are lower. 190 00:14:10,060 --> 00:14:13,840 And that in fact, what we would do is instead of the percent of women, you would just have 191 00:14:13,840 --> 00:14:19,920 the median income for these women and the median income for another group of women in 192 00:14:19,920 --> 00:14:24,960 the country that have not been part of this project and compare their incomes. 193 00:14:24,960 --> 00:14:31,000 Because you would obviously expect the project training to be better at training them and 194 00:14:31,000 --> 00:14:36,320 their incomes would go up and that those income levels would be greater than the median income 195 00:14:36,320 --> 00:14:40,780 of the women who do not have the benefit of the training from the project. 196 00:14:40,780 --> 00:14:45,160 We don't know as you know, if that's going to be the case unless you have a comparison 197 00:14:45,160 --> 00:14:46,840 group. 198 00:14:46,840 --> 00:14:54,080 So those are just some examples of where the outcome indicators do not properly measure 199 00:14:54,080 --> 00:14:56,940 the outcomes. 200 00:14:56,940 --> 00:15:06,200 So what I'm going to do now is share the summary of all of the 18 outcome indicators along 201 00:15:06,200 --> 00:15:11,880 with the performance measurement framework with the representative from the Ghana Rural 202 00:15:11,880 --> 00:15:20,920 Integrated Development Organization and invite them to respond to this critique of part one. 203 00:15:20,920 --> 00:15:25,680 We'll make sure that we go through at least seven outcome indicators covering the seven 204 00:15:25,680 --> 00:15:27,760 outcomes in the project. 205 00:15:27,760 --> 00:15:35,800 But ideally we'd like to go through all 18 to explain why they believe or why they disagree 206 00:15:35,800 --> 00:15:44,760 with these indicators being a problem in supporting their claim that they've achieved their outcomes. 207 00:15:44,760 --> 00:15:50,640 And then once we finish part two, we will send it to the Minister for International Development 208 00:15:50,640 --> 00:15:57,640 as well as the shadow critics for the opposition party and the third party recommending that 209 00:15:57,640 --> 00:16:02,920 again immediately the performance measurement framework for this project and all future 210 00:16:02,920 --> 00:16:09,960 projects for this organization and for all organizations for that matter be included 211 00:16:09,960 --> 00:16:17,360 on the global affairs Canada project browser website so that everybody can see that these 212 00:16:17,360 --> 00:16:22,760 performance measurement frameworks actually show the flaws that we've discussed. 213 00:16:22,760 --> 00:16:29,400 And ideally the second recommendation would be to begin starting to properly measure the 214 00:16:29,400 --> 00:16:35,680 outcomes with better outcome indicators such as three examples I have provided in this 215 00:16:35,680 --> 00:16:37,600 part one. 216 00:16:37,600 --> 00:16:45,780 Thank you for listening and stay tuned for episode four part two featuring the Ghana 217 00:16:45,780 --> 00:16:59,480 rural integrated development international organization.