1 00:00:00,000 --> 00:00:05,000 Welcome to the Improving Development Evaluation Podcast. 2 00:00:05,000 --> 00:00:08,200 I'm your host David Wand and welcome to season two. 3 00:00:08,200 --> 00:00:11,160 This is a trailer where we're gonna talk a little bit 4 00:00:11,160 --> 00:00:14,780 about some of the changes and difference and focus 5 00:00:14,780 --> 00:00:17,600 that we're gonna do in season two. 6 00:00:17,600 --> 00:00:19,520 And I first of all wanna apologize 7 00:00:19,520 --> 00:00:23,920 to the listeners out there where I had said two things 8 00:00:23,920 --> 00:00:26,860 that we were gonna cover 12 episodes, 9 00:00:26,860 --> 00:00:29,960 what I meant was 12 projects in season one 10 00:00:29,960 --> 00:00:34,140 when actually we only managed to cover eight projects. 11 00:00:35,820 --> 00:00:37,200 And I apologize for that. 12 00:00:37,200 --> 00:00:40,920 And I also indicated that we would try to start season two 13 00:00:40,920 --> 00:00:45,720 in September, but there's been a few delays due to, 14 00:00:45,720 --> 00:00:48,400 what do you call it, economic circumstances. 15 00:00:48,400 --> 00:00:53,160 So I am now finally getting in gear for season two. 16 00:00:53,160 --> 00:00:55,240 So I thought I would give you some details 17 00:00:55,240 --> 00:00:58,240 about how we've changed it a bit for season two. 18 00:00:58,240 --> 00:01:03,000 And those changes are in just brief point form 19 00:01:03,000 --> 00:01:04,680 and then I'll go into more detail. 20 00:01:04,680 --> 00:01:07,720 First of all, I just wanna reiterate 21 00:01:07,720 --> 00:01:10,480 for those who aren't familiar with the podcast 22 00:01:10,480 --> 00:01:12,600 and haven't been following it, 23 00:01:12,600 --> 00:01:16,840 the focus of this podcast is to expose those projects 24 00:01:16,840 --> 00:01:19,320 funded by the government of Canada, 25 00:01:19,320 --> 00:01:21,520 which is in the hundreds of millions of dollars, 26 00:01:21,520 --> 00:01:26,520 if not billions, that are funding organizations 27 00:01:26,520 --> 00:01:29,700 to deliver services in developing countries 28 00:01:29,700 --> 00:01:34,140 that are making claims of development outcomes. 29 00:01:34,140 --> 00:01:37,200 So we're not interested in looking at charity projects 30 00:01:37,200 --> 00:01:39,340 and there's a lot of charity projects out there. 31 00:01:39,340 --> 00:01:42,580 For example, money being sent by the government of Canada 32 00:01:42,580 --> 00:01:45,580 through non-governmental organizations to, for example, 33 00:01:45,580 --> 00:01:49,100 pay school fees for children in Africa. 34 00:01:49,100 --> 00:01:50,100 That's a charity. 35 00:01:50,940 --> 00:01:53,100 They don't care after they pay for the school fees. 36 00:01:53,100 --> 00:01:55,060 Everybody goes home. 37 00:01:55,060 --> 00:01:57,440 We're not interested in looking at those projects. 38 00:01:57,440 --> 00:02:02,180 We're also not interested at looking at other projects 39 00:02:02,180 --> 00:02:04,260 that are also vital and very important 40 00:02:04,260 --> 00:02:06,880 and that is emergency funding. 41 00:02:06,880 --> 00:02:08,200 I like to call it emergency. 42 00:02:08,200 --> 00:02:09,700 People call it humanitarian. 43 00:02:09,700 --> 00:02:12,820 It drives me crazy because when they say that, 44 00:02:13,720 --> 00:02:14,860 what do they mean? 45 00:02:14,860 --> 00:02:17,880 So this is money that's going overseas 46 00:02:17,880 --> 00:02:20,700 to keep people alive in emergency situations 47 00:02:20,700 --> 00:02:25,180 like in refugee camps for food, water, clothing, shelter, 48 00:02:25,180 --> 00:02:28,260 medical care, emergency medical care. 49 00:02:28,260 --> 00:02:29,260 That's fine too. 50 00:02:29,260 --> 00:02:30,380 That's really important, 51 00:02:30,380 --> 00:02:33,060 but that is not the focus of this podcast. 52 00:02:33,060 --> 00:02:37,540 The focus of this podcast is to expose those projects 53 00:02:37,540 --> 00:02:40,860 that like to wave the outcome flag 54 00:02:40,860 --> 00:02:43,940 and that is claiming that their project services 55 00:02:43,940 --> 00:02:45,900 are achieving these outcomes. 56 00:02:45,900 --> 00:02:47,740 And as you've noticed from season one, 57 00:02:47,740 --> 00:02:49,900 there's all sorts of projects out there 58 00:02:49,900 --> 00:02:51,820 where they're not properly measuring 59 00:02:51,820 --> 00:02:55,380 and they're not properly evaluating these project services 60 00:02:55,380 --> 00:02:57,820 to support that claim. 61 00:02:57,820 --> 00:03:00,220 So that's the focus of the podcast. 62 00:03:00,220 --> 00:03:01,860 So what we're gonna do in season two 63 00:03:01,860 --> 00:03:06,860 that's a little different is we're going to repeatedly relate 64 00:03:08,380 --> 00:03:11,100 the flaws in these performance measurement frameworks 65 00:03:11,100 --> 00:03:12,820 that the government of Canada requires 66 00:03:12,820 --> 00:03:14,700 before they fund these projects 67 00:03:14,700 --> 00:03:17,240 for services delivered overseas. 68 00:03:17,240 --> 00:03:20,760 We're gonna connect those flaws to the development 69 00:03:20,760 --> 00:03:25,760 assistance committee, DAC, DAC, evaluation criteria. 70 00:03:26,040 --> 00:03:28,060 And for those who are familiar with it, 71 00:03:28,060 --> 00:03:33,060 if you Google OECD, DAC criteria, 72 00:03:33,140 --> 00:03:38,140 you'll notice they come up with six criteria 73 00:03:38,820 --> 00:03:42,260 to properly evaluate a project. 74 00:03:42,260 --> 00:03:45,580 And those six are one, relevance, two, coherence, 75 00:03:45,580 --> 00:03:48,380 three, effectiveness, four, efficiency, 76 00:03:48,380 --> 00:03:52,460 five, impact, and six, sustainability. 77 00:03:52,460 --> 00:03:56,420 But for my podcast, I've kind of done a brief review 78 00:03:56,420 --> 00:04:01,420 and we really are only looking at three of those six 79 00:04:02,500 --> 00:04:06,300 that keep coming up because in the trailer for season one, 80 00:04:06,300 --> 00:04:08,900 I identified five problem areas. 81 00:04:08,900 --> 00:04:10,740 And those five problem areas 82 00:04:10,740 --> 00:04:13,300 for these performance measurement frameworks 83 00:04:13,300 --> 00:04:17,140 fall into three areas under the DAC criteria. 84 00:04:17,140 --> 00:04:21,320 And they are one, relevance, two, coherence, 85 00:04:21,320 --> 00:04:23,540 and three, impact. 86 00:04:23,540 --> 00:04:27,340 So we're gonna be looking at in detail 87 00:04:27,340 --> 00:04:30,420 how the performance measurement frameworks 88 00:04:30,420 --> 00:04:33,700 fall into those three problem areas 89 00:04:33,700 --> 00:04:35,860 related to the DAC criteria. 90 00:04:35,860 --> 00:04:39,580 So I'm not gonna go into the detail right now about that, 91 00:04:39,580 --> 00:04:43,220 but perhaps you'll get an idea of why it fall, 92 00:04:43,220 --> 00:04:45,180 they fall into those three areas. 93 00:04:45,180 --> 00:04:48,100 Particularly impact, because that's the big one. 94 00:04:48,100 --> 00:04:52,020 Because the impact definition is basically claiming 95 00:04:52,020 --> 00:04:55,060 that the project services and not other factors 96 00:04:55,060 --> 00:04:58,940 are indeed responsible for achieving the outcomes. 97 00:04:58,940 --> 00:05:00,940 And that the project beneficiaries 98 00:05:00,940 --> 00:05:03,340 who show up to these services 99 00:05:03,340 --> 00:05:05,860 don't figure it out on their own through YouTube 100 00:05:05,860 --> 00:05:08,740 or talking to colleagues or whatever. 101 00:05:08,740 --> 00:05:12,900 They actually have learned the outcomes 102 00:05:12,900 --> 00:05:16,400 from the project, i.e. impact. 103 00:05:17,340 --> 00:05:19,100 The second thing I'd like to talk about 104 00:05:19,100 --> 00:05:21,340 that we're doing differently in season two 105 00:05:21,340 --> 00:05:26,020 is despite the fact that I'm Canadian, 106 00:05:26,020 --> 00:05:31,020 we can Canadians get access to information requests 107 00:05:31,020 --> 00:05:34,500 on US Agency for International Development Projects, 108 00:05:34,500 --> 00:05:36,180 USAID projects. 109 00:05:37,540 --> 00:05:38,460 So that's pretty cool. 110 00:05:38,460 --> 00:05:41,980 So I'm gonna expand to the USAID projects 111 00:05:41,980 --> 00:05:46,260 and it took me a while, but a colleague down in the States 112 00:05:46,260 --> 00:05:50,180 told me through an evaluation group 113 00:05:50,180 --> 00:05:55,180 that the key magical word is activity MEL plan. 114 00:05:55,500 --> 00:06:00,500 So that's an activity monitoring evaluation learning plan. 115 00:06:00,600 --> 00:06:03,740 That's the document that is equivalent 116 00:06:03,740 --> 00:06:07,600 to a Canadian performance measurement framework. 117 00:06:07,600 --> 00:06:09,120 So what we're gonna try to do now 118 00:06:09,120 --> 00:06:13,980 is get some activity MEL plans for projects 119 00:06:13,980 --> 00:06:16,420 through an access to information request 120 00:06:16,420 --> 00:06:18,020 in the United States. 121 00:06:18,020 --> 00:06:20,420 I already sent a couple of months ago 122 00:06:22,420 --> 00:06:24,900 an annual report from one organization 123 00:06:24,900 --> 00:06:28,460 that received $5 million from the United States government 124 00:06:28,460 --> 00:06:31,200 according to their annual report. 125 00:06:31,200 --> 00:06:33,260 So I sent it to USAID and I said, 126 00:06:33,260 --> 00:06:36,060 here's the annual report, 5 million bucks. 127 00:06:36,060 --> 00:06:41,060 Give me the activity MEL plan that or MEL plans plural 128 00:06:41,060 --> 00:06:43,700 that falls under this 5 million bucks. 129 00:06:43,700 --> 00:06:45,860 They took about two months, they got back to me 130 00:06:45,860 --> 00:06:50,260 and then they said, we can't find any activity MEL plan 131 00:06:50,260 --> 00:06:52,740 for this 5 million bucks. 132 00:06:52,740 --> 00:06:56,460 So I then sent back and I said, well, if you can't find it, 133 00:06:56,460 --> 00:07:00,020 can you tell me where this money is coming from? 134 00:07:00,020 --> 00:07:02,740 And that was about a month ago. 135 00:07:02,740 --> 00:07:05,320 And they said, by the end of October, 136 00:07:05,320 --> 00:07:08,860 they said they'd get back to me on first of all, 137 00:07:08,860 --> 00:07:11,100 finding where the money came from 138 00:07:11,100 --> 00:07:12,680 and hopefully secondly, 139 00:07:13,820 --> 00:07:17,580 giving me the corresponding activity MEL plans 140 00:07:17,580 --> 00:07:19,700 so that I can go through them and do the same thing 141 00:07:19,700 --> 00:07:23,500 that we did in season one and see if any of the outcome 142 00:07:23,500 --> 00:07:25,480 indicators don't smell right. 143 00:07:25,480 --> 00:07:27,940 So that's another thing we're gonna be doing. 144 00:07:27,940 --> 00:07:30,660 And number three, the format changes. 145 00:07:30,660 --> 00:07:33,100 Instead of just doing two parts, 146 00:07:33,100 --> 00:07:36,700 where part one I described the project and then in part two, 147 00:07:36,700 --> 00:07:39,560 we go through all the outcome indicators 148 00:07:39,560 --> 00:07:42,760 with evaluation experts, two different changes there. 149 00:07:42,760 --> 00:07:46,260 I'm gonna combine parts one and part two, 150 00:07:46,260 --> 00:07:48,860 which hopefully will lead to me covering more projects 151 00:07:48,860 --> 00:07:50,500 in season two. 152 00:07:50,500 --> 00:07:54,220 And also I've dropped evaluation experts 153 00:07:54,220 --> 00:07:57,420 because it's basically the same issue. 154 00:07:57,420 --> 00:08:00,220 So that's what I'm gonna do there, just do it by myself. 155 00:08:00,220 --> 00:08:05,220 However, since I have an ambush mentality, 156 00:08:05,260 --> 00:08:07,940 if you haven't noticed on this podcast, 157 00:08:07,940 --> 00:08:11,020 where we're trying to confront organizations 158 00:08:11,020 --> 00:08:12,780 that are waving this outcome flag 159 00:08:12,780 --> 00:08:14,340 when they should be burning it 160 00:08:15,140 --> 00:08:18,260 because they're really aren't measuring the outcomes 161 00:08:18,260 --> 00:08:22,460 and they're either doing this fraudulently and knowingly 162 00:08:22,460 --> 00:08:25,860 or they like to claim, oh, we don't have the money 163 00:08:25,860 --> 00:08:29,580 to do it properly or we don't have the technical expertise, 164 00:08:29,580 --> 00:08:31,700 which is another excuse they make. 165 00:08:31,700 --> 00:08:33,700 And I do wanna point out 166 00:08:33,700 --> 00:08:37,620 that this is not some right wing conservative podcast 167 00:08:37,620 --> 00:08:42,060 in the sense that we wanna cancel aid. 168 00:08:42,060 --> 00:08:44,500 Yes, people on the right side of the spectrum 169 00:08:44,500 --> 00:08:48,940 wanna reduce aid relative to the left side of the spectrum, 170 00:08:48,940 --> 00:08:51,820 but the left side of the spectrum should also be concerned 171 00:08:51,820 --> 00:08:55,320 because if they want aid to continue 172 00:08:55,320 --> 00:08:58,740 and receive continued political support 173 00:08:58,740 --> 00:09:00,020 from people like me, 174 00:09:00,020 --> 00:09:02,340 cause I'm on the left side, doesn't matter. 175 00:09:02,340 --> 00:09:04,340 We should all be concerned 176 00:09:04,340 --> 00:09:08,860 that our tax dollars are making a difference, i.e. impact. 177 00:09:08,860 --> 00:09:11,500 So it has to be properly measured and evaluated. 178 00:09:12,500 --> 00:09:15,140 It's not some right wing concept saying 179 00:09:15,140 --> 00:09:19,340 that you want to have your programs that you pay for 180 00:09:19,340 --> 00:09:24,340 properly measured, properly monitored, properly evaluated. 181 00:09:24,380 --> 00:09:27,200 Everybody should do that because in the end, 182 00:09:27,200 --> 00:09:30,660 the irony is that if you don't show that 183 00:09:30,660 --> 00:09:34,300 that the programs are being effective and having impact, 184 00:09:34,300 --> 00:09:35,860 the conservatives will use that 185 00:09:35,860 --> 00:09:38,540 as an excuse to scrap the program. 186 00:09:38,540 --> 00:09:40,900 So it's best that if you believe in it, 187 00:09:40,900 --> 00:09:43,420 that you support it by showing the evidence 188 00:09:43,420 --> 00:09:46,480 that the programs actually work. 189 00:09:50,020 --> 00:09:51,820 So that's important also. 190 00:09:51,820 --> 00:09:56,420 And finally, a few more points. 191 00:09:56,420 --> 00:10:01,420 What we'll do in relation to that is that I will continue to, 192 00:10:03,420 --> 00:10:08,100 if I find some organizations that have multiple projects 193 00:10:08,100 --> 00:10:10,060 and they're doing the same wrong things 194 00:10:10,060 --> 00:10:13,500 where they're not properly measuring the outcomes 195 00:10:13,500 --> 00:10:15,360 using proper outcome indicators 196 00:10:15,360 --> 00:10:17,500 and proper evaluation design, 197 00:10:17,500 --> 00:10:22,500 then if I find that and I find it repeated going on 198 00:10:22,700 --> 00:10:25,700 through various projects for the same organization, 199 00:10:25,700 --> 00:10:28,860 then I may set up an ambush podcast 200 00:10:28,860 --> 00:10:32,900 where I invite them to respond to my critique 201 00:10:32,900 --> 00:10:36,400 and I also will have evaluation experts on board 202 00:10:36,400 --> 00:10:37,820 that agree with me. 203 00:10:37,820 --> 00:10:40,220 So let's be honest about this. 204 00:10:40,220 --> 00:10:43,940 We are confronting these organizations 205 00:10:43,940 --> 00:10:47,700 who are making these claims when these claims 206 00:10:47,700 --> 00:10:51,680 are not being supported by the data that's coming in, 207 00:10:51,680 --> 00:10:54,220 let alone the performance measurement framework 208 00:10:54,220 --> 00:10:58,940 or the activity MEL plan, USAID that they're using. 209 00:11:01,160 --> 00:11:02,540 So we'll still do that, 210 00:11:02,540 --> 00:11:05,880 but I'll be more selective in the organizations I select 211 00:11:05,880 --> 00:11:10,880 to invite to attend a part two of our podcast 212 00:11:10,880 --> 00:11:14,800 where they can respond to the critique that we do. 213 00:11:14,800 --> 00:11:17,840 I will, just like season one, 214 00:11:17,840 --> 00:11:22,400 I will continue to offer to anyone if they want 215 00:11:22,400 --> 00:11:27,400 to email me at evaluatecanadaaidatgmail.com 216 00:11:28,640 --> 00:11:31,560 if they want any of the performance measurement frameworks 217 00:11:31,560 --> 00:11:35,180 from season one, as well as my Excel summary 218 00:11:35,180 --> 00:11:37,880 of the outcome indicators 219 00:11:37,880 --> 00:11:40,760 and how each one of the outcome indicators are flawed. 220 00:11:40,760 --> 00:11:43,080 I'm happy to send those out. 221 00:11:43,080 --> 00:11:45,080 And I did receive a few emails 222 00:11:45,080 --> 00:11:47,360 and I'm happy to continue to do that. 223 00:11:48,720 --> 00:11:52,600 And I will also continue since there's clearly 224 00:11:52,600 --> 00:11:56,840 an activist advocacy intent from this podcast, 225 00:11:56,840 --> 00:12:00,440 which is to get these hundreds of millions of dollars 226 00:12:00,440 --> 00:12:03,520 of projects paid for by the Canadian taxpayer 227 00:12:03,520 --> 00:12:07,200 to start properly evaluating them, 228 00:12:07,200 --> 00:12:11,200 is I will continue to send my critique 229 00:12:11,200 --> 00:12:14,460 and the episodes to the Minister 230 00:12:14,460 --> 00:12:15,760 for International Development, 231 00:12:15,760 --> 00:12:19,720 as well as the shadow critics for the Conservative Party, 232 00:12:19,720 --> 00:12:22,900 the New Democratic Party and the Green Party, 233 00:12:22,900 --> 00:12:25,280 because like I said earlier, 234 00:12:25,280 --> 00:12:27,780 proper evaluation should not be an issue 235 00:12:27,780 --> 00:12:29,160 of the right or the left. 236 00:12:29,160 --> 00:12:33,240 It should be an issue for all Canadians to be concerned, 237 00:12:33,240 --> 00:12:35,540 to make sure that their dollars overseas 238 00:12:35,540 --> 00:12:37,460 are making a difference. 239 00:12:39,400 --> 00:12:44,400 And finally, I wanted to invite another format change 240 00:12:44,960 --> 00:12:48,880 is that there is a growing industry 241 00:12:48,880 --> 00:12:50,120 in the evaluation industry, 242 00:12:50,120 --> 00:12:52,800 especially in international development 243 00:12:52,800 --> 00:12:56,800 for three types of experts that are being used 244 00:12:56,800 --> 00:13:01,320 to evaluate projects, which I'd like to learn more about 245 00:13:01,320 --> 00:13:03,040 because I'm still having issues 246 00:13:03,040 --> 00:13:07,200 with how these three different evaluation techniques 247 00:13:07,200 --> 00:13:10,480 actually properly, validly measure 248 00:13:10,480 --> 00:13:12,200 the achievement of outcomes, 249 00:13:12,200 --> 00:13:16,960 and in particular show that the project has impact. 250 00:13:16,960 --> 00:13:20,280 That is, in the absence of control groups 251 00:13:20,280 --> 00:13:24,320 or comparison groups or repeated measures, 252 00:13:24,320 --> 00:13:27,760 how does the project show that it is actually achieving 253 00:13:27,760 --> 00:13:30,920 impact, making a difference, rather than other factors? 254 00:13:30,920 --> 00:13:35,920 And those three areas of expertise are outcome harvesting, 255 00:13:35,920 --> 00:13:40,920 most significant change, and appreciative inquiry. 256 00:13:41,040 --> 00:13:43,560 So those are three evaluation techniques 257 00:13:43,560 --> 00:13:48,560 that have come up quite frequently on various platforms, 258 00:13:49,580 --> 00:13:52,880 claiming that they are proper ways 259 00:13:52,880 --> 00:13:56,080 of evaluating impact of a project, 260 00:13:56,080 --> 00:14:01,080 or at least showing relevance or coherence, 261 00:14:01,280 --> 00:14:03,720 which we'll talk about in more detail 262 00:14:03,720 --> 00:14:06,640 when we get to reviewing these performance measurement 263 00:14:06,640 --> 00:14:11,080 frameworks or the USAID activity MEL plans. 264 00:14:11,080 --> 00:14:15,000 So if there's anyone out there who has expertise 265 00:14:15,000 --> 00:14:17,120 and would be willing to come on my podcast, 266 00:14:17,120 --> 00:14:18,800 I'd be happy to interview you 267 00:14:18,800 --> 00:14:21,360 so you could explain to our listeners 268 00:14:21,360 --> 00:14:24,660 how outcome harvesting, most significant change, 269 00:14:24,660 --> 00:14:28,000 or appreciative inquiry actually evaluate 270 00:14:28,000 --> 00:14:30,400 these projects properly. 271 00:14:30,400 --> 00:14:35,240 So feel free to email me at evaluatecanadaaid.gmail.com 272 00:14:35,240 --> 00:14:37,600 and we'll set up a time, 273 00:14:37,600 --> 00:14:39,240 because I would love to put that 274 00:14:39,240 --> 00:14:42,720 as a separate episode on the podcast. 275 00:14:42,720 --> 00:14:46,280 So thank you for your time and thank you for listening. 276 00:14:46,280 --> 00:14:51,280 And soon we will begin introducing new projects 277 00:14:51,280 --> 00:14:56,280 from the government of Canada and hopefully from USAID. 278 00:14:56,280 --> 00:15:19,120 Bye for now.