1 00:00:00,000 --> 00:00:03,700 I'm your host David Wand and welcome to episode five, 2 00:00:03,700 --> 00:00:06,280 part one, where we introduce a new 3 00:00:06,280 --> 00:00:08,840 international development organization. 4 00:00:08,840 --> 00:00:12,440 That international development organization is 5 00:00:12,440 --> 00:00:16,080 the Mennonite Development Associates, 6 00:00:16,080 --> 00:00:19,940 located headquarters in Waterloo, Canada. 7 00:00:19,940 --> 00:00:22,480 You can learn more about the Mennonite Development 8 00:00:22,480 --> 00:00:26,100 Associates International Development Organization 9 00:00:26,100 --> 00:00:31,100 by going to their website at www.meda.org. 10 00:00:33,800 --> 00:00:36,240 We're gonna be talking about one of the projects 11 00:00:36,240 --> 00:00:38,960 that they have been provided money from the government 12 00:00:38,960 --> 00:00:42,080 of Canada, courtesy of the Canadian taxpayer. 13 00:00:42,080 --> 00:00:47,080 This project cost the Canadian taxpayer $19,040,307. 14 00:00:51,020 --> 00:00:53,080 The title of this project is, 15 00:00:53,080 --> 00:00:57,160 A New Partnership for Sustainable Impact Investing 16 00:00:57,160 --> 00:00:59,080 in Frontier Markets. 17 00:01:00,640 --> 00:01:03,520 This project, unfortunately, doesn't specify 18 00:01:03,520 --> 00:01:06,060 the specific developing countries 19 00:01:06,060 --> 00:01:08,100 where these monies are spent. 20 00:01:08,100 --> 00:01:12,840 Instead, it only indicates that 30% of the $19 million 21 00:01:12,840 --> 00:01:17,840 will be spent in Africa, 30% spent in the Americas, 22 00:01:18,080 --> 00:01:22,480 and the remaining 40% being spent in Asia. 23 00:01:22,480 --> 00:01:25,880 I have acquired, through an access to information request, 24 00:01:25,880 --> 00:01:29,760 the performance measurement framework, or PMF, 25 00:01:29,760 --> 00:01:31,800 for this project. 26 00:01:31,800 --> 00:01:36,640 Reviewing the PMF, I have discovered there are nine outcomes 27 00:01:36,640 --> 00:01:41,000 that are expected to be achieved from this $19 million, 28 00:01:41,000 --> 00:01:45,640 and 23 outcome indicators are in the PMF 29 00:01:45,640 --> 00:01:49,200 that are being used to determine whether or not 30 00:01:49,200 --> 00:01:52,440 these nine outcomes have been achieved. 31 00:01:52,440 --> 00:01:55,720 Unfortunately, based on my review of all 23 outcome 32 00:01:55,720 --> 00:01:58,980 indicators, none of those indicators are able 33 00:01:58,980 --> 00:02:02,600 to adequately measure the achievement of the outcomes. 34 00:02:02,600 --> 00:02:05,780 Therefore, the Mennonite Development Associates 35 00:02:05,780 --> 00:02:09,320 cannot make the claim that they are achieving 36 00:02:09,320 --> 00:02:12,480 these nine outcomes based on the performance 37 00:02:12,480 --> 00:02:15,120 measurement framework that they have designed. 38 00:02:16,920 --> 00:02:19,400 So in this part one, I will just give you 39 00:02:19,400 --> 00:02:22,160 a brief description of what the services are 40 00:02:22,160 --> 00:02:25,800 that this $19 million is being paid for, 41 00:02:25,800 --> 00:02:27,680 and it falls into three areas. 42 00:02:27,680 --> 00:02:30,780 The first area is funds that are provided 43 00:02:30,780 --> 00:02:34,760 to investment firms, and funds or grants 44 00:02:34,760 --> 00:02:38,440 that are provided to small, medium enterprises. 45 00:02:38,440 --> 00:02:42,840 And a third component of this project is paying 46 00:02:42,840 --> 00:02:47,840 for Canadian firm managers, investment firm managers, 47 00:02:47,840 --> 00:02:52,840 to mentor investment firms in developing countries. 48 00:02:52,980 --> 00:02:57,440 That is, mentoring the managers of investment firms 49 00:02:57,440 --> 00:02:59,140 in developing countries. 50 00:03:00,980 --> 00:03:05,260 In those three areas, particularly investment funds 51 00:03:05,260 --> 00:03:09,340 and small, medium enterprises in these developing countries 52 00:03:09,340 --> 00:03:10,420 receive money. 53 00:03:11,360 --> 00:03:13,740 They also receive, these investment firms 54 00:03:13,740 --> 00:03:17,440 and small, medium enterprises, technical support 55 00:03:17,440 --> 00:03:20,540 on social and environmental principles 56 00:03:20,540 --> 00:03:24,300 so that they can access these funds from this project. 57 00:03:26,020 --> 00:03:28,540 And as I mentioned, the investment firms 58 00:03:28,540 --> 00:03:30,900 in these development countries, their managers 59 00:03:30,900 --> 00:03:34,820 also receive mentoring from investment fund managers 60 00:03:34,820 --> 00:03:35,820 from Canada. 61 00:03:37,060 --> 00:03:39,860 The small, medium enterprises receive grants 62 00:03:39,860 --> 00:03:41,460 for social innovation. 63 00:03:42,400 --> 00:03:45,420 And they also receive technical support 64 00:03:45,420 --> 00:03:47,400 on social innovation. 65 00:03:48,700 --> 00:03:51,320 So basically, those are what the funds, 66 00:03:51,320 --> 00:03:53,740 the $19 million are used for. 67 00:03:54,980 --> 00:03:59,220 And what I'm now going to do is give you two examples 68 00:03:59,220 --> 00:04:02,940 of outcomes from the performance measurement framework 69 00:04:02,940 --> 00:04:05,700 where the indicator, the outcome indicator 70 00:04:05,700 --> 00:04:08,700 that's been designed to determine whether or not 71 00:04:08,700 --> 00:04:13,080 the outcome has been achieved is flawed. 72 00:04:13,080 --> 00:04:15,700 So we'll start with this outcome from the performance 73 00:04:15,700 --> 00:04:17,220 measurement framework. 74 00:04:17,220 --> 00:04:21,340 The outcome is strengthened management capacity 75 00:04:21,340 --> 00:04:25,820 amongst investment fund managers in selected countries. 76 00:04:27,340 --> 00:04:30,620 The indicator for measuring this strengthened 77 00:04:30,620 --> 00:04:33,700 management capacity is, and I quote, 78 00:04:33,700 --> 00:04:36,020 level of business skills and knowledge 79 00:04:36,020 --> 00:04:39,980 among mentored fund managers. 80 00:04:39,980 --> 00:04:42,200 Sounds pretty good. 81 00:04:42,200 --> 00:04:44,980 It sounds like they're ready to measure some skills 82 00:04:44,980 --> 00:04:46,460 and knowledge levels. 83 00:04:46,460 --> 00:04:48,820 But unfortunately, when you look at the performance 84 00:04:48,820 --> 00:04:52,340 measurement framework, once again, it falls into 85 00:04:52,340 --> 00:04:55,380 the problem of self-reporting bias, 86 00:04:55,380 --> 00:04:58,860 where these mentored fund managers are themselves 87 00:04:58,860 --> 00:05:02,340 expected to self-assess their quote, 88 00:05:02,340 --> 00:05:05,220 acquisition of knowledge and skill, 89 00:05:05,220 --> 00:05:08,820 rather than being tested objectively on their knowledge 90 00:05:08,820 --> 00:05:13,820 and skill levels regarding how to work as a fund manager 91 00:05:17,220 --> 00:05:19,880 in terms of their management capacity. 92 00:05:20,780 --> 00:05:23,100 They need to be objectively tested on their knowledge 93 00:05:23,100 --> 00:05:24,100 and skill levels. 94 00:05:24,100 --> 00:05:27,300 So that's the problem with that indicator. 95 00:05:27,300 --> 00:05:30,660 Even though it sounds good, it's only by getting a hold 96 00:05:30,660 --> 00:05:34,140 of the performance measurement framework that we see 97 00:05:34,140 --> 00:05:37,380 that once again, there's a bias and they're being asked 98 00:05:37,380 --> 00:05:40,580 to self-assess themselves rather than being tested 99 00:05:40,580 --> 00:05:42,740 objectively from somebody externally. 100 00:05:43,940 --> 00:05:48,140 The second outcome that I'm going to look at is called 101 00:05:48,140 --> 00:05:52,580 strengthened business practices for economic growth, 102 00:05:52,580 --> 00:05:57,580 job creation, and social impact amongst investment firms 103 00:05:57,820 --> 00:06:00,540 and small, medium enterprises. 104 00:06:00,540 --> 00:06:04,260 The indicator to measure whether or not these 105 00:06:04,260 --> 00:06:08,620 strengthened business practices have been achieved is 106 00:06:08,620 --> 00:06:12,620 qualitative evidence of strengthened capacity 107 00:06:12,620 --> 00:06:17,340 of investment firms to promote improved social 108 00:06:17,340 --> 00:06:19,200 and environmental practices. 109 00:06:20,380 --> 00:06:23,620 The problem here again is self-reporting. 110 00:06:23,620 --> 00:06:27,300 These investment firms self-report their quote, 111 00:06:27,300 --> 00:06:31,140 strengthened capacity to promote through case studies 112 00:06:31,140 --> 00:06:34,940 and most significant change stories. 113 00:06:34,940 --> 00:06:37,500 Again, it would be better to test people in these 114 00:06:37,500 --> 00:06:40,860 investment firms on their knowledge of social 115 00:06:40,860 --> 00:06:45,580 and environmental principles, plus their skill levels 116 00:06:45,580 --> 00:06:48,920 on how to promote these principles. 117 00:06:50,380 --> 00:06:54,380 Plus, a comparison group of people in investment firms 118 00:06:54,380 --> 00:06:59,380 outside of this $19 million project would be also helpful 119 00:06:59,380 --> 00:07:04,220 so that you can show that the $19 million project 120 00:07:04,220 --> 00:07:06,780 had people at these investment firms that you were 121 00:07:06,780 --> 00:07:09,980 providing them with training and technical support 122 00:07:09,980 --> 00:07:11,660 on these principles. 123 00:07:11,660 --> 00:07:14,200 You would hope that they would have higher knowledge 124 00:07:14,200 --> 00:07:17,500 and skill levels on how to promote these social 125 00:07:17,500 --> 00:07:21,620 and environmental business practices compared to another 126 00:07:21,620 --> 00:07:25,180 group of people in investment firms that don't receive 127 00:07:25,180 --> 00:07:26,460 this project training. 128 00:07:26,460 --> 00:07:31,460 So in part two, we're going to invite the Mennonite 129 00:07:33,340 --> 00:07:38,340 Development Associates to respond to our critique 130 00:07:38,340 --> 00:07:43,340 of all 23 outcome indicators and why they're flawed. 131 00:07:44,740 --> 00:07:48,880 And we'll also have another evaluation expert on part two 132 00:07:48,880 --> 00:07:52,540 of the podcast who will work with me in going through 133 00:07:52,540 --> 00:07:56,140 at least nine of those outcome indicators to make sure 134 00:07:56,140 --> 00:07:59,660 that we cover all nine outcomes of the project. 135 00:07:59,660 --> 00:08:02,300 Hopefully we'll go through more than just nine outcome 136 00:08:02,300 --> 00:08:07,180 indicators and hopefully cover all 23 outcome indicators 137 00:08:07,180 --> 00:08:10,460 to explain how each one fails to properly measure 138 00:08:10,460 --> 00:08:11,300 the outcome. 139 00:08:11,300 --> 00:08:14,180 So stay tuned for part two where we will go through 140 00:08:14,180 --> 00:08:19,180 in more detail all 23 outcome indicators as to why they 141 00:08:20,460 --> 00:08:25,020 have a problem with not properly measuring the outcomes. 142 00:08:25,020 --> 00:08:27,220 So if you're interested in receiving the performance 143 00:08:27,220 --> 00:08:32,180 measurement framework for this project and also the outcome 144 00:08:32,180 --> 00:08:36,060 indicator summary for all 23 outcome indicators, 145 00:08:36,060 --> 00:08:40,620 explaining which of the five problem areas each outcome 146 00:08:40,620 --> 00:08:44,300 indicator falls into, feel free to send me an email 147 00:08:44,300 --> 00:08:49,300 at evaluatecanadaaid at gmail.com. 148 00:08:49,460 --> 00:08:53,420 And if there's any evaluation experts out there who would 149 00:08:53,420 --> 00:08:57,740 be interested in being on a part two of the podcast 150 00:08:57,740 --> 00:09:00,700 where we go into depth on each outcome indicator, 151 00:09:02,180 --> 00:09:05,460 feel free to email me and I will send you the remaining 152 00:09:05,460 --> 00:09:07,860 performance measurement frameworks that I have 153 00:09:07,860 --> 00:09:12,820 for season one and you can choose one and then I can also 154 00:09:12,820 --> 00:09:17,020 send you the notes that I have from my perspective 155 00:09:17,020 --> 00:09:20,300 of why the outcome indicators are flawed or not. 156 00:09:20,300 --> 00:09:23,260 I look forward to doing part two and stay tuned for part 157 00:09:23,260 --> 00:09:27,980 two which will be coming up very soon in a couple of weeks. 158 00:09:27,980 --> 00:09:56,820 Bye for now.