1 00:00:00,000 --> 00:00:05,720 Welcome to the Improving Development Evaluation Podcast. 2 00:00:05,720 --> 00:00:14,480 I'm your host David Wann and welcome to episode five, part two, where we continue our discussion 3 00:00:14,480 --> 00:00:21,240 with the Mennonite Economic Development Associates International Development Organization and 4 00:00:21,240 --> 00:00:26,800 their project entitled, A New Partnership for Sustainable Impact Investing in Frontier 5 00:00:26,800 --> 00:00:28,900 Markets. 6 00:00:28,900 --> 00:00:31,000 We would like to give you one update. 7 00:00:31,000 --> 00:00:36,720 Thanks to the Vice President of MEDA, they responded and said that of the $19 million, 8 00:00:36,720 --> 00:00:41,520 $15 million was paid back to the Government of Canada. 9 00:00:41,520 --> 00:00:46,440 So I guess we're going to be discussing the remaining $4 million, which is good to hear. 10 00:00:46,440 --> 00:00:51,240 So we have with us today an evaluation expert that we've had on our podcast before, Benjamin 11 00:00:51,240 --> 00:00:59,520 Serebor from Ghana, and we also have representing MEDA Yasser Dildar, who is an evaluation expert 12 00:00:59,520 --> 00:01:03,160 with the Mennonite Economic Development Associates. 13 00:01:03,160 --> 00:01:05,160 And it's pleasure to have you both. 14 00:01:05,160 --> 00:01:06,680 So welcome to you both. 15 00:01:06,680 --> 00:01:11,300 And what we will do now is go through a statement of the outcomes. 16 00:01:11,300 --> 00:01:17,760 And then Benjamin and I will take turns providing one outcome indicator and our explanation 17 00:01:17,760 --> 00:01:23,780 about issues around that measurement of that outcome indicator. 18 00:01:23,780 --> 00:01:28,240 So starting with the first outcome, we're going to call them immediate outcomes. 19 00:01:28,240 --> 00:01:34,760 We're going to start with immediate outcome, increased access to improved social impact 20 00:01:34,760 --> 00:01:40,720 measurement and reporting standards by small medium enterprises, investment firms, and 21 00:01:40,720 --> 00:01:43,920 the impact investment community. 22 00:01:43,920 --> 00:01:49,000 Benjamin, what outcome indicator for that outcome will you be looking at? 23 00:01:49,000 --> 00:01:53,440 Yes, so I'll be looking at the second outcome indicator. 24 00:01:53,440 --> 00:01:54,440 Which is? 25 00:01:54,440 --> 00:02:04,000 Which is the qualitative evidence of changing capacity in environmental and gender reporting. 26 00:02:04,000 --> 00:02:09,400 Okay, you seem to have issues with that measurement for that outcome. 27 00:02:09,400 --> 00:02:17,360 Okay, so the first issue I have with this outcome is that it's self-reporting. 28 00:02:17,360 --> 00:02:23,960 When you are measuring immediate outcomes, sometimes you need to move away from self-reporting 29 00:02:23,960 --> 00:02:32,040 to demonstration or evidence of the application of the knowledge which was acquired by your 30 00:02:32,040 --> 00:02:33,040 beneficiary. 31 00:02:33,040 --> 00:02:49,480 And so here, if you want them to report themselves, as you have here, 25 cases, 25% of case study 32 00:02:49,480 --> 00:02:53,520 and MSC study participants indicating strengthened capacity. 33 00:02:53,520 --> 00:02:55,760 So here they are indicating strengthened capacity. 34 00:02:55,760 --> 00:03:03,960 You as an M&E person should be able to observe and independently assess whether indeed they 35 00:03:03,960 --> 00:03:08,320 are doing an improvement in their capacity. 36 00:03:08,320 --> 00:03:11,400 And then there are several ways in which this can be done. 37 00:03:11,400 --> 00:03:18,400 I mean, for instance, after the training, you can give them maybe a case study or something 38 00:03:18,400 --> 00:03:23,200 to work on to see whether indeed, for instance, they are now able to report on gender and 39 00:03:23,200 --> 00:03:25,200 environmental issues or not. 40 00:03:25,200 --> 00:03:29,880 So here, the only issue I have here is with the self-reporting. 41 00:03:29,880 --> 00:03:39,080 Now, secondly, when you look at the way the indicator itself qualitative evidence of strengthened 42 00:03:39,080 --> 00:03:42,080 capacity in environmental and gender reporting. 43 00:03:42,080 --> 00:03:48,400 Now, indicators are supposed to be smart so that when we are reporting, it will be very 44 00:03:48,400 --> 00:03:50,880 much easier and then straightforward. 45 00:03:50,880 --> 00:03:59,880 So I made a recommendation of using an indicator which reads percentage of project beneficiaries 46 00:03:59,880 --> 00:04:07,880 who have demonstrated a high knowledge and skills in reporting on gender and environmental 47 00:04:07,880 --> 00:04:08,880 issues. 48 00:04:08,880 --> 00:04:15,880 So this is an indicator for example that could consider using instead of what you have here 49 00:04:15,880 --> 00:04:16,880 because then it makes it very smart. 50 00:04:16,880 --> 00:04:17,880 You can have a baseline. 51 00:04:17,880 --> 00:04:18,880 I know you've done it already. 52 00:04:18,880 --> 00:04:23,880 And then when you have an indicator like this, you can have a baseline and then it becomes 53 00:04:23,880 --> 00:04:26,880 very much easy for you to measure the demonstration of knowledge. 54 00:04:26,880 --> 00:04:29,880 So this is what I have for the sense here. 55 00:04:29,880 --> 00:04:33,880 So Yasser, what would your response be to that? 56 00:04:33,880 --> 00:04:41,880 Yeah, I think I agree and this is how it was done because for each of the outcomes, we 57 00:04:41,880 --> 00:04:45,880 usually have two indicators, qualitative and quantitative. 58 00:04:45,880 --> 00:04:50,880 The first one, if you look at it was like numbers or percent of SMEs reporting against 59 00:04:50,880 --> 00:04:53,880 the enhanced social impact measurement system. 60 00:04:53,880 --> 00:04:59,880 So at the baseline, they were not reporting against any social impact measurement system. 61 00:04:59,880 --> 00:05:04,880 And then we introduced the system and that took us a while because we had a criteria 62 00:05:04,880 --> 00:05:07,880 what the social impact measurement system will look like. 63 00:05:07,880 --> 00:05:12,880 So we piloted that social impact measurement system in phase one and then refined that 64 00:05:12,880 --> 00:05:17,880 in phase two because the criteria was like the social impact measurement system should 65 00:05:17,880 --> 00:05:23,880 be applicable to all sorts of companies regardless of size, geography and sector. 66 00:05:23,880 --> 00:05:26,880 So we wanted to have this standardized system. 67 00:05:26,880 --> 00:05:31,880 And then once that was developed, so we built the capacity of those SMEs on how to report 68 00:05:31,880 --> 00:05:35,880 on this impact measurement system. 69 00:05:35,880 --> 00:05:40,880 And then we asked them to send us back their reports and we looked at them and that's how 70 00:05:40,880 --> 00:05:46,880 we reported against this indicator and provided back data to global affairs like how many 71 00:05:46,880 --> 00:05:51,880 percent of SMEs reporting against enhanced social impact measurement system. 72 00:05:51,880 --> 00:05:55,880 Now, the qualitative evidence was just to triangulate the numbers. 73 00:05:55,880 --> 00:05:59,880 So we provided some individual case studies at the end of the project. 74 00:05:59,880 --> 00:06:05,880 We wrote like 30 different case studies just to augment the findings based on that percent 75 00:06:05,880 --> 00:06:07,880 of numbers. 76 00:06:07,880 --> 00:06:08,880 So I agree. 77 00:06:08,880 --> 00:06:12,880 Yeah, so that's how we did it in actual reporting. 78 00:06:12,880 --> 00:06:17,880 And as you understand from the monitoring and evaluation perspective, so these indicators 79 00:06:17,880 --> 00:06:21,880 once you develop, they are further kind of elaborated. 80 00:06:21,880 --> 00:06:26,880 We call that like indicator definition sheet in the USAID world. 81 00:06:26,880 --> 00:06:28,880 It's the performance information reference sheet. 82 00:06:28,880 --> 00:06:34,880 So we define each of these terms and how they will be reported, measured and reported internally 83 00:06:34,880 --> 00:06:39,880 and externally because the PMF is just providing you a list of indicators. 84 00:06:39,880 --> 00:06:44,880 Within indicators, you have certain terminologies that need further elaboration. 85 00:06:44,880 --> 00:06:47,880 What do we mean by social impact measurement? 86 00:06:47,880 --> 00:06:51,880 All those kind of things are further defined. 87 00:06:51,880 --> 00:06:57,880 And one thing, I think the overall thing that we need to keep in mind that this is a blended 88 00:06:57,880 --> 00:06:59,880 finance project. 89 00:06:59,880 --> 00:07:08,880 So the core of the blended finance project is to incentivize companies to implement social 90 00:07:08,880 --> 00:07:16,880 impact measurement and ESG related practices, but also to crowd in the private capital to 91 00:07:16,880 --> 00:07:18,880 amplify the impact. 92 00:07:18,880 --> 00:07:23,880 And some of those traditional performance measurement framework might not be so relevant 93 00:07:23,880 --> 00:07:28,880 for the blended finance project, but we try to include all of those aspects. 94 00:07:28,880 --> 00:07:34,880 So we have some learnings coming out of this PMF and for the project PMF was just one reporting 95 00:07:34,880 --> 00:07:36,880 instrument. 96 00:07:36,880 --> 00:07:41,880 But just one question is you're talking about small and medium enterprises, but it's the 97 00:07:41,880 --> 00:07:47,880 individuals inside of them that you're claiming you've strengthened their capacity to be able 98 00:07:47,880 --> 00:07:52,880 to report better on environmental and gender reporting. 99 00:07:52,880 --> 00:07:58,880 So it sounds like you've got some tool there where you're actually measuring a real human 100 00:07:58,880 --> 00:08:02,880 being inside of this small and medium enterprise. 101 00:08:02,880 --> 00:08:05,880 And somehow it's been aggregated up. 102 00:08:05,880 --> 00:08:07,880 So we missed that. 103 00:08:07,880 --> 00:08:09,880 So what could we get at? 104 00:08:09,880 --> 00:08:14,880 It sounds like there might be some tool that you have where you actually measure a human 105 00:08:14,880 --> 00:08:18,880 being, because that's what we're trying to get at here, because it's not the institution 106 00:08:18,880 --> 00:08:19,880 or the organization. 107 00:08:19,880 --> 00:08:20,880 It's inside of them. 108 00:08:20,880 --> 00:08:26,880 You can't just say a small medium enterprise is reporting with strength and capacity unless 109 00:08:26,880 --> 00:08:28,880 you measure people inside of that. 110 00:08:28,880 --> 00:08:33,880 And what we're saying is there's a bias because when you use most significant change, it's 111 00:08:33,880 --> 00:08:37,880 based on asking them how they feel they're doing on that. 112 00:08:37,880 --> 00:08:40,880 What we're claiming is that you need to do something more objectively. 113 00:08:40,880 --> 00:08:47,880 So do you have something lower down where a human being inside that small medium enterprise 114 00:08:47,880 --> 00:08:53,880 is actually being measured on their ability to report on environmental and gender reporting 115 00:08:53,880 --> 00:08:56,880 in a better way? 116 00:08:56,880 --> 00:08:58,880 Yeah, we did do this. 117 00:08:58,880 --> 00:09:04,880 We were like evaluation conducted for this project and they did ask these SME to extend 118 00:09:04,880 --> 00:09:05,880 to- 119 00:09:05,880 --> 00:09:06,880 Well, it's not an SME. 120 00:09:06,880 --> 00:09:07,880 Sorry to interrupt. 121 00:09:07,880 --> 00:09:10,880 It's a human being inside of the SME. 122 00:09:10,880 --> 00:09:13,880 Yeah, that is entities are made of human beings. 123 00:09:13,880 --> 00:09:14,880 So whenever, yeah. 124 00:09:14,880 --> 00:09:15,880 So all the entities are- 125 00:09:15,880 --> 00:09:17,880 Right, so you're measuring those human beings. 126 00:09:17,880 --> 00:09:22,880 Yeah, but we are measuring the practice of those entities. 127 00:09:22,880 --> 00:09:27,880 Of course, all the works are done like for instance, if we say in an organization, their 128 00:09:27,880 --> 00:09:30,880 organizational human resource practices are great. 129 00:09:30,880 --> 00:09:35,880 So it's not like the organization practices, it's the individual within those organizations 130 00:09:35,880 --> 00:09:37,880 implementing the practices. 131 00:09:37,880 --> 00:09:43,880 So that's like, you know, all the organizations are made up of individuals and then definitely 132 00:09:43,880 --> 00:09:49,880 we worked with certain individuals who were the focal persons for us receiving incentives 133 00:09:49,880 --> 00:09:52,880 to implement social impact measurement system. 134 00:09:52,880 --> 00:09:58,880 And we worked definitely with them and they had increased ability to report on this, but 135 00:09:58,880 --> 00:10:01,880 it was the practice of those SME that- 136 00:10:01,880 --> 00:10:03,880 Benjamin, any comments? 137 00:10:03,880 --> 00:10:09,880 Yeah, I mean, Ash is very rightly mentioned. 138 00:10:09,880 --> 00:10:17,880 Once we have the entity individuals mentioned here, obviously we need to measure that. 139 00:10:17,880 --> 00:10:24,880 I agree with yourself that yes, those individuals are going to make a certain contribution at 140 00:10:24,880 --> 00:10:26,880 the institutional level. 141 00:10:26,880 --> 00:10:31,880 And so we can measure that contribution they are making at the institutional level. 142 00:10:31,880 --> 00:10:37,880 I mean, or at the institution now for instance, we've participated in this training. 143 00:10:37,880 --> 00:10:44,880 The institution has not instituted measures to ensure reporting on gender and environmental 144 00:10:44,880 --> 00:10:46,880 issues at the institutional level. 145 00:10:46,880 --> 00:10:49,880 But then at the individual level, so what do we do? 146 00:10:49,880 --> 00:10:51,880 Probably there could be other practices, but not them. 147 00:10:51,880 --> 00:10:56,880 So then it's always good that we look at the individuals also in addition to looking at 148 00:10:56,880 --> 00:10:59,880 what the institutional level is. 149 00:10:59,880 --> 00:11:06,880 Yes, and just saying the number of SMEs are reporting against this system is, in my opinion, 150 00:11:06,880 --> 00:11:11,880 not an adequate measure to make the claim that their capacity is strengthened. 151 00:11:11,880 --> 00:11:16,880 So we can leave it at that, but I understand what you're saying is that the PMF is maybe 152 00:11:16,880 --> 00:11:21,880 at a higher aggregate level where that's just simply not the case. 153 00:11:21,880 --> 00:11:22,880 So that's fine. 154 00:11:22,880 --> 00:11:23,880 We can move on. 155 00:11:23,880 --> 00:11:28,880 But I mean, I think our point is still valid that you can't make these claims of strength 156 00:11:28,880 --> 00:11:34,880 and capacity unless you're going to measure individuals on their strength and capacity. 157 00:11:34,880 --> 00:11:40,880 I mean, and that's a common problem that we have, but we won't repeat ourselves. 158 00:11:40,880 --> 00:11:45,880 I'll move on to the second, unless, yes, here, you want to make another comment before we 159 00:11:45,880 --> 00:11:48,880 move on to the next outcome. 160 00:11:48,880 --> 00:11:54,880 Yeah, I think it's the same like whether we were focusing on individuals or entities. 161 00:11:54,880 --> 00:11:56,880 So here the focus was entities. 162 00:11:56,880 --> 00:12:03,880 Even those individuals leave, those SMEs will continue reporting on that enhanced social 163 00:12:03,880 --> 00:12:04,880 impact measurement system. 164 00:12:04,880 --> 00:12:07,880 So because the purpose was to institutionalize these practices. 165 00:12:07,880 --> 00:12:12,880 So we make this system available to those SMEs. 166 00:12:12,880 --> 00:12:16,880 So even those individuals who participated in initial discussions and trainings, they 167 00:12:16,880 --> 00:12:22,880 would leave, but SMEs will still continue reporting on the social impact measurement 168 00:12:22,880 --> 00:12:23,880 system. 169 00:12:23,880 --> 00:12:28,880 And there may be, you may have an internal system where the percent of those SMEs that 170 00:12:28,880 --> 00:12:34,880 are reporting against, but it's not here, but ideally you'd want to have the percent 171 00:12:34,880 --> 00:12:41,880 of those SMEs that actually reach a minimum standard, like a SMART, a minimum standard 172 00:12:41,880 --> 00:12:43,880 of acceptable strength and capacity. 173 00:12:43,880 --> 00:12:49,880 And you just tick the box and say, now, based on our assessment of those individuals in 174 00:12:49,880 --> 00:12:54,880 that SME, we can conclude that that SME as an entity has strengthened its capacity. 175 00:12:54,880 --> 00:12:59,880 And we don't see that here, but we suspect you've got it somewhere else. 176 00:12:59,880 --> 00:13:02,880 So we'll move on to the next outcome. 177 00:13:02,880 --> 00:13:07,880 Improved business knowledge and skills in relation to social and environmental issues 178 00:13:07,880 --> 00:13:10,880 amongst selected small, medium enterprises. 179 00:13:10,880 --> 00:13:16,880 And the indicator I picked there would be the number of small, medium enterprise innovation 180 00:13:16,880 --> 00:13:23,880 grant recipients reporting improved business knowledge and skills in relation to social 181 00:13:23,880 --> 00:13:24,880 and environmental issues. 182 00:13:24,880 --> 00:13:30,880 And it's the same issue, basically, where you're claiming that the SME, the small, medium 183 00:13:30,880 --> 00:13:36,880 enterprise, is the number is reporting on improved basic knowledge and skills. 184 00:13:36,880 --> 00:13:39,880 But, and you do actually have an assessment. 185 00:13:39,880 --> 00:13:41,880 So I've actually rated it as good. 186 00:13:41,880 --> 00:13:48,880 The only issue there, yes, here, we believe, is that meta, your organization should not 187 00:13:48,880 --> 00:13:49,880 be involved in that assessment. 188 00:13:49,880 --> 00:13:54,880 Ideally, you'd want to have a third party go in and do that assessment to see actually 189 00:13:54,880 --> 00:14:00,880 if the individuals inside of that small, medium enterprise actually have experienced 190 00:14:00,880 --> 00:14:03,880 improvements in their basic knowledge and skills. 191 00:14:03,880 --> 00:14:05,880 So we would prefer a third party. 192 00:14:05,880 --> 00:14:10,880 And then the other reason is, if you look at the frequency that you're measuring that, 193 00:14:10,880 --> 00:14:16,880 you probably want to have a comparison group of another small, medium enterprise somewhere 194 00:14:16,880 --> 00:14:22,880 else in the country, because you're doing this globally, where they're not experiencing 195 00:14:22,880 --> 00:14:29,880 your training on this improved business knowledge and skills, where they could be figuring it 196 00:14:29,880 --> 00:14:35,880 out all on their own without $4 million in a project, where they're going to YouTube 197 00:14:35,880 --> 00:14:37,880 or talking to colleagues or other entrepreneurs. 198 00:14:37,880 --> 00:14:40,880 So that's my response there. 199 00:14:40,880 --> 00:14:45,880 You'd want to use a third party on this assessment tool that you've developed and maybe have 200 00:14:45,880 --> 00:14:52,880 a comparison group where you can show that your SME that you've trained or through the 201 00:14:52,880 --> 00:14:59,880 assessment shown that the knowledge and skill levels of that SME is going up greater than 202 00:14:59,880 --> 00:15:02,880 another SME where they're not involved in the project. 203 00:15:02,880 --> 00:15:06,880 What would be your response to that? 204 00:15:06,880 --> 00:15:09,880 Yeah, I completely agree with what you said. 205 00:15:09,880 --> 00:15:19,880 It's only the limitation in terms of resources and also some logistical constraints, why 206 00:15:19,880 --> 00:15:26,880 businesses which are not part of the project will give their data or would like to go through 207 00:15:26,880 --> 00:15:27,880 the assessment. 208 00:15:27,880 --> 00:15:34,880 So if I'm a business and I'm not benefiting in any way, I'm least likely to provide data, 209 00:15:34,880 --> 00:15:41,880 even those SMEs because these are businesses and I have written in one of my responses 210 00:15:41,880 --> 00:15:48,880 for them like reporting on social and environmental and gender aspects is something new. 211 00:15:48,880 --> 00:15:51,880 They are good at financial reporting and things like that. 212 00:15:51,880 --> 00:15:58,880 So even for those who are participating in the project, the challenge and if we are asking 213 00:15:58,880 --> 00:16:02,880 those SMEs or those businesses who are not part of the project and they are not benefiting 214 00:16:02,880 --> 00:16:06,880 in any way, why I should be spending my time and money. 215 00:16:06,880 --> 00:16:11,880 So that was the constraint, practical constraint that we couldn't do the comparison group from 216 00:16:11,880 --> 00:16:12,880 evaluation perspective. 217 00:16:12,880 --> 00:16:14,880 I 100% agree with you. 218 00:16:14,880 --> 00:16:20,880 And also I take your point on the third party assessment is the again, function of resources 219 00:16:20,880 --> 00:16:22,880 so there was that limitation. 220 00:16:22,880 --> 00:16:27,880 And how frequently, another option as you've probably seen from our trailer and the podcast 221 00:16:27,880 --> 00:16:33,880 is another option is if you can't get a comparison group, you could just measure more frequently 222 00:16:33,880 --> 00:16:35,880 their levels of knowledge and skill. 223 00:16:35,880 --> 00:16:40,880 How many in the PMF I have it, but I thought I'd just ask you how many times do you do 224 00:16:40,880 --> 00:16:41,880 this assessment? 225 00:16:41,880 --> 00:16:42,880 I think it's not very frequent. 226 00:16:42,880 --> 00:16:47,880 I think it's just what three times a year, but correct me if I'm wrong. 227 00:16:47,880 --> 00:16:48,880 Yeah. 228 00:16:48,880 --> 00:16:49,880 Yeah. 229 00:16:49,880 --> 00:16:51,880 It was done quarterly basis. 230 00:16:51,880 --> 00:16:58,880 If you look at, and that's why it becomes limiting is, you know, we have like in the 231 00:16:58,880 --> 00:17:03,880 response that I sent to you, we have enhanced impact measurement system developed as part 232 00:17:03,880 --> 00:17:06,880 of the project and research was an integral component. 233 00:17:06,880 --> 00:17:09,880 So university of Waterloo was supporting us on that one. 234 00:17:09,880 --> 00:17:15,880 So they did look at some like secondary kind of businesses and tried to do the comparison. 235 00:17:15,880 --> 00:17:20,880 It was not like robust, but at least there was something that was, that was done and 236 00:17:20,880 --> 00:17:21,880 it is not part of PMF. 237 00:17:21,880 --> 00:17:23,880 But I agree and take your point. 238 00:17:23,880 --> 00:17:25,880 Like always comparison group is always helpful. 239 00:17:25,880 --> 00:17:27,880 It's not just pre and post. 240 00:17:27,880 --> 00:17:29,880 So you need to have a counter factor. 241 00:17:29,880 --> 00:17:30,880 Yeah. 242 00:17:30,880 --> 00:17:31,880 To prove your point. 243 00:17:31,880 --> 00:17:32,880 Yeah. 244 00:17:32,880 --> 00:17:33,880 Quarterly is not bad. 245 00:17:33,880 --> 00:17:34,880 Okay. 246 00:17:34,880 --> 00:17:36,880 So let's move to the next one. 247 00:17:36,880 --> 00:17:38,880 Benjamin, I think you're going to do the indicator here. 248 00:17:38,880 --> 00:17:46,880 The outcome is strengthened management capacity amongst investment fund managers in selected 249 00:17:46,880 --> 00:17:47,880 countries. 250 00:17:47,880 --> 00:17:51,880 And the outcome indicator you're doing there, Benjamin. 251 00:17:51,880 --> 00:17:52,880 Yes. 252 00:17:52,880 --> 00:18:01,880 So the outcome indicator I'm doing here is level of business skills and knowledge among 253 00:18:01,880 --> 00:18:05,880 men and women. 254 00:18:05,880 --> 00:18:10,880 So I think here again, because we are, we are measuring their level of knowledge and 255 00:18:10,880 --> 00:18:14,880 then we are looking at like a scale of 70 percent. 256 00:18:14,880 --> 00:18:18,880 So we are using some of the new knowledge and skills. 257 00:18:18,880 --> 00:18:21,880 At the higher, at four or higher. 258 00:18:21,880 --> 00:18:26,880 So I mean here again, we are looking at it like self-reporting. 259 00:18:26,880 --> 00:18:35,880 And we, or is it not going to be good to have an independent assessment instead of the people 260 00:18:35,880 --> 00:18:37,880 doing the assessment themselves? 261 00:18:37,880 --> 00:18:41,880 Because probably there could be that assessment bias. 262 00:18:41,880 --> 00:18:50,880 So what usually happens is that you as the implementer establish a scale which you are 263 00:18:50,880 --> 00:18:53,880 going to use to measure this knowledge. 264 00:18:53,880 --> 00:18:58,880 And so you give them an assignment or an exam or something just to measure the knowledge 265 00:18:58,880 --> 00:19:02,880 that is acquired before and then after. 266 00:19:02,880 --> 00:19:08,880 Now based on this criteria, you'll be able to establish whether indeed there has been 267 00:19:08,880 --> 00:19:11,880 an improvement in their knowledge. 268 00:19:11,880 --> 00:19:16,880 But when you ask them to assess it, they can say, oh yeah, we've found an improvement in 269 00:19:16,880 --> 00:19:17,880 knowledge, blah, blah, blah. 270 00:19:17,880 --> 00:19:22,880 You can, later at the intermediate outcome, you can measure it. 271 00:19:22,880 --> 00:19:28,880 But initially it's important for you to know the kind of knowledge they have acquired here 272 00:19:28,880 --> 00:19:30,880 before you get to the intermediate. 273 00:19:30,880 --> 00:19:34,880 Because at the intermediate, if they did not get the right knowledge, then you will not 274 00:19:34,880 --> 00:19:37,880 be able to achieve your target at the intermediate level. 275 00:19:37,880 --> 00:19:45,880 So it's always important that you independently assess whether indeed what you taught them 276 00:19:45,880 --> 00:19:47,880 has acquired the knowledge. 277 00:19:47,880 --> 00:19:53,880 Because I remember we did the training, we did before, and then we did after assessment. 278 00:19:53,880 --> 00:19:58,880 And then even after assessment, some people were getting lower than what they got in the 279 00:19:58,880 --> 00:19:59,880 before. 280 00:19:59,880 --> 00:20:02,880 I mean, I was surprised that we got that result. 281 00:20:02,880 --> 00:20:08,880 So it's always important that sometimes you do some of this assessment before you get 282 00:20:08,880 --> 00:20:10,880 to the intermediate level. 283 00:20:10,880 --> 00:20:15,880 So if there are any gaps in the knowledge at that level, you'll be able to correct it 284 00:20:15,880 --> 00:20:16,880 before you move. 285 00:20:16,880 --> 00:20:21,880 And then we can also look at the frequency of measuring this also, during the training 286 00:20:21,880 --> 00:20:22,880 or immediately after. 287 00:20:22,880 --> 00:20:26,880 That's what I have for this outcome. 288 00:20:26,880 --> 00:20:27,880 Yes, Asir. 289 00:20:27,880 --> 00:20:28,880 Yes, Asir, sorry. 290 00:20:28,880 --> 00:20:31,880 Yeah, Medneban, I agree. 291 00:20:31,880 --> 00:20:38,880 So for the mentorship program, we did develop a very robust kind of assessment criteria, 292 00:20:38,880 --> 00:20:46,880 and that same form was used at the beginning of the mentorship program, like when a fund 293 00:20:46,880 --> 00:20:52,880 managers would decide to participate and towards the end and in between as well. 294 00:20:52,880 --> 00:20:56,880 And then we also tried to do the satisfaction with the process itself. 295 00:20:56,880 --> 00:20:57,880 I agree. 296 00:20:57,880 --> 00:20:58,880 It is self-explanatory. 297 00:20:58,880 --> 00:20:59,880 Well, this is it. 298 00:20:59,880 --> 00:21:01,880 What you're introducing is not in the PMF. 299 00:21:01,880 --> 00:21:06,880 We've just got both significant change and case studies, I think. 300 00:21:06,880 --> 00:21:08,880 Correct me if I'm wrong. 301 00:21:08,880 --> 00:21:09,880 Right, Benjamin? 302 00:21:09,880 --> 00:21:11,880 What exactly was used for that? 303 00:21:11,880 --> 00:21:17,880 It says level of business skills and knowledge among mentored fund managers. 304 00:21:17,880 --> 00:21:22,880 So I think the fund manager, people being mentored were reporting. 305 00:21:22,880 --> 00:21:25,880 So that's a self-reporting bias right there. 306 00:21:25,880 --> 00:21:31,880 But you're talking about an independent assessment, Asir, which I don't think that's going on 307 00:21:31,880 --> 00:21:32,880 here, correct? 308 00:21:32,880 --> 00:21:37,880 Or are you saying that's in addition to the self-reporting bias, there was an external 309 00:21:37,880 --> 00:21:40,880 assessment of those people being mentored? 310 00:21:40,880 --> 00:21:43,880 No, it was a self-reporting assessment. 311 00:21:43,880 --> 00:21:45,880 So we agree then. 312 00:21:45,880 --> 00:21:46,880 All right. 313 00:21:46,880 --> 00:21:47,880 Gotcha. 314 00:21:47,880 --> 00:21:49,880 Yeah, so it was, but it was like pre and post. 315 00:21:49,880 --> 00:21:51,880 So we, they were like identical. 316 00:21:51,880 --> 00:21:52,880 Sorry to interrupt. 317 00:21:52,880 --> 00:21:54,880 So let's just cut to the chase. 318 00:21:54,880 --> 00:22:00,880 If we've got this self-reporting bias, would you not prefer to have some external objective 319 00:22:00,880 --> 00:22:03,880 assessment of their business skills and knowledge? 320 00:22:03,880 --> 00:22:07,880 And that the only reason you didn't do that is maybe you didn't have the money. 321 00:22:07,880 --> 00:22:08,880 Correct. 322 00:22:08,880 --> 00:22:10,880 Resources have been one of the constraints. 323 00:22:10,880 --> 00:22:15,880 And second part, as I have been saying, based on my experience working with the private 324 00:22:15,880 --> 00:22:22,880 sector, especially investment fund managers, and these relatively speaking, bigger businesses, 325 00:22:22,880 --> 00:22:27,880 is the challenge in getting data is not always easy. 326 00:22:27,880 --> 00:22:34,880 So we have like PMF developed from the perspective of traditional social economic NGO sector or 327 00:22:34,880 --> 00:22:41,880 civil society or sector, but from the private sector, it's really, really tough. 328 00:22:41,880 --> 00:22:45,880 Even getting this self-reporting assessment was really easy. 329 00:22:45,880 --> 00:22:46,880 Interesting. 330 00:22:46,880 --> 00:22:48,880 So we need to understand that. 331 00:22:48,880 --> 00:22:50,880 And this is not the first project. 332 00:22:50,880 --> 00:22:52,880 Like we did another one. 333 00:22:52,880 --> 00:22:53,880 So same thing. 334 00:22:53,880 --> 00:22:58,880 Like we need to actually sell and make a business case by this data is important. 335 00:22:58,880 --> 00:23:02,880 So data should be used for like making a business case. 336 00:23:02,880 --> 00:23:07,880 If they don't see a business case for providing data, so they won't do it because they are 337 00:23:07,880 --> 00:23:12,880 putting in their money and everything because they will be attracting private capital. 338 00:23:12,880 --> 00:23:19,880 And the success of this project was this project established a fund which is still functional. 339 00:23:19,880 --> 00:23:24,880 And they're still providing investment money to SMEs in developing countries to create 340 00:23:24,880 --> 00:23:28,880 jobs and was able to attract around $150 million. 341 00:23:28,880 --> 00:23:34,880 It makes you wonder why even have these outcomes of human development in this project if it's 342 00:23:34,880 --> 00:23:42,880 just a bag of money going in and if it's actually developing what you said, crowding in of $85 343 00:23:42,880 --> 00:23:43,880 million, et cetera. 344 00:23:43,880 --> 00:23:49,880 That means that they in theory have the capacity to do whatever they got to do. 345 00:23:49,880 --> 00:23:54,880 And it makes you wonder why all these human measurement performance outcomes are in here 346 00:23:54,880 --> 00:23:56,880 and why you had this mentoring. 347 00:23:56,880 --> 00:24:03,880 Maybe you didn't need it if these SMEs and investment fund managers in these developing 348 00:24:03,880 --> 00:24:10,880 countries succeeded in generating all this profit and capital. 349 00:24:10,880 --> 00:24:11,880 Yeah, no. 350 00:24:11,880 --> 00:24:13,880 I think they still need like when we do mentoring. 351 00:24:13,880 --> 00:24:18,880 So they were paired, for instance, fund managers from those developing countries with one of 352 00:24:18,880 --> 00:24:24,880 the fund managers in North America or Europe, how it worked like establishing networks. 353 00:24:24,880 --> 00:24:29,880 So what people, you know, fund managers in those developing countries lack is the networking. 354 00:24:29,880 --> 00:24:36,880 There were two or three other areas of improvement like how to pitch their ideas to attract funding. 355 00:24:36,880 --> 00:24:38,880 So they were successful. 356 00:24:38,880 --> 00:24:42,880 They benefited from this aspect of mentorship. 357 00:24:42,880 --> 00:24:51,880 So they, you know, they learned how to properly pitch their business ideas, how to network 358 00:24:51,880 --> 00:24:52,880 and expand their network. 359 00:24:52,880 --> 00:24:54,880 So these are all things that are important. 360 00:24:54,880 --> 00:24:55,880 It's interesting. 361 00:24:55,880 --> 00:25:00,880 You're making those claims, but based on the measurements, they're not supported yet. 362 00:25:00,880 --> 00:25:05,880 I mean, if you could have an objective assessment, then you could actually support what you've 363 00:25:05,880 --> 00:25:09,880 just claimed right now is that they have the technical capacity. 364 00:25:09,880 --> 00:25:12,880 But on the other hand, you could say, who cares? 365 00:25:12,880 --> 00:25:16,880 Because in the end, your private capital worked. 366 00:25:16,880 --> 00:25:17,880 I mean, it worked. 367 00:25:17,880 --> 00:25:22,880 I mean, on the profit side, you got all the money that you developed. 368 00:25:22,880 --> 00:25:27,880 So maybe you should give yourself a break and say, it doesn't really matter because 369 00:25:27,880 --> 00:25:33,880 we managed to in the long term generate the profits through the investment. 370 00:25:33,880 --> 00:25:38,880 But if you've got your outcomes in the PMF, all we're saying is they should be more 371 00:25:38,880 --> 00:25:44,880 objectively measured so you can actually support that claim that they actually did increase 372 00:25:44,880 --> 00:25:45,880 their capacity. 373 00:25:45,880 --> 00:25:51,880 Speaking of that, the next outcome is increased capacity of investment firms to serve the 374 00:25:51,880 --> 00:25:55,880 financial needs of small, medium enterprise clients. 375 00:25:55,880 --> 00:26:01,880 And the outcome indicator for that I'll be looking at is qualitative evidence of increased 376 00:26:01,880 --> 00:26:06,880 capacity to serve the financial needs of the small, medium enterprise clients. 377 00:26:06,880 --> 00:26:14,880 And again, the issue there is the self-reporting bias through most significant change in case 378 00:26:14,880 --> 00:26:15,880 studies. 379 00:26:15,880 --> 00:26:22,880 But the other thing that's interesting is instead of asking the investment firm, another 380 00:26:22,880 --> 00:26:27,880 approach would be to go to the clients and ask them. 381 00:26:27,880 --> 00:26:31,880 And this will come up, I think, later with Benjamin on another outcome indicator is instead 382 00:26:31,880 --> 00:26:37,880 of looking at the investment firms that you're trying to develop, their increased capacity 383 00:26:37,880 --> 00:26:42,880 is to look at the clients who got the services from them and ask them what their satisfaction 384 00:26:42,880 --> 00:26:49,880 levels are on the services that were delivered by your project beneficiaries. 385 00:26:49,880 --> 00:26:56,880 And also again, again, focus on a more technical objective measure rather than the self-reporting 386 00:26:56,880 --> 00:27:02,880 What would your response be to actually going to some of the clients instead of the beneficiaries 387 00:27:02,880 --> 00:27:05,880 in the project? 388 00:27:05,880 --> 00:27:07,880 Yeah, you are right. 389 00:27:07,880 --> 00:27:10,880 So in this case, our beneficiaries were SMEs. 390 00:27:10,880 --> 00:27:16,880 So we did, it's not represented here, but we did talk to all the SMEs, like to what 391 00:27:16,880 --> 00:27:21,880 extent they feel supported by the investment firms. 392 00:27:21,880 --> 00:27:27,880 And at one level, investment firms or funds, they were also clients because of why we have 393 00:27:27,880 --> 00:27:34,880 this indicator here, because eventually this project was to raise private capital and attract 394 00:27:34,880 --> 00:27:41,880 money from private investors and then invest in these investment funds in developing countries. 395 00:27:41,880 --> 00:27:47,880 So we asked them, do they have like improved ability to serve the financial needs of clients 396 00:27:47,880 --> 00:27:51,880 in this case SMEs. 397 00:27:51,880 --> 00:27:56,880 And then we also asked SME to what extent they feel supported in terms of meeting their 398 00:27:56,880 --> 00:27:58,880 financial needs. 399 00:27:58,880 --> 00:28:03,880 But I take your point on again, on self, so there is definitely self-reporting. 400 00:28:03,880 --> 00:28:08,880 So it sounds like then you do have another indicator there. 401 00:28:08,880 --> 00:28:15,880 You actually went to the clients, the SME clients, and you asked them what do they think 402 00:28:15,880 --> 00:28:20,880 of the investment firms in terms of serving their needs. 403 00:28:20,880 --> 00:28:23,880 But interestingly enough, it's not here in the PMF. 404 00:28:23,880 --> 00:28:29,880 All we've got here is the investment firms doing self-reporting on whether or not they 405 00:28:29,880 --> 00:28:34,880 thought they had increased their capacity in serving those SME clients. 406 00:28:34,880 --> 00:28:35,880 So that's interesting. 407 00:28:35,880 --> 00:28:36,880 Okay. 408 00:28:36,880 --> 00:28:37,880 So it's not there in the PMF. 409 00:28:37,880 --> 00:28:40,880 That's unfortunate, but it is what it is. 410 00:28:40,880 --> 00:28:42,880 But that's moving in the right direction at least. 411 00:28:42,880 --> 00:28:44,880 That's good. 412 00:28:44,880 --> 00:28:45,880 Yeah. 413 00:28:45,880 --> 00:28:50,880 So just one comment, because there are so many things which are not in the PMF, which we 414 00:28:50,880 --> 00:28:52,880 did as part of our measurement. 415 00:28:52,880 --> 00:28:58,880 And that's why I sent an email like this traditional PMF is just one component. 416 00:28:58,880 --> 00:29:03,880 There were like three other different approaches we used to measure the results. 417 00:29:03,880 --> 00:29:05,880 But I take your point. 418 00:29:05,880 --> 00:29:07,880 Well, I mean, it should be in the PMF. 419 00:29:07,880 --> 00:29:10,880 I mean, all the indicators should be there to cover the outcomes. 420 00:29:10,880 --> 00:29:13,880 And I mean, that's what we're working from. 421 00:29:13,880 --> 00:29:14,880 It's easy to put it in. 422 00:29:14,880 --> 00:29:15,880 But yeah, sure. 423 00:29:15,880 --> 00:29:19,880 That's what happens, I guess, when the PMF is not adequate. 424 00:29:19,880 --> 00:29:24,880 I understand that on the economic side, but I still think Benjamin and I are strongly 425 00:29:24,880 --> 00:29:29,880 concerned about the self-reporting bias when you're talking about human beings, increasing 426 00:29:29,880 --> 00:29:34,880 the capacity of human beings in training, because a lot of the projects, as you know, 427 00:29:34,880 --> 00:29:36,880 are in training. 428 00:29:36,880 --> 00:29:41,880 So that's really where we're hoping international development organizations will start to move 429 00:29:41,880 --> 00:29:43,880 in that direction. 430 00:29:43,880 --> 00:29:46,880 But we don't want to repeat ourselves too much here. 431 00:29:46,880 --> 00:29:54,880 The next outcome is improved access to social and environmental investment services and 432 00:29:54,880 --> 00:30:03,880 financing for select small medium enterprises in Asia, Latin America and Africa. 433 00:30:03,880 --> 00:30:04,880 And again, it's the same issue. 434 00:30:04,880 --> 00:30:09,880 Now, I think I'm going to skip this because improved access is just an output. 435 00:30:09,880 --> 00:30:12,880 So, I'm going to jump and just make the note that... 436 00:30:12,880 --> 00:30:15,880 Just one comment on access. 437 00:30:15,880 --> 00:30:22,880 If you look at results-based management strategy of global affairs Canada, access is at the 438 00:30:22,880 --> 00:30:24,880 higher level, intermediate level. 439 00:30:24,880 --> 00:30:25,880 Yeah, we can dispute that. 440 00:30:25,880 --> 00:30:27,880 But I mean, when I say you're just... 441 00:30:27,880 --> 00:30:33,880 When you say number of SMEs receiving investment from local investment funds, that's just a 442 00:30:33,880 --> 00:30:35,880 delivery of a bag of money. 443 00:30:35,880 --> 00:30:37,880 And I consider that to be an output. 444 00:30:37,880 --> 00:30:41,880 I'm wondering, I'm more interested in what happens after. 445 00:30:41,880 --> 00:30:45,880 And of course, it's improved access because you're giving them something they didn't have 446 00:30:45,880 --> 00:30:47,880 before. 447 00:30:47,880 --> 00:30:51,880 But in terms of an expected outcome, that's not adequate enough. 448 00:30:51,880 --> 00:30:53,880 You go home after you deliver the money. 449 00:30:53,880 --> 00:30:57,880 What you want to know is after you deliver the money, what happens? 450 00:30:57,880 --> 00:31:04,880 And so that second part, qualitative evidence of improved access, that again, is just showing 451 00:31:04,880 --> 00:31:05,880 that they got the money. 452 00:31:05,880 --> 00:31:10,880 But what we're more interested in knowing is after that, what's the expected outcome? 453 00:31:10,880 --> 00:31:16,880 So that's why I moved to the next outcome where it says select small and medium enterprises 454 00:31:16,880 --> 00:31:23,880 and investment firms are used, quote, improved social and environmental impact measurement 455 00:31:23,880 --> 00:31:25,880 and reporting standards. 456 00:31:25,880 --> 00:31:35,880 So in that one, what I looked at is the outcome indicator is interestingly enough, it says 457 00:31:35,880 --> 00:31:38,880 they use the standards. 458 00:31:38,880 --> 00:31:43,880 And I guess what you're doing here is if I look at the indicator and you have two here, 459 00:31:43,880 --> 00:31:49,880 which is interesting, you have the perception of fund managers and small and medium enterprises 460 00:31:49,880 --> 00:31:53,880 on user friendliness of improved measurement standards. 461 00:31:53,880 --> 00:31:59,880 And you also have another indicator, number of investment funds and small and medium enterprises 462 00:31:59,880 --> 00:32:04,880 with improved social and environmental impact measurement and reporting standards. 463 00:32:04,880 --> 00:32:15,880 But the issue is, how do you know that the number reporting is actually experienced an 464 00:32:15,880 --> 00:32:21,880 improvement in social and environmental impact measurement and reporting standards? 465 00:32:21,880 --> 00:32:27,880 Like you've got the number reporting, but ideally what we want to know is within that 466 00:32:27,880 --> 00:32:33,880 number is the percent who are up to the standard that you've described. 467 00:32:33,880 --> 00:32:38,880 So you may want to elaborate on how do you get at the conclusion that the number you 468 00:32:38,880 --> 00:32:43,880 are reporting has indeed reached this standard. 469 00:32:43,880 --> 00:32:45,880 So number by itself is really not sufficient. 470 00:32:45,880 --> 00:32:50,880 You need to say number and percent of that number that actually reached the standard. 471 00:32:50,880 --> 00:32:56,880 Did you, for this indicator, did you actually have an assessment or something that you 472 00:32:56,880 --> 00:33:01,880 used to claim that the number actually equals those who've actually improved on the 473 00:33:01,880 --> 00:33:05,880 reporting standards? 474 00:33:05,880 --> 00:33:06,880 Yes. 475 00:33:06,880 --> 00:33:08,880 Yeah, we did like number and percent. 476 00:33:08,880 --> 00:33:13,880 So what we need to understand here is as part of this project, there was no social and 477 00:33:13,880 --> 00:33:20,880 environmental impact measurement reporting that investment firms or SMEs were using. 478 00:33:20,880 --> 00:33:24,880 So we came up with 22 IRS indicators. 479 00:33:24,880 --> 00:33:27,880 So this is this global impact investing network. 480 00:33:27,880 --> 00:33:32,880 So they have developed a social and environmental impact measurement system. 481 00:33:32,880 --> 00:33:35,880 And this is industry wide kind of practice. 482 00:33:35,880 --> 00:33:44,880 So we identified 22 of those and we asked these funds and SMEs to report on those IRS 483 00:33:44,880 --> 00:33:45,880 indicators. 484 00:33:45,880 --> 00:33:46,880 So that was the standard. 485 00:33:46,880 --> 00:33:51,880 So it was not that these SMEs or funds were reporting on those IRS indicators before 486 00:33:51,880 --> 00:33:53,880 participating in the project. 487 00:33:53,880 --> 00:33:58,880 So they were introduced and then we would have the quarterly reporting from them on 488 00:33:58,880 --> 00:33:59,880 those indicators. 489 00:33:59,880 --> 00:34:04,880 And along the way, so we had trainings, you know, all the coaching to them. 490 00:34:04,880 --> 00:34:06,880 So they can make improvement. 491 00:34:06,880 --> 00:34:09,880 Yes, it was done as number as well as percent. 492 00:34:09,880 --> 00:34:15,880 But the issue is, and Benjamin, having lived in Ghana for three years, I'm thinking, how 493 00:34:15,880 --> 00:34:19,880 do we know they're just not sitting under their favorite palm tree, drinking palm wine, 494 00:34:19,880 --> 00:34:26,880 ticking off the box with these indicators and that they as an organization, the people 495 00:34:26,880 --> 00:34:35,880 inside of that organization don't have any improved ability when it comes to reporting 496 00:34:35,880 --> 00:34:38,880 on social environmental impact standards? 497 00:34:38,880 --> 00:34:43,880 I mean, how do we know the report comes in every quarter, as you described, but how do 498 00:34:43,880 --> 00:34:50,880 we know within that organization if they really are knowledgeable on what we mean by 499 00:34:50,880 --> 00:34:54,880 social environmental impact? 500 00:34:54,880 --> 00:34:59,880 Yeah, we did what we call like periodic audits, which means our fund manager and then the 501 00:34:59,880 --> 00:35:03,880 impact and knowledge management team and ESD team would visit these funds. 502 00:35:03,880 --> 00:35:09,880 And at least all of them were visited maybe a few times during the life of the project. 503 00:35:09,880 --> 00:35:15,880 And we actually talked to not only SMEs, but also to their customers. 504 00:35:15,880 --> 00:35:19,880 So, for instance, when we developed those case studies, so the case studies were not 505 00:35:19,880 --> 00:35:22,880 developed just talking to these people. 506 00:35:22,880 --> 00:35:24,880 So, I can give you an example. 507 00:35:24,880 --> 00:35:28,880 So, we visited a few educational institutions, talked to students, the improvement that had 508 00:35:28,880 --> 00:35:32,880 been made and compared it with the indicators that they reported. 509 00:35:32,880 --> 00:35:35,880 So, those were some of the like verification mechanisms. 510 00:35:35,880 --> 00:35:40,880 So it begs the question, why not put that in as an indicator where you have the percentage 511 00:35:40,880 --> 00:35:45,880 of investment funds and SMEs with improved social environmental impact measurement and 512 00:35:45,880 --> 00:35:51,880 reporting standards based on that audit or based on audit reports? 513 00:35:51,880 --> 00:35:53,880 Why not change the indicator to that? 514 00:35:53,880 --> 00:36:00,880 And that would actually show that up to a minimum standard, X percent of those SMEs 515 00:36:00,880 --> 00:36:04,880 that have reached that standard and that your data source should be those audit reports, 516 00:36:04,880 --> 00:36:09,880 not just the number of reports that come in. 517 00:36:09,880 --> 00:36:14,880 Yeah, so that's why I think at the beginning I said like we do develop indicator definitions 518 00:36:14,880 --> 00:36:16,880 or elaboration sheets. 519 00:36:16,880 --> 00:36:21,880 So, it further says, okay, reports is one source of data that we have in the PMF. 520 00:36:21,880 --> 00:36:24,880 In addition to that, what else needs to be improved? 521 00:36:24,880 --> 00:36:26,880 Yeah, I think you should change the indicator here. 522 00:36:26,880 --> 00:36:33,880 You should put it to the number in percent with improved as verified by the audit reports. 523 00:36:33,880 --> 00:36:38,880 And then you can go to the data source, which is the audit reports, not just the reports 524 00:36:38,880 --> 00:36:39,880 coming in every quarter. 525 00:36:39,880 --> 00:36:44,880 And then you can go in and explain the data collection method, which is, you know, the 526 00:36:44,880 --> 00:36:46,880 audits come in, blah, blah, blah, blah, blah. 527 00:36:46,880 --> 00:36:52,880 And then when people read it, they can go, oh, okay, that's a better indicator that shows 528 00:36:52,880 --> 00:36:59,880 that your project is actually verifiably showing validly that there's an improvement. 529 00:36:59,880 --> 00:37:04,880 Yeah, I think that would be this aggregation is really nasty, in my opinion. 530 00:37:04,880 --> 00:37:09,880 You just can't aggregate up because at the bottom, at the end of the day, it's human beings 531 00:37:09,880 --> 00:37:19,880 and individuals, not institutions, not SMEs, not investment firms that this money is trying to improve. 532 00:37:19,880 --> 00:37:24,880 Yes, there's a bag of money separate that you want to bring in and have capital growth. 533 00:37:24,880 --> 00:37:25,880 I get it. 534 00:37:25,880 --> 00:37:30,880 But on the human side, you really need to stick to, you know, basic psychometrics. 535 00:37:30,880 --> 00:37:35,880 Otherwise, it just gets gets lost in the mix. 536 00:37:35,880 --> 00:37:43,880 Yeah, I don't necessarily agree on that one because some of these funds, they after now the project ended in 537 00:37:43,880 --> 00:37:52,880 2016 or 17 still funds keep reporting and many of their staff have left who were part of the trainings. 538 00:37:52,880 --> 00:37:55,880 So it has been institutionalized. So it's not about individuals. 539 00:37:55,880 --> 00:37:58,880 It's about those entities that we were trying to improve. 540 00:37:58,880 --> 00:38:05,880 But still, as we just mentioned, if they produce a report, somebody's got to look at it and see if it's up to standard. 541 00:38:05,880 --> 00:38:07,880 I mean, there's there's there's a connection. 542 00:38:07,880 --> 00:38:14,880 So you claim it's institutionalized. You've got to look at the reports they're producing and see, is it is it up to standard? 543 00:38:14,880 --> 00:38:16,880 Is it still up to standard? 544 00:38:16,880 --> 00:38:22,880 I mean, and that's where the audit, I think, comes in, where you where you have to be more precise 545 00:38:22,880 --> 00:38:29,880 and not just assume that the number that reports is it equals improved standards because it may not. 546 00:38:29,880 --> 00:38:36,880 And in particular, especially when you have turnover with staff within the firm or within the SME. 547 00:38:36,880 --> 00:38:39,880 But, you know, that's the same point. 548 00:38:39,880 --> 00:38:47,880 You just got to you got to measure a little lower on your level because once you aggregate up, we don't know where it's coming from. 549 00:38:47,880 --> 00:38:59,880 So we're going to the next outcome, which is strengthened business practices for economic growth, job creation and social impact amongst investment firms and small medium enterprises. 550 00:38:59,880 --> 00:39:02,880 I think that's me again. 551 00:39:02,880 --> 00:39:06,880 So I'm thinking it's a bit redundant here. 552 00:39:06,880 --> 00:39:12,880 But what I again talked about on this one was the qualitative evidence of strength and capacity. 553 00:39:12,880 --> 00:39:14,880 And again, the same issue. 554 00:39:14,880 --> 00:39:22,880 So I think what we'll do is we'll move on because it's the same issue that we've been discussing, which is trying to get more objective measurement. 555 00:39:22,880 --> 00:39:30,880 And we will move on to the the next outcome, which I think, Benjamin, you're doing, which is the outcome. 556 00:39:30,880 --> 00:39:31,880 Go ahead. 557 00:39:31,880 --> 00:39:35,880 Yeah, just trying to understand on 1200. 558 00:39:35,880 --> 00:39:40,880 So here we are talking about actual business practices. 559 00:39:40,880 --> 00:39:46,880 So and those whenever we do like business performance, there are standard measure in the industry. 560 00:39:46,880 --> 00:39:56,880 So if you have improved business performance, either you are creating more jobs or you are more profitable, you have expanded customer base or you have like greater inventory. 561 00:39:56,880 --> 00:39:59,880 All those kind of things are part of business performance. 562 00:39:59,880 --> 00:40:09,880 And that's what we were trying to include here because number of jobs created is one of the key for this project from social side. 563 00:40:09,880 --> 00:40:16,880 So we wanted to create jobs for women and men by these investing companies and of course underneath by there. 564 00:40:16,880 --> 00:40:22,880 So I'm trying to understand from from my own perspective, how would you measure the business performance? 565 00:40:22,880 --> 00:40:25,880 Well, I put in the summary. 566 00:40:25,880 --> 00:40:29,880 We asked the question, there is a baseline measure that measured once a year. 567 00:40:29,880 --> 00:40:36,880 And my question was, is that enough to show the project created the 15,000 jobs at 10 million dollars in tax revenue? 568 00:40:36,880 --> 00:40:46,880 How did you show that, you know, that your project funds created this and in which countries? 569 00:40:46,880 --> 00:40:53,880 Yeah, we have all the disaggregation by SMEs, by funds, by sector, all these disaggregation was there. 570 00:40:53,880 --> 00:40:55,880 Of course, we can provide here. 571 00:40:55,880 --> 00:41:00,880 But so how many women, how many men, how many youth, which company, which year? 572 00:41:00,880 --> 00:41:01,880 So all the disaggregation. 573 00:41:01,880 --> 00:41:08,880 Yes, but the challenge there, and to be fair to you, it must be very difficult and an economist could do it. 574 00:41:08,880 --> 00:41:23,880 But you would need to show that the the monies that you brought in, the financial contributions were statistically significantly greater for your project compared to SMEs of a similar size that did not get capital infusions. 575 00:41:23,880 --> 00:41:29,880 And that's a complex question to answer. 576 00:41:29,880 --> 00:41:35,880 Yeah, no, I agree. The comparison that is a valid comment across all the PMF indicators. 577 00:41:35,880 --> 00:41:46,880 And it's something, especially with investment related projects, it's hugely difficult, not only a matter of resources, but also why other companies will participate. 578 00:41:46,880 --> 00:41:56,880 But that comment is well received and definitely we will try to include in our future interventions and we are trying to the extent possible. 579 00:41:56,880 --> 00:42:02,880 For me, it doesn't really matter. More importantly, is the human side for me, the fact that the measurement isn't up to standard. 580 00:42:02,880 --> 00:42:08,880 But you could argue who cares? You created 15,000 jobs, $10 million in tax revenue. 581 00:42:08,880 --> 00:42:14,880 Who cares if another SME that wasn't involved in the project, you know, as long as you got your money back. 582 00:42:14,880 --> 00:42:23,880 And that's the other issue is you could put the 15 million you got back as part of your PMF, although that's more an efficiency issue. 583 00:42:23,880 --> 00:42:28,880 But you could give yourself credit for that. And that's why we're noting it at the beginning of the podcast. 584 00:42:28,880 --> 00:42:33,880 Let me move on to the next outcome. 585 00:42:33,880 --> 00:42:40,880 Expanded provision of investment services by investment firms to small, medium enterprise clients. 586 00:42:40,880 --> 00:42:46,880 And Benjamin, I think you're looking at an indicator there, outcome indicator for that outcome. 587 00:42:46,880 --> 00:42:55,880 Yes. So I'm looking at the outcome indicator, quality, evidence of overall improvement in investment services. 588 00:42:55,880 --> 00:43:00,880 Yeah. Also, I think we've discussed it over and over. 589 00:43:00,880 --> 00:43:10,880 When you get to this point, especially on the intermediate outcome, then the focus is going to be on whoever is receiving that service. 590 00:43:10,880 --> 00:43:16,880 They're talking about improvement in services. So I would like to hear from the client. 591 00:43:16,880 --> 00:43:20,880 And I think when you go to the ultimate outcome, then you understand this better. 592 00:43:20,880 --> 00:43:25,880 I'd like to hear from the client what was the standard of services they were receiving before. 593 00:43:25,880 --> 00:43:29,880 What is the standard of services they are receiving after the project. 594 00:43:29,880 --> 00:43:38,880 Let's do that comparison to see whether they have been some improvement or whether the quality of the services provided by SME have improved over time. 595 00:43:38,880 --> 00:43:46,880 So this is also the issue with this indicator we have here. 596 00:43:46,880 --> 00:43:51,880 Yes, sir. Your response. 597 00:43:51,880 --> 00:43:57,880 Yeah, partially agree, because if you look at the outcome is expanded provision of investment services. 598 00:43:57,880 --> 00:44:07,880 One of the provision of investment services, the capital itself, and then other investment related technical assistance that was provided to funds and SMEs. 599 00:44:07,880 --> 00:44:22,880 And those were not listed here in the indicator itself, but we did capture data through interviews with these SMEs and investment funds, but other kind of technical assistance actively received as part of the investment from the fund. 600 00:44:22,880 --> 00:44:29,880 And we don't like there was like, you know, governance practices that have been improved as a result and some other things. 601 00:44:29,880 --> 00:44:38,880 But probably it's not reflected in the PMF and we could have done it, I think maybe elaborated, but it was part of the indicator. 602 00:44:38,880 --> 00:44:44,880 Yeah, that's unfortunate because it should be in here. If you're if you're doing technical training. 603 00:44:44,880 --> 00:44:57,880 You should have an indicator on their technical capacity, objectively, not through self reporting, not through most significant change not through case studies, you should actually have a third party coming in and saying yep, they're up to standard. 604 00:44:57,880 --> 00:44:59,880 Thanks to the training. 605 00:44:59,880 --> 00:45:14,880 We can all go home now, but you got to give yourself credit if you're doing technical training support, you need to show that technically, you know, objectively rather than this qualitative evidence. 606 00:45:14,880 --> 00:45:16,880 And yes, I mean, 607 00:45:16,880 --> 00:45:29,880 I think that's why I think we've done a lot and then we don't have it in the PMF. One trick in developing this PMF is that we don't want it to be too bulky. 608 00:45:29,880 --> 00:45:44,880 But then we just have to make sure that some of the key things you are going to do is reflected in the PMF, which you have mentioned, because I mean, I'm getting a little of what you have done compared to what I see in the PMF and then the kind of instruments you are using to measure everything. 609 00:45:44,880 --> 00:46:01,880 So in subsequent ones, I think it will be good if you are able to elaborate, specify, I mean, some of the things in your PMF. If you make it much easier for a third person or someone outside the organization to understand what you are actually doing. 610 00:46:01,880 --> 00:46:08,880 What you have here now doesn't give that a language. 611 00:46:08,880 --> 00:46:22,880 Yeah, well taken. Yeah, bias and there's a point where at least you're doing self reporting bias. Yes, here we've got some projects coming up on a future podcast where all they do is take attendance and everybody goes home. 612 00:46:22,880 --> 00:46:35,880 So you're at least at the second level where you're at least attempting. You're asking the wrong people, of course, but at least you're attempting to get something out of the people that got, you know, the services from the project. So that's good. 613 00:46:35,880 --> 00:46:50,880 And we just think you should give yourself more credit than just self reporting. But, you know, we understand we get it. Sometimes you don't have the money to hire a psychologist to come in and do a proper objective, technical measurement. 614 00:46:50,880 --> 00:46:56,880 Yeah, and only one comment. 615 00:46:56,880 --> 00:47:14,880 You know, I agree with what you're saying, Benjamin. Only thing is, we consider PMF is just one instrument, it can be improved based on what you said like always having comparison group and more objective measures, but then it's a matter of resources. 616 00:47:14,880 --> 00:47:34,880 How much resources you want to put in there. But we do have for all our interventions, including this one, and this one in particular, we had a very elaborated search and learning agenda developed and which is not part of the PMF as such, but when it comes to reporting, we would always benefit from that research and learning activities. 617 00:47:34,880 --> 00:47:50,880 Yeah. But again, well, you need to put it in the PMF in terms of a more valid indicator for the outcome, because you can't have it both ways. You can't claim you're achieving these outcomes of strength and capacity and then have a weak measure of that. 618 00:47:50,880 --> 00:48:04,880 If you are doing a better measure somewhere else in the research and learning document, or wherever it is, bring it into the PMF, because that's where you're going to be measuring performance. So, you know, if you're doing it, then you should bring it in. 619 00:48:04,880 --> 00:48:10,880 That's, that's all we're really trying to get at. The last outcome. 620 00:48:10,880 --> 00:48:29,880 Sorry. Yeah, on that one, I agree, but PMF is a linear kind of framework and it has its limitations. So for instance, we usually in our projects do a lot of things around gender equality, environmental sustainability and those kind of things. 621 00:48:29,880 --> 00:48:44,880 You can't put everything in the PMF, it will become really, really difficult to manage, there will be hundred indicators and so many other things. So, you can only do with the PMF certain things, but not everything that the project is trying to do. 622 00:48:44,880 --> 00:49:02,880 Yeah, well, yeah, but in the outcomes that you have here. That's all we're saying the outcomes here are clear your statements of what you are claiming that you're achieving then you just got to make sure the indicator for that outcome is reliable and a valid measure 623 00:49:02,880 --> 00:49:20,880 and what we're saying is it's not there. Yes, you could have another 50 outcomes flying around in the project but you've already made the decision that these are the outcomes and there's only nine that you've listed, all we're saying is for those nine, make sure that your outcome indicators are tight, 624 00:49:20,880 --> 00:49:41,880 they're reliable, they're valid, that's all. But yeah, I get your point, you can have other outcomes flying around, but for the nine that you got here, you got to improve on the outcome indicator for measuring those nine, so that you can even show all the hard work you've done is attributable to the project. 625 00:49:41,880 --> 00:50:01,880 So yeah, it's yeah, I'm just repeating myself so let's move on to the last outcome. The expansion of up to 130 socially and environmentally responsible, small and medium enterprises, serving up to 5,000,000 and 50,000 women and men includes suppliers, employees 626 00:50:01,880 --> 00:50:15,880 and consumers of small and medium enterprises, small and medium enterprise goods and services in emerging frontier markets. And Benjamin, you were going to look at an outcome indicator for that outcome. 627 00:50:15,880 --> 00:50:17,880 Yes. 628 00:50:17,880 --> 00:50:24,880 So, the indicator we have here is change in select SME supply and demand. 629 00:50:24,880 --> 00:50:43,880 Which I believe is a good thing to measure. But then one thing usually is that how you pick the indicator, it will determine how you are going to measure it. That's why your indicators always need to be smart. 630 00:50:43,880 --> 00:51:01,880 It needs to be specifically measurable, achievable, realistic, time bound. Now, here I see, immediately I see it, I see that the SMEs involved in the project, we expect them to have an interest in clients. 631 00:51:01,880 --> 00:51:15,880 So when you look at the discussions we've had on the immediate and intermediate outcomes, where we talked about the clients, the clients, we want you to assess what the clients are saying about the SMEs. 632 00:51:15,880 --> 00:51:29,880 Now they all end up here talking about why this SME is going to have more clients, probably because they are satisfied with something. That is why most of them will subscribe to work with this SME. 633 00:51:29,880 --> 00:51:39,880 So this is a good indicator. But the way it has been put here, I think you can do something about it or you can work on it. 634 00:51:39,880 --> 00:51:50,880 So, let's say you can look at a number of clients or new clients that have subscribed to the services of the SME. 635 00:51:50,880 --> 00:52:00,880 So then before you implemented the project, you record the clients or customers that they have and then after the project you record the customers that they have. 636 00:52:00,880 --> 00:52:11,880 So all that we have done, all the discussions that we've had on the channels, we now deal with gender and environmental issues in the services that we provide. 637 00:52:11,880 --> 00:52:27,880 Then how did all this contribute to increase in the client base or increase in their customers? So then once you get this indicator right, it helps you to develop tools which will enable you to measure this objective. 638 00:52:27,880 --> 00:52:35,880 So I agree with you that measuring the number of clients is good, but then you can look at the indicator. 639 00:52:35,880 --> 00:52:42,880 And I mean one thing I also found out is that we are talking about expansion. So what expansion are we looking at? 640 00:52:42,880 --> 00:52:48,880 We are looking at expansion in terms of client base here. We are looking at expansion in terms of new services. 641 00:52:48,880 --> 00:52:53,880 We are looking at expansion in increased employment, which we have mentioned already. 642 00:52:53,880 --> 00:52:59,880 We are looking at expansion in terms of branches of the...all these are some of the things that you could consider. 643 00:52:59,880 --> 00:53:06,880 How probably increase in profit margin or...so when you talk about expansion abroad, sometimes you need to narrow it down. 644 00:53:06,880 --> 00:53:21,880 Here you circled on the client as the expansion you are talking about. Probably there could be other expansions or other measures which you can use to look at how these things have expanded their operations. 645 00:53:21,880 --> 00:53:24,880 Yes, here? 646 00:53:24,880 --> 00:53:30,880 Yeah, no, I think that makes sense. Yeah, we take one major one, the client base, I think. 647 00:53:30,880 --> 00:53:35,880 That's what we focused on in terms of like how they become socially and environmentally responsible. 648 00:53:35,880 --> 00:53:43,880 That was a function of like what kind of clients they have. So they are reaching to more women and other groups which they traditionally wouldn't do. 649 00:53:43,880 --> 00:53:51,880 And this was not...this is not in the PMF, but in the detailed reporting because we were receiving quarterly reports from them in terms of their client base. 650 00:53:51,880 --> 00:53:57,880 So that's how we reported on this outcome. But I get your point. 651 00:53:57,880 --> 00:54:01,880 Benjamin, any further comment? 652 00:54:01,880 --> 00:54:18,880 No, I mean, the other comment I'll make is that talking to him, I get a clear understanding of their...so in future it will be good to just try to make things clear so that whoever thinks it will be able to get what we are trying to make. 653 00:54:18,880 --> 00:54:34,880 Okay, so this has been a very interesting discussion. And as usual at the end of these episodes, we usually make an assessment, but I think we've made our points throughout each outcome indicator that we've raised. 654 00:54:34,880 --> 00:54:55,880 So we won't repeat that. But we do want to know, yes sir, if your organization would be willing to consider in the future all your performance measurement frameworks, if you'd be willing to put them on your website, both the blank ones, which we have, but also the data that as it comes in for each of your indicators. 655 00:54:55,880 --> 00:55:16,880 If you'd be willing to do that, we understand, of course, the government of Canada does not require you to do that. And in fact, we're advocating that the government of Canada put on their website, all of the performance measurement frameworks for all of their projects, you know, roughly $4 billion a year. 656 00:55:16,880 --> 00:55:27,880 What would you be able to, I know you don't represent the organization, but would that be something that you could look at? 657 00:55:27,880 --> 00:55:48,880 Yeah, definitely. I think we have some like partial PMF data in terms of impact on our website. So, but internally we had a discussion, maybe we should put PMF data on our web. There are like issues related to like data privacy and all that. We could do aggregated data, I don't see any harm in it. 658 00:55:48,880 --> 00:56:13,880 If we are not providing individual SME or individual funds name, I think it is possible. So we are having internal discussions, how we could do that. So yeah, that is something up for discussion. So I can't promise anything, but partially I think we will have some impact. 659 00:56:13,880 --> 00:56:35,880 Well, they say the first quality of leadership is to lead. So if you did that, you would be a pioneer, because I don't think there are many Canadian NGOs are out there putting their blank PMFs on their websites, as well as the data for the indicators from those performance measurement frameworks. 660 00:56:35,880 --> 00:56:50,880 It would be great because then they could actually show that they are actually achieving their outcomes. That is assuming that they are properly designing their outcome indicators to measure reliably and validly those achievements of those outcomes. 661 00:56:50,880 --> 00:57:12,880 So it is a movement in the right direction. It would be great. At the same time, at the end of this podcast, we will be sending both parts one and part two, which is today, to the Minister for International Development, as well as the International Development Shadow Critics for the Conservative Opposition and the NDP. 662 00:57:12,880 --> 00:57:37,880 And we'll also be copying you on that, along with the PMF and the Excel summary sheet. And we'll be advocating for exactly that. That they, one, put the PMFs for all projects that the Government of Canada funds on a Government of Canada website, preferably the Global Affairs Canada project browser, because that's where all the projects are. 663 00:57:37,880 --> 00:57:55,880 And number two, that they improve their own standards, because if you know Global Affairs Canada, their own standards on designing the indicators in a performance measurement framework is to stop doing this qualitative self-reporting bias and use objective measures. 664 00:57:55,880 --> 00:58:11,880 It's in their own guidelines, but they're not following them, as you can tell from a lot of performance measurement frameworks that we review on this podcast. So that's the second point we're trying to say is improve the design of the outcome indicators and then make them public. 665 00:58:11,880 --> 00:58:28,880 So that's where we're at. We want to thank you for taking your time to respond to our critique of your performance measurement framework. And we look forward to maybe having you on the podcast again for another PMF. 666 00:58:28,880 --> 00:58:37,880 Yeah, no, I'll be happy to do that. And thanks, David, for providing the opportunity. And thanks, Benjamin, for your comments as well. 667 00:58:37,880 --> 00:58:43,880 Thank you both, and stay tuned for episode six. 668 00:58:43,880 --> 00:59:11,880 And bye for now.