1 00:00:00,000 --> 00:00:05,000 Welcome to the Improving Development Evaluation Podcast. 2 00:00:05,000 --> 00:00:14,000 I'm your host David Wand and welcome to Episode 3, Part 2, where we continue our discussion 3 00:00:14,000 --> 00:00:22,700 about the United Nations Population Fund South Sudan Project, which is called Human Resources 4 00:00:22,700 --> 00:00:27,200 for Sexual Reproductive Health in South Sudan. 5 00:00:27,200 --> 00:00:34,800 As you may recall from Part 1, I reviewed 16 outcome indicators covering 7 outcomes 6 00:00:34,800 --> 00:00:41,200 and came to the conclusion that the South Sudan UNFPA Country Office could not make 7 00:00:41,200 --> 00:00:47,960 the claim that they were achieving their project outcomes simply because their outcome indicators 8 00:00:47,960 --> 00:00:51,380 were not properly measuring their outcomes. 9 00:00:51,380 --> 00:00:58,840 On November 18th, I sent to the UNFPA South Sudan Country Office through their contact 10 00:00:58,840 --> 00:01:04,280 form an invitation to attend the podcast today on December 14th. 11 00:01:04,280 --> 00:01:08,200 But as of today, we have not received a reply from UNFPA. 12 00:01:08,200 --> 00:01:15,100 I also sent it the same invitation to the headquarters in New York, HQ at UNFPA.org, 13 00:01:15,100 --> 00:01:23,240 as well as the East South African Regional Office at isarro.info at UNFPA.org. 14 00:01:23,240 --> 00:01:29,740 And I also sent it through a contact form at UNFPA.org in New York, indicating it should 15 00:01:29,740 --> 00:01:33,300 go to the evaluation office in New York. 16 00:01:33,300 --> 00:01:39,160 Even despite those four contacts, I haven't received a reply from any of them regarding 17 00:01:39,160 --> 00:01:42,000 this invitation. 18 00:01:42,000 --> 00:01:49,860 But fortunately, we do have an evaluation expert with us today from Accra, Ghana. 19 00:01:49,860 --> 00:01:56,280 His name is Benjamin Serbo and he holds a master's degree in political science from 20 00:01:56,280 --> 00:01:59,600 the University of Zululand in South Africa. 21 00:01:59,600 --> 00:02:02,520 And he is a monitoring and evaluation specialist. 22 00:02:02,520 --> 00:02:09,640 And we're glad to have him on the podcast to provide more insight into the flaws in 23 00:02:09,640 --> 00:02:15,080 the performance measurement framework for this project that the UNFPA is delivering 24 00:02:15,080 --> 00:02:16,440 in South Sudan. 25 00:02:16,440 --> 00:02:17,920 Welcome, Benjamin. 26 00:02:17,920 --> 00:02:18,920 How are you? 27 00:02:18,920 --> 00:02:19,920 Yes, I'm good. 28 00:02:19,920 --> 00:02:20,920 Thank you. 29 00:02:20,920 --> 00:02:21,920 And yourself? 30 00:02:21,920 --> 00:02:22,920 I'm fine. 31 00:02:22,920 --> 00:02:27,240 I'm very happy that you could join us. 32 00:02:27,240 --> 00:02:34,180 So what we're going to do now is we're going to go through each of the seven outcomes 33 00:02:34,180 --> 00:02:37,360 and to make sure we cover the whole spectrum of the project. 34 00:02:37,360 --> 00:02:47,880 And then if we have additional time, we'll go back to some of the other outcome indicators. 35 00:02:47,880 --> 00:02:57,160 So starting with the outcome in this project, improved national and state government capacities 36 00:02:57,160 --> 00:03:05,280 to develop and implement sexual reproductive health, gender-based violence, HIV policies, 37 00:03:05,280 --> 00:03:08,800 legislation and coordination mechanisms. 38 00:03:08,800 --> 00:03:12,320 And Benjamin, you're going to be taking a look at an outcome indicator for that outcome. 39 00:03:12,320 --> 00:03:13,320 Okay, you're right. 40 00:03:13,320 --> 00:03:15,480 Thank you very much. 41 00:03:15,480 --> 00:03:26,640 So the outcome indicator for the number of state coordinating mechanisms and coordination 42 00:03:26,640 --> 00:03:27,640 plan. 43 00:03:27,640 --> 00:03:30,760 David, is that correct? 44 00:03:30,760 --> 00:03:31,760 That's correct. 45 00:03:31,760 --> 00:03:38,880 So first of all, let me say this, when you talk about the state coordinating mechanisms, 46 00:03:38,880 --> 00:03:45,560 it is not only the project which is going to develop the state coordinating mechanisms. 47 00:03:45,560 --> 00:03:49,560 And the number of guests who are going to develop this mechanism. 48 00:03:49,560 --> 00:03:53,360 And so the project cannot claim that. 49 00:03:53,360 --> 00:04:00,660 If you're not paying those state mechanisms for it, then you can sue the project. 50 00:04:00,660 --> 00:04:08,360 So then you need to look at the kind of contribution that the project made. 51 00:04:08,360 --> 00:04:13,960 So for instance, when you look at having an hour with black, that's not necessarily mean 52 00:04:13,960 --> 00:04:18,600 you supported or you did it, you supported it. 53 00:04:18,600 --> 00:04:25,880 As with either some of the areas that the project will have to look at. 54 00:04:25,880 --> 00:04:32,160 Now, one thing we need to know is that when you talk about having a state coordinating 55 00:04:32,160 --> 00:04:38,680 mechanism, and I know it will not be able to justify how the project was continued, 56 00:04:38,680 --> 00:04:40,720 improvement of state capacities. 57 00:04:40,720 --> 00:04:50,580 Rather, I would like to see the number of project officials who contributed in the development 58 00:04:50,580 --> 00:04:55,080 and the implementation of those annual coordinating. 59 00:04:55,080 --> 00:05:01,280 Because when you are conducting an evaluation of the project and you know how the project 60 00:05:01,280 --> 00:05:07,840 contributed to improving state and national capacities, you have to assess how the beneficiaries 61 00:05:07,840 --> 00:05:14,720 of your project played a direct role in the development of those mechanisms. 62 00:05:14,720 --> 00:05:22,160 So one of the indicators we can look at in this, for instance, the number of project 63 00:05:22,160 --> 00:05:29,960 beneficiaries who participated in the implementation of the annual coordinating. 64 00:05:29,960 --> 00:05:37,080 So assuming the five of your project beneficiaries participated, then we can get some qualitative 65 00:05:37,080 --> 00:05:44,160 information that can support the provided to the development. 66 00:05:44,160 --> 00:05:50,480 So in that case, we'll be able to know the contribution that our project made to strengthen 67 00:05:50,480 --> 00:05:52,200 the state. 68 00:05:52,200 --> 00:05:55,320 So these are some things that we can look out for. 69 00:05:55,320 --> 00:06:02,720 But if we just did the number of state mechanisms, annual coordination, yes, we can have the 70 00:06:02,720 --> 00:06:03,720 number. 71 00:06:03,720 --> 00:06:13,480 The project will not be able to clearly demonstrate how it contributed to the development of the 72 00:06:13,480 --> 00:06:18,120 mechanisms or development of the annual. 73 00:06:18,120 --> 00:06:20,320 That's a good point. 74 00:06:20,320 --> 00:06:23,800 And also it seems to be misplaced, the indicator. 75 00:06:23,800 --> 00:06:29,640 It's not quite right, let you say, is focusing on who there's no who in here. 76 00:06:29,640 --> 00:06:36,280 And so, yeah, it should absolutely be specific to the project participants who were trained 77 00:06:36,280 --> 00:06:38,160 on developing these mechanisms. 78 00:06:38,160 --> 00:06:40,080 Yes, that's a good point. 79 00:06:40,080 --> 00:06:42,680 And maybe you're going to say the next part. 80 00:06:42,680 --> 00:06:48,000 So they also, maybe I will just add, if you look at the outcome, it's very clear that 81 00:06:48,000 --> 00:06:52,080 they want to improve capacity. 82 00:06:52,080 --> 00:06:54,960 And just saying number is not good enough. 83 00:06:54,960 --> 00:07:00,160 You have to show that these mechanisms and coordination plans have actually been improved 84 00:07:00,160 --> 00:07:02,020 in terms of quality. 85 00:07:02,020 --> 00:07:07,160 So once you get the project participants involved, then you have to show that from the trainings 86 00:07:07,160 --> 00:07:13,480 that the project participants received, they actually improve the quality of these coordination 87 00:07:13,480 --> 00:07:16,440 mechanisms or annual coordination plans. 88 00:07:16,440 --> 00:07:19,600 Perhaps you were going to say that next. 89 00:07:19,600 --> 00:07:20,600 Yeah. 90 00:07:20,600 --> 00:07:21,600 Okay. 91 00:07:21,600 --> 00:07:23,560 So that's great. 92 00:07:23,560 --> 00:07:26,480 So it really actually shows two things that are missing. 93 00:07:26,480 --> 00:07:32,080 It's misplaced, the indicator, and it's not measuring increased levels of capacity. 94 00:07:32,080 --> 00:07:33,080 That's great. 95 00:07:33,080 --> 00:07:39,760 So let's move to the next outcome, which is strengthened capacities of health training 96 00:07:39,760 --> 00:07:46,880 institutions to produce quality human resources for sexual reproductive health rights, gender 97 00:07:46,880 --> 00:07:49,920 based violence, and HIV services. 98 00:07:49,920 --> 00:07:52,720 And Benjamin, I think you were going to look at an indicator there. 99 00:07:52,720 --> 00:07:53,720 Okay. 100 00:07:53,720 --> 00:08:00,880 So with this indicator, with this outcome, they indicate in the outcome proportion of 101 00:08:00,880 --> 00:08:11,240 targeted health training institutions, MOUs signed with targeted practical site. 102 00:08:11,240 --> 00:08:19,360 And the number of supportive machines conducted at a health teaching. 103 00:08:19,360 --> 00:08:27,280 With this outcome also, now let's indicate the proportion of targeted health institutions, 104 00:08:27,280 --> 00:08:28,600 MOUs. 105 00:08:28,600 --> 00:08:37,520 Strengthening MOU does not necessarily translate into strengthening the health institution. 106 00:08:37,520 --> 00:08:50,600 It doesn't show any, because MOUs now by health institutions has strengthened. 107 00:08:50,600 --> 00:08:58,360 So then when we talk about strengthened capacities, we are looking at how the training institute 108 00:08:58,360 --> 00:09:06,680 participating in the project have improved their structure, process, human resources 109 00:09:06,680 --> 00:09:13,200 to deliver quality training to various healthcare providers. 110 00:09:13,200 --> 00:09:20,000 So just finding MOUs conducting supportive visits, that's not the most ridiculous. 111 00:09:20,000 --> 00:09:22,680 So we're looking at these two indicators. 112 00:09:22,680 --> 00:09:28,520 We cannot use it to measure the strength and capacity of the health institute. 113 00:09:28,520 --> 00:09:34,040 So and liberal you rightly mentioned, we can measure the achievement by looking at the 114 00:09:34,040 --> 00:09:41,200 percentage of faculty that's reached certain level of technical capacity. 115 00:09:41,200 --> 00:09:46,360 So in that case, the UNFPA considered a freeway index. 116 00:09:46,360 --> 00:09:54,640 Number one, can you get a percentage of teachers or trainers of beneficial health, who meet 117 00:09:54,640 --> 00:09:59,640 the required national or international certification for expertise. 118 00:09:59,640 --> 00:10:07,000 Usually we have this certification which can be used by the national. 119 00:10:07,000 --> 00:10:12,960 So in order for this to be strengthened, we need the people working in this institution 120 00:10:12,960 --> 00:10:18,440 to find all the expertise. 121 00:10:18,440 --> 00:10:25,680 So then in that case, we can say our institution, the institution has strengthened with the 122 00:10:25,680 --> 00:10:28,160 staff achieving these issues. 123 00:10:28,160 --> 00:10:30,720 Even that's one indicator. 124 00:10:30,720 --> 00:10:38,320 Now another indicator you can number health training institutions, who are currently meet 125 00:10:38,320 --> 00:10:42,580 the required industry standards. 126 00:10:42,580 --> 00:10:52,040 So I believe the training curriculum for the HRHRN keeping as opposed to meet certain standards. 127 00:10:52,040 --> 00:10:58,800 Obviously there will be criteria to determine it meets the standards. 128 00:10:58,800 --> 00:11:04,720 So the institute meeting the standards be a very good indication of its strength and 129 00:11:04,720 --> 00:11:10,640 capacity to deliver quality training and produce quality human resources. 130 00:11:10,640 --> 00:11:16,920 Actually this is another indicator, those can be considered. 131 00:11:16,920 --> 00:11:24,680 And then other areas that can be considered are the satisfaction of the health care that 132 00:11:24,680 --> 00:11:26,440 will receive training. 133 00:11:26,440 --> 00:11:32,960 And even the beneficiary of the gender based fund is and then sexual and reproductive health 134 00:11:32,960 --> 00:11:40,720 service from this health care by this because when you get a fee from the institute, they 135 00:11:40,720 --> 00:11:45,000 will satisfy the kind of service that we provide. 136 00:11:45,000 --> 00:11:51,280 It's going to help you to mention or to determine whether the institutions are strengthened 137 00:11:51,280 --> 00:12:00,680 and whether these institutions are providing quality, GVVN, HRHRN services to the brand 138 00:12:00,680 --> 00:12:03,640 and to the people. 139 00:12:03,640 --> 00:12:10,520 So these are the indicators. 140 00:12:10,520 --> 00:12:11,520 Thank you, Benjamin. 141 00:12:11,520 --> 00:12:12,520 That's great. 142 00:12:12,520 --> 00:12:19,720 And so what we know here is that the indicator in its current form is not measuring the outcome 143 00:12:19,720 --> 00:12:20,720 at all. 144 00:12:20,720 --> 00:12:21,720 At all. 145 00:12:21,720 --> 00:12:23,400 It is not measuring the outcome. 146 00:12:23,400 --> 00:12:24,400 That's right. 147 00:12:24,400 --> 00:12:30,920 Yeah, an MOU signed is clearly not a measure of strength and capacity. 148 00:12:30,920 --> 00:12:31,920 Great. 149 00:12:31,920 --> 00:12:32,920 Okay. 150 00:12:32,920 --> 00:12:38,640 So we'll move on to the next outcome, which is increased awareness among women and girls 151 00:12:38,640 --> 00:12:47,400 to utilize gender responsive, sexual reproductive health, gender based violence and HIV services. 152 00:12:47,400 --> 00:12:51,080 And there are two outcome indicators for this outcome. 153 00:12:51,080 --> 00:12:57,840 And I chose the second one, proportion of women, girls from targeted communities who 154 00:12:57,840 --> 00:13:06,480 report making their own decision for seeking sexual health services disaggregated by age. 155 00:13:06,480 --> 00:13:07,760 I correct myself, Benjamin. 156 00:13:07,760 --> 00:13:09,280 I've seen my notes. 157 00:13:09,280 --> 00:13:11,280 This is yours. 158 00:13:11,280 --> 00:13:12,920 Sorry. 159 00:13:12,920 --> 00:13:13,920 Go ahead. 160 00:13:13,920 --> 00:13:17,240 That is mine, right? 161 00:13:17,240 --> 00:13:23,280 The outcome indicator is proportion of women, girls from targeted communities who report 162 00:13:23,280 --> 00:13:30,080 making their own decision for seeking sexual health services disaggregated by age. 163 00:13:30,080 --> 00:13:33,740 So we are talking about increased awareness here. 164 00:13:33,740 --> 00:13:42,040 So when you talk about it, increased awareness, it will mean the procedures and processes 165 00:13:42,040 --> 00:13:50,520 they need to give you the HRH services. 166 00:13:50,520 --> 00:13:55,120 So then this could lead to reported cases. 167 00:13:55,120 --> 00:13:59,640 And this is something I will see. 168 00:13:59,640 --> 00:14:06,440 It was awareness to resolve a number of cases reported. 169 00:14:06,440 --> 00:14:13,880 They understand the processes, you ensuring that their awareness has been treated. 170 00:14:13,880 --> 00:14:18,600 There are GBV cases. 171 00:14:18,600 --> 00:14:21,800 So now let's look at this. 172 00:14:21,800 --> 00:14:29,640 With the original indicator, you don't have any comparison in there. 173 00:14:29,640 --> 00:14:34,120 So now your project is getting its awareness. 174 00:14:34,120 --> 00:14:39,520 It can happen to the natural population somewhere else. 175 00:14:39,520 --> 00:14:46,800 And then they also go in a student. 176 00:14:46,800 --> 00:14:55,200 So then if you have a comparison, you will be able to extend to which your project made 177 00:14:55,200 --> 00:14:56,200 the difference. 178 00:14:56,200 --> 00:15:05,040 So with it, current state without Paris, it will be difficult for the project to make 179 00:15:05,040 --> 00:15:09,760 the contribution to creating awareness. 180 00:15:09,760 --> 00:15:18,840 So in order to ensure that we are to be confident, the project is strong, patient awareness. 181 00:15:18,840 --> 00:15:39,560 We need to make sure that the patient is aware of the patient's awareness. 182 00:15:39,560 --> 00:15:41,640 Yes, thank you, Benjamin. 183 00:15:41,640 --> 00:15:46,800 And if you look at the performance measurement framework for this indicator, they do measure 184 00:15:46,800 --> 00:15:58,400 it three times, baseline, midline, and I think they call it end of the project. 185 00:15:58,400 --> 00:16:03,000 And you'd think, oh, well, if we can show that the percent of women reporting this has 186 00:16:03,000 --> 00:16:07,640 gone up, that's good enough, but not necessarily like you pointed out. 187 00:16:07,640 --> 00:16:13,520 So here we have in the performance measurement framework, three measures for this indicator, 188 00:16:13,520 --> 00:16:17,640 a baseline, midline, and end of project. 189 00:16:17,640 --> 00:16:23,800 But even if the percentage of women who report utilizing services has gone up, which is good, 190 00:16:23,800 --> 00:16:28,680 it could be going up even greater somewhere else in the country, like you said, where 191 00:16:28,680 --> 00:16:34,840 the awareness campaign and money spent by this project to increase awareness like radio 192 00:16:34,840 --> 00:16:35,840 ads, brochures. 193 00:16:35,840 --> 00:16:42,360 I'd have to go back to part one to find out exactly what I said, how they increased awareness. 194 00:16:42,360 --> 00:16:49,800 But still, you need a comparison group to show that the project is doing it better. 195 00:16:49,800 --> 00:16:52,400 So yeah, that's a good point. 196 00:16:52,400 --> 00:17:02,800 Okay, so we can move on to the next outcome, which is strengthening capacity of targeted 197 00:17:02,800 --> 00:17:09,480 national and state level health facilities to provide quality, integrated sexual reproductive 198 00:17:09,480 --> 00:17:13,440 health, gender-based violence, and HIV services. 199 00:17:13,440 --> 00:17:19,800 And the indicator I looked at here, there's only really one, it's a cluster of, I think, 200 00:17:19,800 --> 00:17:20,800 three. 201 00:17:20,800 --> 00:17:26,200 And those three are number of visits for family planning services in targeted locations, number 202 00:17:26,200 --> 00:17:33,040 of visits for antenatal clinics in targeted locations, and number of deliveries by skilled 203 00:17:33,040 --> 00:17:36,480 birth attendants in targeted locations. 204 00:17:36,480 --> 00:17:42,400 And the problem here with these indicators for this outcome is they're not really measuring, 205 00:17:42,400 --> 00:17:48,680 as we've discussed earlier on another outcome, they're not measuring strength and capacity 206 00:17:48,680 --> 00:17:51,520 of the health facilities. 207 00:17:51,520 --> 00:17:57,340 And if you look at part one or listen to part one and look at the project, the number needs 208 00:17:57,340 --> 00:18:05,160 to be changed to the number and percent of targeted facilities that have solar power, 209 00:18:05,160 --> 00:18:12,120 have water supply, have adequate staff trained to maintain the solar and water supply that 210 00:18:12,120 --> 00:18:18,280 have sufficient essential health workers that can deliver gender responsive sexual reproductive 211 00:18:18,280 --> 00:18:22,560 health services, gender-based violence services, and HIV services. 212 00:18:22,560 --> 00:18:25,240 So the number is just inadequate. 213 00:18:25,240 --> 00:18:32,800 You need to go a step further and figure out the percent of staff that actually have the 214 00:18:32,800 --> 00:18:41,800 capacity to provide the services in a minimum quality way, because it says quality integrated 215 00:18:41,800 --> 00:18:42,800 services. 216 00:18:42,800 --> 00:18:48,680 So just counting the number of visits or the number of deliveries or the number of visits 217 00:18:48,680 --> 00:18:52,960 for family planning or antenatal services is not good enough. 218 00:18:52,960 --> 00:18:58,840 And that's why it falls into the problem area of not related to the outcome of showing strengthened 219 00:18:58,840 --> 00:19:02,720 capacity of these targeted health facilities. 220 00:19:02,720 --> 00:19:10,840 And the other point is, even if they modify the indicator to have a criteria of what is 221 00:19:10,840 --> 00:19:19,000 an acceptable minimum percent of staff trained at each facility that knows how to repair 222 00:19:19,000 --> 00:19:27,520 and maintain the solar power panels and the water supply, as well as having a minimum 223 00:19:27,520 --> 00:19:36,080 number of staff that have adequate skill levels to deliver the services, you could argue that 224 00:19:36,080 --> 00:19:43,640 in another part of the country where the project's not operating, they've got even a higher percentage 225 00:19:43,640 --> 00:19:51,200 of health facilities that have strengthened their capacity to deliver services. 226 00:19:51,200 --> 00:19:58,480 So that's basically the issue for that outcome indicator. 227 00:19:58,480 --> 00:20:07,440 Now the next one is the next outcome is strengthened enabling environment at the national, state, 228 00:20:07,440 --> 00:20:13,680 and local level for women and girls access to sexual reproductive health and gender based 229 00:20:13,680 --> 00:20:16,000 violence services. 230 00:20:16,000 --> 00:20:20,600 And we've talked about this earlier, so it kind of overlaps. 231 00:20:20,600 --> 00:20:22,800 But it's the same issue. 232 00:20:22,800 --> 00:20:30,080 The indicator here is the proportion of targeted health science institutions that meet the 233 00:20:30,080 --> 00:20:35,440 minimum educational standard for training health professionals. 234 00:20:35,440 --> 00:20:37,640 Now that's an excellent indicator. 235 00:20:37,640 --> 00:20:46,400 But the problem is we don't know inside of these institutions what's going on. 236 00:20:46,400 --> 00:20:53,680 And in the performance measurement framework, they actually talk about the staff that train 237 00:20:53,680 --> 00:20:57,080 these health professionals being assessed. 238 00:20:57,080 --> 00:21:03,760 And there actually is progress reports based on those assessments being produced. 239 00:21:03,760 --> 00:21:11,920 But we don't know what sort of measurement tool they're using to measure these staff, 240 00:21:11,920 --> 00:21:19,160 to say that they've reached a minimum educational standard, and then for that one health institution, 241 00:21:19,160 --> 00:21:24,760 we don't know if that's sufficient to conclude that the health science institution all by 242 00:21:24,760 --> 00:21:29,300 itself has reached the minimum educational standard. 243 00:21:29,300 --> 00:21:33,960 And like you had mentioned earlier, Benjamin, another indicator could be even looking at 244 00:21:33,960 --> 00:21:39,880 the curriculum content to make sure it's up to a minimum educational standard. 245 00:21:39,880 --> 00:21:43,900 So unfortunately, UNFPA are not here to explain. 246 00:21:43,900 --> 00:21:48,980 And I've looked at their evaluation plan, and there's no measurement tool there talking 247 00:21:48,980 --> 00:21:51,760 about how they come up with this percent. 248 00:21:51,760 --> 00:22:00,300 This is despite the fact that they have a target of 80% of health science institutions 249 00:22:00,300 --> 00:22:03,880 reaching this minimum educational standard. 250 00:22:03,880 --> 00:22:08,680 So we need to ask the UNFPA, how are they measuring the staff? 251 00:22:08,680 --> 00:22:15,400 And coming up with this calculation to conclude that this is the percent of targeted health 252 00:22:15,400 --> 00:22:21,640 facilities that meet the targeted health science institutions that meet this minimum educational 253 00:22:21,640 --> 00:22:23,880 standard for training health professionals. 254 00:22:23,880 --> 00:22:27,080 We simply don't know how they calculate that. 255 00:22:27,080 --> 00:22:34,320 So it's not adequate or related to the outcome of saying we have strengthened the enabling 256 00:22:34,320 --> 00:22:41,360 environment for women and girls access to services, because we assume when they graduate 257 00:22:41,360 --> 00:22:47,100 these midwives, that they're up to standard, and that the women and girls who get services 258 00:22:47,100 --> 00:22:53,500 from them have experienced a strengthened enabling environment. 259 00:22:53,500 --> 00:22:57,460 But we just don't know. 260 00:22:57,460 --> 00:23:04,340 The next outcome is increased use of gender responsive sexual reproductive health rights 261 00:23:04,340 --> 00:23:09,840 and gender based violence services by women and girls, especially those living in vulnerable 262 00:23:09,840 --> 00:23:12,160 situations. 263 00:23:12,160 --> 00:23:19,240 And the outcome indicator for that is the proportion of women aged 15 to 49 who receive 264 00:23:19,240 --> 00:23:24,960 four or more antenatal visits. 265 00:23:24,960 --> 00:23:40,440 Now this again has issues here where they clearly have targeted sites per targeted locations. 266 00:23:40,440 --> 00:23:48,800 So we need to measure these same indicators for areas in the country of South Sudan where 267 00:23:48,800 --> 00:23:55,160 the project is not operating so that we can show that the proportion of women who receive 268 00:23:55,160 --> 00:24:02,600 four or more antenatal visits is higher in the project area for this project, $37 million, 269 00:24:02,600 --> 00:24:08,400 compared to health facilities that haven't been targeted. 270 00:24:08,400 --> 00:24:16,920 Because one of the services is actually to deploy midwives from these institutions to 271 00:24:16,920 --> 00:24:19,080 these targeted health facilities. 272 00:24:19,080 --> 00:24:27,800 So we would expect the percentage of women in that catchment area for that targeted health 273 00:24:27,800 --> 00:24:31,520 facility to get greater access. 274 00:24:31,520 --> 00:24:36,480 So the only way to know that is to have a comparison group, and we don't have any comparison 275 00:24:36,480 --> 00:24:40,720 group mentioned in this performance measurement framework. 276 00:24:40,720 --> 00:24:46,200 So that's something that's important to recognize is that it's really important to have comparison 277 00:24:46,200 --> 00:24:52,800 groups when they are even explicitly saying we are targeting particular locations, which 278 00:24:52,800 --> 00:24:55,960 means other locations we're not targeting. 279 00:24:55,960 --> 00:25:01,500 So the project should be showing that in the targeted locations, they're getting greater 280 00:25:01,500 --> 00:25:07,800 rates of service where higher percentage of women are receiving four or more antenatal 281 00:25:07,800 --> 00:25:08,800 visits. 282 00:25:08,800 --> 00:25:10,280 And the final outcome- 283 00:25:10,280 --> 00:25:12,320 And David, if I can- 284 00:25:12,320 --> 00:25:13,320 Yes, jump in. 285 00:25:13,320 --> 00:25:23,600 I mean, yeah, usually when we get to the outcome level, I usually want to see some kind of 286 00:25:23,600 --> 00:25:27,600 feedback from whoever is benefiting. 287 00:25:27,600 --> 00:25:36,280 So for instance, we are looking at proportional age 15 to 14 who receive or antenatal visit. 288 00:25:36,280 --> 00:25:44,000 Assuming they receive this for more than one, so then on the outcome level, I would like 289 00:25:44,000 --> 00:25:50,680 to see those women who are receiving this, they can observe it. 290 00:25:50,680 --> 00:25:57,440 They receive from this, but to what extent are they satisfied with this kind of service? 291 00:25:57,440 --> 00:26:04,000 So in a matter of fact, they receive the services that normally are not required to function. 292 00:26:04,000 --> 00:26:12,480 Then we want to high number of women or patients receiving this to express satisfaction. 293 00:26:12,480 --> 00:26:17,400 So if we have like only 40% of them being satisfied, then we have a problem. 294 00:26:17,400 --> 00:26:21,360 So then on the outcome level, these are areas that we can look at. 295 00:26:21,360 --> 00:26:27,000 Those who are receiving the service, they are at a level of satisfaction with the service. 296 00:26:27,000 --> 00:26:29,080 Yeah, David, back to you. 297 00:26:29,080 --> 00:26:33,000 That's an excellent point because it reminds me of a point I missed. 298 00:26:33,000 --> 00:26:39,040 The indicator actually says increased use of gender responsive sexual reproductive health 299 00:26:39,040 --> 00:26:41,480 and gender-based violence services. 300 00:26:41,480 --> 00:26:48,160 So that implies that they not only are receiving the service, like you said, but they're satisfied 301 00:26:48,160 --> 00:26:52,480 that the services being delivered are gender responsive. 302 00:26:52,480 --> 00:26:57,640 And that's a big focus of the current government of Canada in a lot of these projects they 303 00:26:57,640 --> 00:27:03,440 fund is not only getting them trained up to capacity to deliver the service, but that 304 00:27:03,440 --> 00:27:05,800 it's gender responsive. 305 00:27:05,800 --> 00:27:08,140 And that's very precise in the outcome. 306 00:27:08,140 --> 00:27:09,680 And you're quite right. 307 00:27:09,680 --> 00:27:13,440 In the outcome indicator, they have missed that. 308 00:27:13,440 --> 00:27:15,280 So that's a very good point. 309 00:27:15,280 --> 00:27:21,520 And that's also needs to be noted in the indicator, not only just the proportion of women who 310 00:27:21,520 --> 00:27:31,560 report getting the service, but the proportion of women who report satisfaction with the 311 00:27:31,560 --> 00:27:35,560 service being delivered being gender responsive. 312 00:27:35,560 --> 00:27:39,000 Yeah, good point. 313 00:27:39,000 --> 00:27:46,780 And the last outcome of the project is as we go up the results chain is the increased 314 00:27:46,780 --> 00:27:54,400 realization of sexual and reproductive health and rights of women and girls in South Sudan. 315 00:27:54,400 --> 00:27:57,000 And the indicators there are excellent. 316 00:27:57,000 --> 00:27:59,400 They're universally agreed upon. 317 00:27:59,400 --> 00:28:07,160 The two indicators are maternal mortality ratio and the unmet need for family planning. 318 00:28:07,160 --> 00:28:12,580 And UNFPA is very, very good at measuring these. 319 00:28:12,580 --> 00:28:19,140 They do large surveys using standard public health surveys. 320 00:28:19,140 --> 00:28:27,680 But the problem here is the one we mentioned earlier is how do we show that the $37 million 321 00:28:27,680 --> 00:28:36,080 that the Canadian taxpayer paid for this project has reduced the maternal deaths at a lower 322 00:28:36,080 --> 00:28:44,440 rate than women and girls who unfortunately did not benefit from this project. 323 00:28:44,440 --> 00:28:49,000 Now UNFPA may come back and say, it's a universal project. 324 00:28:49,000 --> 00:28:52,880 It goes all across the country. 325 00:28:52,880 --> 00:28:57,940 But my understanding is no, because they mentioned targeted locations. 326 00:28:57,940 --> 00:29:06,660 So that means there's an opportunity to pick a non-project group of women and girls and 327 00:29:06,660 --> 00:29:10,880 measure their maternal mortality ratios also. 328 00:29:10,880 --> 00:29:14,200 So otherwise you don't have evaluation. 329 00:29:14,200 --> 00:29:23,840 You just have a charity project delivering necessary services, but we don't know if they 330 00:29:23,840 --> 00:29:32,140 made a statistically significant difference in reducing the maternal death ratio compared 331 00:29:32,140 --> 00:29:39,940 to a group of women and girls that did not receive the project services. 332 00:29:39,940 --> 00:29:42,280 So yeah. 333 00:29:42,280 --> 00:29:45,320 Any comment you want to make Benjamin before we? 334 00:29:45,320 --> 00:29:49,800 No, and I really agree with you on this point. 335 00:29:49,800 --> 00:30:00,440 Because when you are looking at maternal mortality ratio, the project is supposed to reduce maternal 336 00:30:00,440 --> 00:30:02,520 mortality ratio. 337 00:30:02,520 --> 00:30:08,360 So while we are having the project, we expect that by the end of the project we will start 338 00:30:08,360 --> 00:30:11,640 seeing a reduction in maternal mortality ratio. 339 00:30:11,640 --> 00:30:17,240 What about the other areas where the project is not being implemented? 340 00:30:17,240 --> 00:30:18,400 Is it the same? 341 00:30:18,400 --> 00:30:21,880 Maybe they will even do a better project. 342 00:30:21,880 --> 00:30:26,360 So then you ask people how they want to support that, right? 343 00:30:26,360 --> 00:30:31,800 Even though they are not receiving any support or they are not receiving any intervention, 344 00:30:31,800 --> 00:30:37,000 but we are still having a reduced maternal mortality ratio. 345 00:30:37,000 --> 00:30:41,920 And that's why it's always important to have the reason. 346 00:30:41,920 --> 00:30:51,120 Because it can help you to think about the strategies that you are using to implement 347 00:30:51,120 --> 00:30:52,120 your project. 348 00:30:52,120 --> 00:30:59,880 If there is something that you should have done, which is done some way, so I agree with 349 00:30:59,880 --> 00:31:00,880 you. 350 00:31:00,880 --> 00:31:04,000 That's why it's always important to have a comparison group. 351 00:31:04,000 --> 00:31:09,360 It helps you to learn the project. 352 00:31:09,360 --> 00:31:10,360 Thank you, Benjamin. 353 00:31:10,360 --> 00:31:17,800 I mean, to be fair to UNFPA, another option is they may say we don't have the funds to 354 00:31:17,800 --> 00:31:25,360 do a large scale comparison group measuring these people outside of the project. 355 00:31:25,360 --> 00:31:32,160 The other thing they could do is measure more frequently just the project group, before, 356 00:31:32,160 --> 00:31:35,960 before, before, during, during, during, during, after, after, after. 357 00:31:35,960 --> 00:31:41,840 So this is a standard thing that people try to do, as I mentioned in my trailer of this 358 00:31:41,840 --> 00:31:48,720 podcast, where you can sort of get around the problem of not using comparison groups. 359 00:31:48,720 --> 00:31:51,220 It's quasi-experimental, they call it. 360 00:31:51,220 --> 00:31:57,520 So there's a whole bunch of ways you can maybe measure more frequently rather than just, 361 00:31:57,520 --> 00:32:02,200 you know, like they mentioned, baseline, midline end of project. 362 00:32:02,200 --> 00:32:08,360 And you could maybe even do small samples of those women and girls more frequently throughout 363 00:32:08,360 --> 00:32:09,640 the project. 364 00:32:09,640 --> 00:32:15,360 But there's really never any discussion in the evaluation plan about this problem and 365 00:32:15,360 --> 00:32:16,520 how to get around it. 366 00:32:16,520 --> 00:32:23,020 It's always the same cookie cutter approach, which is pre-test, post-test only when they're 367 00:32:23,020 --> 00:32:27,720 training people, which we know is the same problem, it's not good enough. 368 00:32:27,720 --> 00:32:35,320 Or when they get to larger scale measurements like maternal mortality, they just do baseline, 369 00:32:35,320 --> 00:32:37,600 midline end of project. 370 00:32:37,600 --> 00:32:43,240 And they just assume that they can assume that's $37 million is why the maternal deaths 371 00:32:43,240 --> 00:32:44,380 have gone down. 372 00:32:44,380 --> 00:32:46,040 So they have options. 373 00:32:46,040 --> 00:32:52,280 And the other option I've discussed on other podcasts is just be a charity. 374 00:32:52,280 --> 00:32:58,700 Just say, look, these women and girls need these services, full stop. 375 00:32:58,700 --> 00:33:00,400 We're not going to evaluate them. 376 00:33:00,400 --> 00:33:04,860 This is an emergency humanitarian situation. 377 00:33:04,860 --> 00:33:07,540 And we're going to deliver these services. 378 00:33:07,540 --> 00:33:15,120 And we don't care about evaluation because we know we're saving lives by giving them 379 00:33:15,120 --> 00:33:18,120 access to these services. 380 00:33:18,120 --> 00:33:21,960 And that may explain why there's limited evaluation. 381 00:33:21,960 --> 00:33:28,360 A lot of people are just assuming they're a charity, but unfortunately they're making 382 00:33:28,360 --> 00:33:35,720 these claims that they are achieving all these outcomes, but then they're not prepared to 383 00:33:35,720 --> 00:33:37,000 evaluate. 384 00:33:37,000 --> 00:33:43,440 So in the wider scheme of things, we get it. 385 00:33:43,440 --> 00:33:48,400 And we understand that's why I have the podcast to sort of say, maybe you can make a choice 386 00:33:48,400 --> 00:33:57,240 here and just be a charity and not make these broad claims of achieving outcomes. 387 00:33:57,240 --> 00:33:59,160 That's your other option. 388 00:33:59,160 --> 00:34:01,280 So yeah. 389 00:34:01,280 --> 00:34:02,600 Okay. 390 00:34:02,600 --> 00:34:10,200 Before we go, are there any other, I think I was going to say we could add, we could 391 00:34:10,200 --> 00:34:16,320 talk about other indicators, but I think we've done pretty well here. 392 00:34:16,320 --> 00:34:21,980 Did you want to talk about some other indicators or should we just go to the chase, which is 393 00:34:21,980 --> 00:34:29,880 sort of to conclude from you at least, what you think of the PMF in its current form? 394 00:34:29,880 --> 00:34:32,760 Or do you want to talk about some other indicators? 395 00:34:32,760 --> 00:34:35,960 What would you like to do? 396 00:34:35,960 --> 00:34:37,720 Because there is another indicator. 397 00:34:37,720 --> 00:34:43,640 There's the number of policy recommendations to government and key stakeholders on health 398 00:34:43,640 --> 00:34:48,480 and gender-based violence from that first indicator, first outcome, which is improved 399 00:34:48,480 --> 00:34:54,560 national and state government capacities to develop and implement sexual reproductive 400 00:34:54,560 --> 00:35:00,640 health, gender-based violence, HIV policies, legislation, coordination mechanisms. 401 00:35:00,640 --> 00:35:02,600 There is another indicator there. 402 00:35:02,600 --> 00:35:03,840 Number of policy recommendations. 403 00:35:03,840 --> 00:35:04,840 Yes. 404 00:35:04,840 --> 00:35:09,480 I think I have them. 405 00:35:09,480 --> 00:35:21,680 So policy recommendation, you know, so then is it a recommendation project from evaluation 406 00:35:21,680 --> 00:35:28,600 report or my project beneficiary or under development darkness? 407 00:35:28,600 --> 00:35:39,160 I mean, the way it has been stated now is to open and for me when I pick, you know, 408 00:35:39,160 --> 00:35:46,680 all three, I will not know where do I get the information to measure the number of policy 409 00:35:46,680 --> 00:35:47,680 recommendations. 410 00:35:47,680 --> 00:35:50,880 Who is making this recommendation? 411 00:35:50,880 --> 00:35:57,200 So then it needs to be, yeah, because there are a number of policy recommendations that 412 00:35:57,200 --> 00:36:00,560 are made to that. 413 00:36:00,560 --> 00:36:04,360 So then the project will need to be specific. 414 00:36:04,360 --> 00:36:12,240 So you can see a number of recommendations made, the project, a result of our evaluation 415 00:36:12,240 --> 00:36:15,600 findings or monitoring findings. 416 00:36:15,600 --> 00:36:20,480 So of course, we have them planned to monitor and monitor the project. 417 00:36:20,480 --> 00:36:29,800 So probably as a result of findings from our evaluation or monitoring, we made this recommendation 418 00:36:29,800 --> 00:36:30,800 to the government. 419 00:36:30,800 --> 00:36:38,600 But then when I pick, I will not know, so this indicator is beneficiary, as a result 420 00:36:38,600 --> 00:36:42,240 the knowledge they have by making this recommendation to the government. 421 00:36:42,240 --> 00:36:49,440 All the evaluation we want, the monitoring of the assessment, as a result of making this 422 00:36:49,440 --> 00:36:53,600 recommendation to them. 423 00:36:53,600 --> 00:37:00,760 So then this is something that needs to be considered. 424 00:37:00,760 --> 00:37:06,640 Actually it's a good point, because if you look at the performance measurement framework, 425 00:37:06,640 --> 00:37:11,480 the outcome indicator is buried in the outputs. 426 00:37:11,480 --> 00:37:13,600 It's misplaced. 427 00:37:13,600 --> 00:37:20,480 Actually have a number of functional, national, and state coordination mechanisms. 428 00:37:20,480 --> 00:37:25,960 So they actually have a good indicator of increased capacity, but it's buried in the 429 00:37:25,960 --> 00:37:28,100 outputs. 430 00:37:28,100 --> 00:37:40,440 So it's good, but the problem is they just have project reports which report on the quality 431 00:37:40,440 --> 00:37:48,080 of these recommendations based on the training that the project delivers to these Ministry 432 00:37:48,080 --> 00:37:50,720 of Health staff. 433 00:37:50,720 --> 00:37:59,600 The problem is there's no attribution that the project training has contributed to this 434 00:37:59,600 --> 00:38:03,040 increased quality of the policy recommendations. 435 00:38:03,040 --> 00:38:10,200 It just says technical support, but it's only done annually, and that's not good enough 436 00:38:10,200 --> 00:38:17,800 to show that the training, or the technical support they call it, is responsible for the 437 00:38:17,800 --> 00:38:20,960 improved policy recommendations. 438 00:38:20,960 --> 00:38:27,920 So it's buried in the PMF, but it's lower down, and even it's there, even though it's 439 00:38:27,920 --> 00:38:36,560 a good indicator, as I mentioned in the summary report that the listeners can get a copy of, 440 00:38:36,560 --> 00:38:42,280 they don't adequately measure it to claim that the project training was responsible. 441 00:38:42,280 --> 00:38:49,280 It's just an annual report, and they could figure this stuff out all on their own at 442 00:38:49,280 --> 00:38:52,920 the same time or in between the trainings. 443 00:38:52,920 --> 00:38:56,440 So that's one of the issues that's common in a lot of these performance measurement 444 00:38:56,440 --> 00:39:05,440 frameworks is they do testing to show increased skill knowledge acquisition, but they don't 445 00:39:05,440 --> 00:39:10,620 do it frequently enough, or they don't use comparison groups. 446 00:39:10,620 --> 00:39:15,480 So there's no way of knowing if the project was responsible, like you were talking about 447 00:39:15,480 --> 00:39:16,480 contribution. 448 00:39:16,480 --> 00:39:21,400 And a lot of the times they'll say contribution instead of attribution, because attribution 449 00:39:21,400 --> 00:39:25,840 is a dirty word, but they're not even there in this case. 450 00:39:25,840 --> 00:39:27,280 So that's a good point. 451 00:39:27,280 --> 00:39:30,560 I think we're good. 452 00:39:30,560 --> 00:39:38,600 So I wanted to ask you, completely biased of course, the performance measurement framework 453 00:39:38,600 --> 00:39:42,520 and its current form, what do you think? 454 00:39:42,520 --> 00:39:51,560 Is it adequate for $37 million UNFPA South Sudan project to make the claim that they're 455 00:39:51,560 --> 00:39:53,040 achieving their outcomes? 456 00:39:53,040 --> 00:39:57,880 In its current state, we say no. 457 00:39:57,880 --> 00:40:04,120 So the performance management framework needs to be revised. 458 00:40:04,120 --> 00:40:11,600 Because when you look at the indicators in this case, the original indicators, they had 459 00:40:11,600 --> 00:40:16,320 the rain and the rain was that day. 460 00:40:16,320 --> 00:40:28,480 So even using the performance measurement framework says your contribution was held in 461 00:40:28,480 --> 00:40:30,600 South Sudan. 462 00:40:30,600 --> 00:40:35,560 I think it needs to be revised, I mean it needs to be revised. 463 00:40:35,560 --> 00:40:41,760 All the indicators, or most of the indicators, needs to be revised again. 464 00:40:41,760 --> 00:40:49,160 And those were good for probably the M&A office, the representative from the M&A office of 465 00:40:49,160 --> 00:40:51,160 the UNFPA to be here. 466 00:40:51,160 --> 00:40:55,400 I'm not going to say it myself, but you will understand it very well. 467 00:40:55,400 --> 00:40:58,440 So it's a minute to us. 468 00:40:58,440 --> 00:41:05,400 So at this time, for even the PIAF, we denounce to raise other contributions. 469 00:41:05,400 --> 00:41:11,200 Yeah, and even though they're required by Global Affairs Canada to produce an evaluation 470 00:41:11,200 --> 00:41:17,840 plan, all of the evaluation plans I've received, including the one for this project, they're 471 00:41:17,840 --> 00:41:19,560 all generic in nature. 472 00:41:19,560 --> 00:41:27,160 They don't go through each outcome indicator like we have just done and explain how they're 473 00:41:27,160 --> 00:41:33,400 going to show the project is achieving the outcome. 474 00:41:33,400 --> 00:41:34,400 And that's an issue. 475 00:41:34,400 --> 00:41:35,640 Yeah, that's good to know. 476 00:41:35,640 --> 00:41:42,960 So what we're going to do now, and I should point out that the Global Affairs Canada, 477 00:41:42,960 --> 00:41:50,440 their own guidelines on designing performance measurement frameworks makes this clear, that 478 00:41:50,440 --> 00:41:57,440 you have to have outcome indicators that reliably and validly measure the outcomes. 479 00:41:57,440 --> 00:42:00,000 So they're not even following their own guidelines. 480 00:42:00,000 --> 00:42:04,120 This is not some thing I've cooked up myself. 481 00:42:04,120 --> 00:42:06,920 I'm just bringing it out into the public domain. 482 00:42:06,920 --> 00:42:13,760 So what we're going to do now is I will send part one and part two to the Government of 483 00:42:13,760 --> 00:42:20,040 Canada, Minister of International Development, as well as the performance measurement framework 484 00:42:20,040 --> 00:42:26,560 our summary sheet that explains all the indicators and why they have this problem. 485 00:42:26,560 --> 00:42:33,160 And these two episodes, part one and two, and also sending it to the shadow critics 486 00:42:33,160 --> 00:42:39,080 of international development for the Bloc Québécois, the New Democratic Party, and 487 00:42:39,080 --> 00:42:40,960 the Conservative Party. 488 00:42:40,960 --> 00:42:43,920 So everybody knows this is a Canadian problem. 489 00:42:43,920 --> 00:42:49,840 It's not specific to a particular political affiliation. 490 00:42:49,840 --> 00:42:55,180 It should be a concern for all Canadians that there are $4 billion that they spend every 491 00:42:55,180 --> 00:42:58,000 year in developing countries. 492 00:42:58,000 --> 00:43:00,680 It should be properly evaluated. 493 00:43:00,680 --> 00:43:03,680 And we'll leave it at that. 494 00:43:03,680 --> 00:43:09,280 So Benjamin, thank you very much for your time. 495 00:43:09,280 --> 00:43:11,840 Thank you for having me. 496 00:43:11,840 --> 00:43:14,600 And thank you for your expertise and insight. 497 00:43:14,600 --> 00:43:15,600 It was very helpful. 498 00:43:15,600 --> 00:43:18,600 All right, David, thank you too. 499 00:43:18,600 --> 00:43:19,960 Okina. 500 00:43:19,960 --> 00:43:21,560 Ok. 501 00:43:21,560 --> 00:43:23,560 All right. 502 00:43:23,560 --> 00:43:24,560 Bye. 503 00:43:24,560 --> 00:43:26,560 And have some Bangkok for me, please. 504 00:43:26,560 --> 00:43:27,560 In Italy. 505 00:43:27,560 --> 00:43:30,560 Whenever you are coming to Ghana, you know. 506 00:43:30,560 --> 00:43:32,560 That's the same. 507 00:43:32,560 --> 00:43:35,560 That's the same. 508 00:43:35,560 --> 00:43:36,560 All right. 509 00:43:36,560 --> 00:43:37,560 Midasi. 510 00:43:37,560 --> 00:43:38,560 Ok. 511 00:43:38,560 --> 00:43:39,560 Mr. Midasi. 512 00:43:39,560 --> 00:43:40,560 Midasi. 513 00:43:40,560 --> 00:43:41,560 Okina. 514 00:43:41,560 --> 00:43:42,560 Ok. 515 00:43:42,560 --> 00:43:43,560 All right, David. 516 00:43:43,560 --> 00:43:44,560 Bye bye. 517 00:43:44,560 --> 00:43:49,560 Bye bye.