1 00:00:00,000 --> 00:00:04,520 Welcome to the Improving Development Evaluation Podcast. 2 00:00:04,520 --> 00:00:09,480 I'm your host David Wond and welcome to season two, episode one. 3 00:00:09,480 --> 00:00:15,840 And in this episode, we're featuring the International Development Organization United Nations Population 4 00:00:15,840 --> 00:00:18,680 Fund, UNFPA. 5 00:00:18,680 --> 00:00:25,880 You can learn more about the United Nations Population Fund at their website, unfpa.org. 6 00:00:25,880 --> 00:00:32,200 And we're focusing on a project that the UNFPA is delivering in Somalia and its title is 7 00:00:32,200 --> 00:00:38,760 Strengthening Sexual Reproductive Health and Rights Through Midwives. 8 00:00:38,760 --> 00:00:44,360 And the budget for this project is 10 million Canadian dollars, courtesy of the Canadian 9 00:00:44,360 --> 00:00:48,240 taxpayer and funded by the government of Canada. 10 00:00:48,240 --> 00:00:53,800 And we're going to be reviewing their performance measurement framework. 11 00:00:53,800 --> 00:01:00,820 And we will start with a brief project description and then we'll move into how the outcome indicators 12 00:01:00,820 --> 00:01:07,480 either adequately measure the outcomes or do not adequately measure the outcomes. 13 00:01:07,480 --> 00:01:13,480 This project has eight target groups and we'll list them off and then we'll get back and 14 00:01:13,480 --> 00:01:18,960 give you details about the services that each of the target groups receive. 15 00:01:18,960 --> 00:01:24,500 First target group number one are students studying midwifery. 16 00:01:24,500 --> 00:01:29,360 Number two are midwives after they've graduated and are practicing. 17 00:01:29,360 --> 00:01:35,100 Number three are students studying midwifery and practicing midwives. 18 00:01:35,100 --> 00:01:39,600 Target group number four are midwife associations in Somalia. 19 00:01:39,600 --> 00:01:48,080 Number five are midwives who teach at midwifery schools as well as midwifery tutors. 20 00:01:48,080 --> 00:01:52,980 Target group number six, Ministry of Health officials in Somalia. 21 00:01:52,980 --> 00:01:57,400 Target group number seven, traditional birth attendance. 22 00:01:57,400 --> 00:02:03,920 And finally, target group number eight, Chief of Health, midwifery specialist and a family 23 00:02:03,920 --> 00:02:06,960 planning program manager. 24 00:02:06,960 --> 00:02:13,880 So let's get back to the services that these target groups actually receive due to this 25 00:02:13,880 --> 00:02:16,520 project costing about 10 million dollars. 26 00:02:16,520 --> 00:02:22,160 So the first target group, the target group are students who are receiving midwifery training 27 00:02:22,160 --> 00:02:25,280 and that training includes the following. 28 00:02:25,280 --> 00:02:31,080 Developing a plan to hire and pay midwifery graduates. 29 00:02:31,080 --> 00:02:35,560 Equipment is provided to the midwifery schools. 30 00:02:35,560 --> 00:02:39,360 Training curriculum is developed and delivered in the midwifery schools including bringing 31 00:02:39,360 --> 00:02:43,400 in Canadian midwives to the classroom. 32 00:02:43,400 --> 00:02:50,680 Also supervision visits and technical assistance is delivered to the midwifery schools. 33 00:02:50,680 --> 00:02:58,140 The midwifery schools are assessed for accreditation for midwife students to be placed for practicums. 34 00:02:58,140 --> 00:03:01,120 The second target group are midwives themselves. 35 00:03:01,120 --> 00:03:06,840 They receive continuous professional development training. 36 00:03:06,840 --> 00:03:14,680 Midwives are also supported to lobby to get national midwifery act into the national constitution. 37 00:03:14,680 --> 00:03:20,240 The third target group are students studying midwifery as well as practicing midwives. 38 00:03:20,240 --> 00:03:26,540 They receive online midwifery training materials that are established. 39 00:03:26,540 --> 00:03:29,720 The fourth target group, midwife associations. 40 00:03:29,720 --> 00:03:35,580 They're trained to disseminate and develop messages through radio and social media on 41 00:03:35,580 --> 00:03:39,520 safe delivery and nutrition. 42 00:03:39,520 --> 00:03:45,720 They also receive or attend events or exchanges where I'm assuming it's Canadians go to 43 00:03:45,720 --> 00:03:51,920 Somalia to train members of midwife associations to quote, strengthen their leadership and 44 00:03:51,920 --> 00:04:00,960 management capacity including their ability to advocate for midwifery as a profession. 45 00:04:00,960 --> 00:04:07,920 These midwife associations also support to advocate for zero tolerance for female genital 46 00:04:07,920 --> 00:04:10,080 mutilation at hospitals. 47 00:04:10,080 --> 00:04:18,680 They also train these midwife associations, girls at schools, universities and youth representatives 48 00:04:18,680 --> 00:04:25,260 on how to identify cases for referral to midwifery services. 49 00:04:25,260 --> 00:04:32,880 This association also develops a mobile app to reach youth to access midwifery services. 50 00:04:32,880 --> 00:04:38,180 The association is also supported to train traditional birth attendants and health workers 51 00:04:38,180 --> 00:04:46,600 and midwives on breastfeeding and nutrition and food security. 52 00:04:46,600 --> 00:04:52,520 Target group number five, midwives who teach in midwifery schools and midwifery tutors. 53 00:04:52,520 --> 00:04:59,680 We're not clear who does it, but policy guidelines and protocols are developed and taught to 54 00:04:59,680 --> 00:05:06,680 the students studying midwifery by the midwives who teach them. 55 00:05:06,680 --> 00:05:11,400 Target group number six, the Ministry of Health officials. 56 00:05:11,400 --> 00:05:17,940 Their responsibility as part of this project is to review health data to deliver 48 outreach 57 00:05:17,940 --> 00:05:23,920 events where women receive midwifery services. 58 00:05:23,920 --> 00:05:28,800 Now it's not clear what the dollars are used for, but I'm assuming they're used to pay 59 00:05:28,800 --> 00:05:34,280 for those outreach events, those 48 outreach events. 60 00:05:34,280 --> 00:05:37,480 Target group seven, traditional birth attendants. 61 00:05:37,480 --> 00:05:44,480 The midwifery association does outreach to them to promote and increase referrals to 62 00:05:44,480 --> 00:05:45,760 midwives. 63 00:05:45,760 --> 00:05:52,400 Also I should add, midwifery associations deliver 48 awareness events on midwifery and 64 00:05:52,400 --> 00:05:57,240 train midwives on nutrition and nutrition counseling. 65 00:05:57,240 --> 00:06:02,600 Target group number eight, which is the chief of health, the midwifery specialist and the 66 00:06:02,600 --> 00:06:06,200 family planning program manager. 67 00:06:06,200 --> 00:06:13,960 The project funds are used to pay 25% of the salary of the chief of health, 50% of the 68 00:06:13,960 --> 00:06:20,800 salary of the midwifery specialist, and 20% of the family planning program manager in 69 00:06:20,800 --> 00:06:23,800 this project. 70 00:06:23,800 --> 00:06:30,080 And their responsibility, these three individuals, is to deliver a mass media campaign, which 71 00:06:30,080 --> 00:06:37,040 I suspect relates to the training of midwives on message dissemination, which I was talking 72 00:06:37,040 --> 00:06:38,620 about earlier. 73 00:06:38,620 --> 00:06:46,240 So that's a brief overview of the target groups and the actual services that they receive. 74 00:06:46,240 --> 00:06:50,000 And as you can tell, a lot of it has to do with training. 75 00:06:50,000 --> 00:06:56,240 So if we look at the performance measurement framework, there are eight outcomes for this 76 00:06:56,240 --> 00:07:00,440 project related to those services. 77 00:07:00,440 --> 00:07:07,680 And there's 13 outcome indicators that are in the performance measurement framework that 78 00:07:07,680 --> 00:07:12,700 attempt to measure whether or not those outcomes have been achieved. 79 00:07:12,700 --> 00:07:17,440 So I'll just briefly go through the eight outcomes and then I'll go back in detail, 80 00:07:17,440 --> 00:07:23,160 provide the outcome indicators and whether or not they adequately measure the outcomes. 81 00:07:23,160 --> 00:07:31,560 First outcome is project is effectively supported in terms of operations, security, communications, 82 00:07:31,560 --> 00:07:34,280 advocacy, monitoring, and evaluation. 83 00:07:34,280 --> 00:07:41,720 Outcome number two is improve the role of midwives in increasing knowledge awareness 84 00:07:41,720 --> 00:07:47,720 on maternal nutrition and contributing to sexual reproductive health outcomes for target 85 00:07:47,720 --> 00:07:49,160 communities. 86 00:07:49,160 --> 00:07:56,000 Outcome number three, increased marginalized women and girls access to quality and rights-based 87 00:07:56,000 --> 00:08:04,160 midwifery services, including sexual reproductive health rights-based services. 88 00:08:04,160 --> 00:08:11,520 Number four, improved regulatory environment and association leadership for rights-based 89 00:08:11,520 --> 00:08:15,540 quality midwifery practice. 90 00:08:15,540 --> 00:08:22,520 Number five outcome, increased quality of midwifery training and practice both in pre 91 00:08:22,520 --> 00:08:28,800 and in service to provide rights-based and evidence-based knowledge and service delivery, 92 00:08:28,800 --> 00:08:31,760 including sexual reproductive health rights. 93 00:08:31,760 --> 00:08:37,080 Outcome number six, increased use of skilled midwives and reproductive health services 94 00:08:37,080 --> 00:08:43,780 by women and adolescent girls age 14 to 49. 95 00:08:43,780 --> 00:08:52,400 Outcome seven, improved availability and accessibility of professionally trained and certified midwives 96 00:08:52,400 --> 00:09:02,080 able to provide quality rights-based sexual reproductive health services to women aged 97 00:09:02,080 --> 00:09:03,080 14 to 49. 98 00:09:03,080 --> 00:09:09,000 And finally, outcome number eight, reduced maternal mortality and increased sexual and 99 00:09:09,000 --> 00:09:16,320 reproductive health and rights for women aged 14 to 49 in Somalia. 100 00:09:16,320 --> 00:09:21,360 So if we look at the first outcome, project is effectively supported in terms of operations, 101 00:09:21,360 --> 00:09:26,440 security, communications, advocacy, monitoring and evaluation. 102 00:09:26,440 --> 00:09:31,160 There is no outcome indicators at all for this and it may have something to do with 103 00:09:31,160 --> 00:09:37,080 it's been thrown in there just for operations to make sure that the project is going smoothly. 104 00:09:37,080 --> 00:09:42,640 I will point out that one thing they should be doing, which a lot of these projects are 105 00:09:42,640 --> 00:09:49,840 not doing is taking a certain percentage of the budget and allocating it towards evaluation 106 00:09:49,840 --> 00:09:55,440 because a lot of these organizations claim there is a lack of funding for the evaluation. 107 00:09:55,440 --> 00:10:02,480 So you could crudely put, take 10% of the 10 million or 1 million for proper evaluation. 108 00:10:02,480 --> 00:10:06,960 And in fact, you could use that for measuring the outcome indicators recommended that I'll 109 00:10:06,960 --> 00:10:14,880 be doing in this episode because if you've watched season one, one particular NGO that 110 00:10:14,880 --> 00:10:20,440 did show up to respond to our critique was saying they don't have enough resources and 111 00:10:20,440 --> 00:10:23,040 that is possible. 112 00:10:23,040 --> 00:10:27,000 But you would think if they do proper outcome indicator design that they would put their 113 00:10:27,000 --> 00:10:31,880 money where their mouth is and the government of Canada would make sure that there was sufficient 114 00:10:31,880 --> 00:10:34,540 funding within the project. 115 00:10:34,540 --> 00:10:38,500 So if it's 10 million bucks, they should take a certain percent of it immediately and put 116 00:10:38,500 --> 00:10:43,600 it aside for evaluation and then the remainder can be used to deliver the services rather 117 00:10:43,600 --> 00:10:47,560 than using it all just on delivering the services. 118 00:10:47,560 --> 00:10:51,760 So that's the recommendation there is at least have a fixed amount. 119 00:10:51,760 --> 00:10:57,000 It depends of course on the project, but 10% would be a good rule that you could put aside 120 00:10:57,000 --> 00:11:01,160 for the evaluation, the measurement of the outcome indicators. 121 00:11:01,160 --> 00:11:07,280 So outcome number two, improve the role of midwives in increasing knowledge, awareness 122 00:11:07,280 --> 00:11:13,320 on maternal nutrition and contributing sexual reproductive health outcomes for target communities. 123 00:11:13,320 --> 00:11:20,920 The outcome indicator there is community awareness engagement and training package developed 124 00:11:20,920 --> 00:11:22,120 and implemented. 125 00:11:22,120 --> 00:11:28,800 And as you can tell, that's not an adequate measure of increasing knowledge and awareness. 126 00:11:28,800 --> 00:11:32,960 So that is not related to the outcome. 127 00:11:32,960 --> 00:11:37,440 So what they should be doing instead is they need to measure levels of awareness among 128 00:11:37,440 --> 00:11:43,160 the target groups, which are youth and girls that the midwife associations are sending 129 00:11:43,160 --> 00:11:48,360 messages to and training them to make referrals. 130 00:11:48,360 --> 00:11:54,400 Another option is to track the number of referrals against locations where these project midwives 131 00:11:54,400 --> 00:11:57,620 are not sending out messages. 132 00:11:57,620 --> 00:12:04,800 So in terms of the violation of the OECD development assistance committee criteria, it's violating 133 00:12:04,800 --> 00:12:11,040 the effectiveness criteria, which the DAC question is, is the intervention achieving 134 00:12:11,040 --> 00:12:12,040 its objectives? 135 00:12:12,040 --> 00:12:18,040 No, it is not because the outcome indicator is not measuring levels of awareness, which 136 00:12:18,040 --> 00:12:22,760 the intervention is expected to increase. 137 00:12:22,760 --> 00:12:28,880 Also you could argue it's not addressing the impact question, which is did the interventions 138 00:12:28,880 --> 00:12:30,400 make a difference? 139 00:12:30,400 --> 00:12:35,840 Now it's a bit debatable, but you could argue here that you could look at impact in the 140 00:12:35,840 --> 00:12:41,400 areas where the project is trying to increase referrals by sending out messages via social 141 00:12:41,400 --> 00:12:42,600 media. 142 00:12:42,600 --> 00:12:48,400 And you could look at areas where those messages are not being sent out and compare whether 143 00:12:48,400 --> 00:12:53,200 or not the referrals are lower in the areas where the project is not operating. 144 00:12:53,200 --> 00:12:56,080 That would be an example where you could show impact. 145 00:12:56,080 --> 00:12:58,460 But again, they're not even doing that here. 146 00:12:58,460 --> 00:13:04,320 So that's another violation of impact using the DAC criteria. 147 00:13:04,320 --> 00:13:11,300 Third outcome is increased marginalized women and girls access to quality and rights-based 148 00:13:11,300 --> 00:13:17,280 midwifery services, including sexual reproductive health services. 149 00:13:17,280 --> 00:13:22,760 One of the indicators there is the number of midwives graduating in accordance to WHO 150 00:13:22,760 --> 00:13:26,160 standards, World Health Organization standards. 151 00:13:26,160 --> 00:13:32,200 So even if they graduate up to standard, we don't know if they've got access, increased 152 00:13:32,200 --> 00:13:38,040 access these marginalized women and girls, unless we go to the facilities and make sure 153 00:13:38,040 --> 00:13:41,720 that these midwives show up and start working, right? 154 00:13:41,720 --> 00:13:45,060 This outcome indicator doesn't measure the outcome directly. 155 00:13:45,060 --> 00:13:47,580 So it's not related to the outcome. 156 00:13:47,580 --> 00:13:52,100 So they graduate, but they do not show up at the health facilities. 157 00:13:52,100 --> 00:13:57,160 So what you would do instead is measure the number of midwives who are at the health 158 00:13:57,160 --> 00:14:02,760 facilities against a standard considered acceptable for adequate access. 159 00:14:02,760 --> 00:14:06,400 And the WHO does have standards, global standards. 160 00:14:06,400 --> 00:14:12,080 In fact, they're probably even country level standards where you can say there's enough 161 00:14:12,080 --> 00:14:20,920 health professionals per capita or per catchment area of population that's acceptable. 162 00:14:20,920 --> 00:14:27,280 So again, this violates the effectiveness criteria of the development assistance committee 163 00:14:27,280 --> 00:14:29,320 criteria on evaluation. 164 00:14:29,320 --> 00:14:31,760 Is the intervention achieving its objectives? 165 00:14:31,760 --> 00:14:38,840 No, it is not because the outcome indicator is not measuring levels of access, which the 166 00:14:38,840 --> 00:14:43,380 intervention is expected to increase. 167 00:14:43,380 --> 00:14:49,500 Another indicator for the same outcome is the number of midwifery schools where curriculum 168 00:14:49,500 --> 00:14:55,740 has been updated to include rights-based approach to sexual reproductive health. 169 00:14:55,740 --> 00:15:03,720 So again, it's not relevant in the sense that you could have the curriculum updated, but 170 00:15:03,720 --> 00:15:05,760 you really want to go a step further. 171 00:15:05,760 --> 00:15:08,440 So the measure is not related to the outcome. 172 00:15:08,440 --> 00:15:11,640 And what they want to do is if even if you improve the curriculum, it has nothing to 173 00:15:11,640 --> 00:15:16,760 do with monitoring increases in access, which we want to see. 174 00:15:16,760 --> 00:15:21,560 So again, you want to go to the health facilities instead where midwives are working to see 175 00:15:21,560 --> 00:15:27,760 if the number of patients served or the wait lists are considered adequate access against 176 00:15:27,760 --> 00:15:29,360 the agreed upon standard. 177 00:15:29,360 --> 00:15:37,120 So again, it violates the effectiveness criteria of DAT, development assistance committee for 178 00:15:37,120 --> 00:15:39,500 evaluation. 179 00:15:39,500 --> 00:15:46,760 So outcome number four, improved regulatory environment and association leadership for 180 00:15:46,760 --> 00:15:49,980 rights-based quality midwifery practice. 181 00:15:49,980 --> 00:15:57,120 The first indicator there is number of midwifery associations supported for rights-based midwifery 182 00:15:57,120 --> 00:15:59,680 and sexual reproductive health services. 183 00:15:59,680 --> 00:16:07,640 Again, the measure is not related to the outcome of improving leadership skills. 184 00:16:07,640 --> 00:16:12,320 So they say support it, but even if they're supported through training or whatever, what 185 00:16:12,320 --> 00:16:17,200 you really want to do is you want to measure the leadership skills and the skill to advocate 186 00:16:17,200 --> 00:16:21,380 for rights-based quality would be better. 187 00:16:21,380 --> 00:16:27,680 So again, it violates the DAT criteria on effectiveness because the outcome indicator 188 00:16:27,680 --> 00:16:33,560 is not measuring levels of skill to lead and advocate, which the intervention is expected 189 00:16:33,560 --> 00:16:36,440 to increase. 190 00:16:36,440 --> 00:16:42,360 So the other indicator for outcome number four, number of midwifery association members 191 00:16:42,360 --> 00:16:48,320 supported to lead right-based quality midwifery practice efforts. 192 00:16:48,320 --> 00:16:53,400 Again, it's the number of associations, but it's got nothing to do with measuring the 193 00:16:53,400 --> 00:16:56,820 outcome of improved leadership. 194 00:16:56,820 --> 00:17:01,960 So again, they need to measure the leadership skills and the ability to advocate. 195 00:17:01,960 --> 00:17:05,760 So again, it violates the DAT criteria of effectiveness. 196 00:17:05,760 --> 00:17:12,580 Then you have outcome number five, increased quality of midwifery training and practice 197 00:17:12,580 --> 00:17:20,200 both in pre and in service to provide rights-based and evidence-based knowledge and service delivery, 198 00:17:20,200 --> 00:17:23,480 including sexual reproductive health services. 199 00:17:23,480 --> 00:17:30,840 The indicator there is the number of midwifery tutors train in new content and evidence-based 200 00:17:30,840 --> 00:17:33,920 training methods or methodologies. 201 00:17:33,920 --> 00:17:37,360 Here they're just taking attendance of how many got trained. 202 00:17:37,360 --> 00:17:41,660 They're not even looking at the increased quality of the training. 203 00:17:41,660 --> 00:17:49,180 And one way to do that is to look at the test scores of these tutors on their tutoring methods 204 00:17:49,180 --> 00:17:54,920 to see if they've improved in how to use certain methods of how to tutor. 205 00:17:54,920 --> 00:18:02,360 So again, it violates the effectiveness DAT criteria because the outcome indicator is 206 00:18:02,360 --> 00:18:09,320 not measuring levels of skill to tutor, which the intervention is expected to increase. 207 00:18:09,320 --> 00:18:14,760 Another indicator they've used for this outcome number five is the number of midwives trained. 208 00:18:14,760 --> 00:18:21,480 So it's just a different group, not the tutors, in new midwifery content, including rights-based 209 00:18:21,480 --> 00:18:24,020 sexual reproductive health. 210 00:18:24,020 --> 00:18:29,040 So again, it's not just the number of midwives showing up to get the training. 211 00:18:29,040 --> 00:18:30,320 That's just taking attendance. 212 00:18:30,320 --> 00:18:31,440 That's not good enough. 213 00:18:31,440 --> 00:18:37,320 You have to show how you're going to increase the quality of these midwives who have been 214 00:18:37,320 --> 00:18:38,320 trained. 215 00:18:38,320 --> 00:18:45,080 So again, what you need to do instead is look at the test scores of midwives on their knowledge 216 00:18:45,080 --> 00:18:49,280 of rights-based sexual reproductive health content. 217 00:18:49,280 --> 00:18:55,200 So it doesn't measure effectiveness because they're not measuring levels of knowledge 218 00:18:55,200 --> 00:19:01,400 on rights-based sexual reproductive health, which the intervention, the training, is expected 219 00:19:01,400 --> 00:19:04,400 to increase or achieve. 220 00:19:04,400 --> 00:19:10,080 Outcome number six, increased use of skilled midwives in reproductive health services by 221 00:19:10,080 --> 00:19:14,920 women and adolescent girls aged 14 to 49. 222 00:19:14,920 --> 00:19:22,580 One of the indicators there is number of people provided with modern contraception by method. 223 00:19:22,580 --> 00:19:31,240 This is a great measure of expected outcome because the skilled midwives in reproductive 224 00:19:31,240 --> 00:19:35,080 health services delivered would include contraceptives. 225 00:19:35,080 --> 00:19:40,000 Now, we don't know if they're going to actually use them, but at least if they go to the health 226 00:19:40,000 --> 00:19:44,760 centers and they can show this, it's measured quarterly. 227 00:19:44,760 --> 00:19:50,760 We can see that if it's going up that the, we're assuming when they graduate from these 228 00:19:50,760 --> 00:19:56,680 midwifery schools, they show up at these facilities and they start handing out these contraceptives. 229 00:19:56,680 --> 00:20:03,920 So hopefully it will show that there has been increased use and this is a good measure of 230 00:20:03,920 --> 00:20:04,920 that. 231 00:20:04,920 --> 00:20:10,600 So this is a good example, but the challenge here is we haven't got the data and it's not 232 00:20:10,600 --> 00:20:15,020 on the Project Browser Global Affairs Canada website. 233 00:20:15,020 --> 00:20:20,280 So even though it's a good measure, the next step is we need to see the data to show that 234 00:20:20,280 --> 00:20:24,340 it's actually going up, the number of people receiving them. 235 00:20:24,340 --> 00:20:30,520 Another indicator that's very good for this outcome number six is the percentage increase 236 00:20:30,520 --> 00:20:37,580 in births per facility from the baseline determined at the project start. 237 00:20:37,580 --> 00:20:43,560 This again is an excellent indicator of that achievement of that outcome and it's measured 238 00:20:43,560 --> 00:20:44,980 annually. 239 00:20:44,980 --> 00:20:51,160 But again, we don't know that percentage increase and we need to see that data on the Project 240 00:20:51,160 --> 00:20:53,160 Browser. 241 00:20:53,160 --> 00:20:56,640 And that would be helpful, but it's a great indicator. 242 00:20:56,640 --> 00:21:03,000 Outcome number seven is improved availability and accessibility of professionally trained 243 00:21:03,000 --> 00:21:09,680 and certified midwives able to provide quality rights-based sexual reproductive health services 244 00:21:09,680 --> 00:21:13,800 to women aged 14 to 49. 245 00:21:13,800 --> 00:21:18,960 One of the indicators for that outcome is number of midwives graduating in accordance 246 00:21:18,960 --> 00:21:21,920 with WHO standards. 247 00:21:21,920 --> 00:21:28,920 Accessibility of course is not enough to show improved availability and access to services, 248 00:21:28,920 --> 00:21:29,920 right? 249 00:21:29,920 --> 00:21:32,700 We have to make sure they show up at the health facilities. 250 00:21:32,700 --> 00:21:36,800 So we have to measure the number of midwives who are at the health facilities against a 251 00:21:36,800 --> 00:21:40,680 standard considered acceptable for adequate access. 252 00:21:40,680 --> 00:21:45,940 Another indicator for this improved availability and accessibility to services outcome number 253 00:21:45,940 --> 00:21:53,980 seven is number of facilities with all the signal functions to provide skilled deliveries. 254 00:21:53,980 --> 00:21:58,980 I'm not sure what that means, but people in the field of this area of expertise probably 255 00:21:58,980 --> 00:22:00,600 know what it means. 256 00:22:00,600 --> 00:22:01,600 It's great. 257 00:22:01,600 --> 00:22:03,300 It's a good indicator. 258 00:22:03,300 --> 00:22:07,960 We also want to know the percentage of the facilities where all the signal functions 259 00:22:07,960 --> 00:22:11,080 are there to provide skilled deliveries. 260 00:22:11,080 --> 00:22:15,620 But again, we would like to know the number and the percentage, the actual data. 261 00:22:15,620 --> 00:22:17,120 So this is a good indicator. 262 00:22:17,120 --> 00:22:21,320 It would just be ideal if it was available to the public. 263 00:22:21,320 --> 00:22:27,180 The other indicator on outcome number seven is percent of respondents over the baseline 264 00:22:27,180 --> 00:22:29,520 reporting quality service. 265 00:22:29,520 --> 00:22:31,080 Now this is interesting. 266 00:22:31,080 --> 00:22:36,680 It's a great indicator, but what they're asking is quality. 267 00:22:36,680 --> 00:22:38,520 What do you think of the quality of the service? 268 00:22:38,520 --> 00:22:41,420 They're not asking about access or quantity. 269 00:22:41,420 --> 00:22:43,200 They want to know about the quality. 270 00:22:43,200 --> 00:22:44,280 So it's a great indicator. 271 00:22:44,280 --> 00:22:45,880 They just have to reword it. 272 00:22:45,880 --> 00:22:51,400 So instead of improved availability and access, they should change it to improve quality of 273 00:22:51,400 --> 00:22:53,400 services delivered. 274 00:22:53,400 --> 00:22:54,680 And that's an excellent indicator. 275 00:22:54,680 --> 00:22:59,860 So these are some examples of where some of the indicators are actually properly measuring 276 00:22:59,860 --> 00:23:01,840 the outcomes, which is great. 277 00:23:01,840 --> 00:23:08,600 And finally, outcome number eight, reduced maternal mortality and increased sexual and 278 00:23:08,600 --> 00:23:14,960 reproductive health and rights for women aged 14 to 49 in Somalia. 279 00:23:14,960 --> 00:23:19,880 And the indicator they have there is number of women and girls provided with access to 280 00:23:19,880 --> 00:23:24,520 sexual and reproductive health services, including modern methods of contraception. 281 00:23:24,520 --> 00:23:33,360 Well, as you've noticed, some of the previous outcome indicators already cover access to 282 00:23:33,360 --> 00:23:35,540 sexual and reproductive health services. 283 00:23:35,540 --> 00:23:38,400 So we don't need to worry about that. 284 00:23:38,400 --> 00:23:41,280 They need to measure the maternal mortality rate. 285 00:23:41,280 --> 00:23:43,960 And that's not in there, which is interesting. 286 00:23:43,960 --> 00:23:48,640 And it should be because that's critical because in the end, they're claiming with that outcome, 287 00:23:48,640 --> 00:23:50,200 they're claiming they're going to achieve it. 288 00:23:50,200 --> 00:23:55,240 They're going to reduce maternal mortality, but they haven't measured the maternal mortality 289 00:23:55,240 --> 00:23:56,980 rate. 290 00:23:56,980 --> 00:24:00,500 All they're measuring is access. 291 00:24:00,500 --> 00:24:09,260 So in that case, it's failing in terms of the DAC criteria, you could argue impact or 292 00:24:09,260 --> 00:24:11,120 you could say effectiveness. 293 00:24:11,120 --> 00:24:16,520 Has the maternal mortality rate, thanks to the project, actually gone down? 294 00:24:16,520 --> 00:24:21,920 And to show that you would need to show where the project is not operating if the maternal 295 00:24:21,920 --> 00:24:26,080 mortality rate is higher. 296 00:24:26,080 --> 00:24:31,600 Now to be fair, they have targeted all of the midwifery schools in Somalia. 297 00:24:31,600 --> 00:24:34,080 So you could argue it's blanketed. 298 00:24:34,080 --> 00:24:36,160 They've blanketed the entire country. 299 00:24:36,160 --> 00:24:41,280 There's no comparison group or control group where they haven't done the training and all 300 00:24:41,280 --> 00:24:44,660 those interventions that I described in the project. 301 00:24:44,660 --> 00:24:49,000 For some of them, you could argue like where they're trying to increase levels of knowledge 302 00:24:49,000 --> 00:24:53,240 and awareness on nutrition or breastfeeding, et cetera. 303 00:24:53,240 --> 00:24:56,000 And they're sending targeted messages in the community. 304 00:24:56,000 --> 00:24:59,400 There could be communities where they're not sending those messages. 305 00:24:59,400 --> 00:25:04,040 Then you could compare the two groups and see if levels of awareness is lower where 306 00:25:04,040 --> 00:25:07,080 the project is not operating. 307 00:25:07,080 --> 00:25:11,480 But definitely they need to measure maternal mortality rate. 308 00:25:11,480 --> 00:25:13,800 That is quite important. 309 00:25:13,800 --> 00:25:19,520 So here you could argue either effectiveness is being violated or when it comes to raising 310 00:25:19,520 --> 00:25:25,400 levels of awareness, they're not looking at the differences or impact, which is how the 311 00:25:25,400 --> 00:25:28,360 DAC criteria is described. 312 00:25:28,360 --> 00:25:32,500 What difference does the intervention make? 313 00:25:32,500 --> 00:25:36,320 So you could argue, well, maternal mortality rate has to go down. 314 00:25:36,320 --> 00:25:38,000 That's a difference. 315 00:25:38,000 --> 00:25:41,560 And also the knowledge and levels have to go up. 316 00:25:41,560 --> 00:25:44,160 So there's a summary of the project. 317 00:25:44,160 --> 00:25:50,200 And what I'm going to do now is send the performance measurement framework along with this analysis 318 00:25:50,200 --> 00:25:54,800 of the outcome indicators, send it to the minister for international development, as 319 00:25:54,800 --> 00:25:57,360 well as the shadow critics. 320 00:25:57,360 --> 00:26:05,200 And also if any of you would like any of the performance measurement frameworks and the 321 00:26:05,200 --> 00:26:10,800 Excel summary of the outcome indicators that I've just described in this episode, I'm happy 322 00:26:10,800 --> 00:26:12,520 to email those to you. 323 00:26:12,520 --> 00:26:20,120 You can request them by email and send an email to evaluatecanadaaid.gmail.com. 324 00:26:20,120 --> 00:26:23,340 I'll put it in the episode notes. 325 00:26:23,340 --> 00:26:28,800 Thank you for listening and stay tuned for episode two, which hopefully will come a little 326 00:26:28,800 --> 00:26:29,800 quicker. 327 00:26:29,800 --> 00:26:53,800 Thanks for now.