WEBVTT 00:00:00.000 --> 00:00:02.540 Welcome to the Improving Development Evaluation 00:00:02.540 --> 00:00:06.419 Podcast. I'm your host David Wand and welcome 00:00:06.419 --> 00:00:09.460 to this episode where we feature World Vision 00:00:09.460 --> 00:00:12.359 Canada. You can learn more about World Vision 00:00:12.359 --> 00:00:16.800 Canada at their website worldvisioncanada .ca. 00:00:17.539 --> 00:00:20.859 This project we're going to talk about cost the 00:00:20.859 --> 00:00:25.780 Canadian taxpayer 41 million dollars and it is 00:00:25.780 --> 00:00:30.359 delivering services and products in Kenya, Somalia, 00:00:30.519 --> 00:00:34.560 Bangladesh, Tanzania, and Cambodia, as well as 00:00:34.560 --> 00:00:40.159 Canada as a country. And it is delivering a variety 00:00:40.159 --> 00:00:43.460 of services and products. The title of the project 00:00:43.460 --> 00:00:47.960 is Realizing Gender Equality, Attitudinal Change, 00:00:48.320 --> 00:00:53.770 and Transformative Systems in Nutrition. But 00:00:53.770 --> 00:00:55.829 before I get into the details of the project, 00:00:56.109 --> 00:00:58.270 I want to give you a little bit of background 00:00:58.270 --> 00:01:01.649 as to why it's taken me so long to do another 00:01:01.649 --> 00:01:05.209 episode. My episodes depend on receiving performance 00:01:05.209 --> 00:01:09.609 measurement frameworks, PMFs, that list the outcomes 00:01:09.609 --> 00:01:12.870 that they are claiming they're going to achieve 00:01:12.870 --> 00:01:14.969 with the project, as well as, more importantly, 00:01:15.170 --> 00:01:17.829 the outcome indicators, because we need to know 00:01:17.829 --> 00:01:23.359 if they're valid or not. As I've said before 00:01:23.359 --> 00:01:26.879 on this podcast, before the money can be dispersed, 00:01:26.959 --> 00:01:30.420 the $41 million, they have to produce this performance 00:01:30.420 --> 00:01:33.099 measurement framework to the government of Canada, 00:01:33.180 --> 00:01:35.659 and the government of Canada has to approve it. 00:01:36.420 --> 00:01:39.519 If you go to the Project Browser website, which 00:01:39.519 --> 00:01:42.780 I'll provide in the episode notes where the project 00:01:42.780 --> 00:01:45.420 is, you'll notice they've been receiving this 00:01:45.420 --> 00:01:51.319 $41 million since May of 2023. So the PMF has 00:01:51.319 --> 00:01:56.459 already been done since 2023. I requested this 00:01:56.459 --> 00:02:01.379 PMF in October of 2024, but now it's January 00:02:01.379 --> 00:02:07.120 of 2026 when I finally received the PMF, even 00:02:07.120 --> 00:02:09.280 though I'd been complaining for several months, 00:02:09.400 --> 00:02:12.360 even sent a complaint to the Independent Information 00:02:12.360 --> 00:02:15.819 Commissioner. It wasn't until I sent an email 00:02:15.819 --> 00:02:18.060 to my local member of parliament who happens 00:02:18.060 --> 00:02:20.719 to be a liberal in the government, happens to 00:02:20.719 --> 00:02:23.180 be the minister of immigration. And I said to 00:02:23.180 --> 00:02:26.699 her, look, I'm not getting this PMF. Somebody's 00:02:26.699 --> 00:02:30.139 stalling. Come on. It's been over almost two 00:02:30.139 --> 00:02:33.919 years since I've received this PMF. What's going 00:02:33.919 --> 00:02:37.599 on? And I kind of jokingly said, I'm not going 00:02:37.599 --> 00:02:39.900 to vote for you the next time. Within a week, 00:02:39.960 --> 00:02:44.000 I received an email from, quote, her team. saying 00:02:44.000 --> 00:02:47.120 her team would get back to me. And they did. 00:02:47.280 --> 00:02:50.960 Next, about a week later, in January of 2026, 00:02:51.219 --> 00:02:54.520 after initially requesting the PMF in October 00:02:54.520 --> 00:02:59.860 of 2024, I finally received it. So that's some 00:02:59.860 --> 00:03:03.099 background for you about the politics of information. 00:03:03.900 --> 00:03:08.360 So I'll shut up now and get to the details. To 00:03:08.360 --> 00:03:09.979 give you a bit of background about the target 00:03:09.979 --> 00:03:12.479 groups and the products and services that this 00:03:12.479 --> 00:03:15.960 project spent $41 million delivering in these 00:03:15.960 --> 00:03:19.560 various developing countries. One of the target 00:03:19.560 --> 00:03:22.039 groups was Canada and the developing countries 00:03:22.039 --> 00:03:24.960 I mentioned, their populations. And one of the 00:03:24.960 --> 00:03:27.840 products they produced was art to raise awareness 00:03:27.840 --> 00:03:31.159 on gender equality for those populations. Another 00:03:31.159 --> 00:03:33.620 target group was World Vision staff themselves. 00:03:33.939 --> 00:03:37.000 They were trained on how to develop message products 00:03:37.000 --> 00:03:39.819 to raise awareness on gender equality. Probably 00:03:39.819 --> 00:03:43.259 the World Vision offices in the developing countries, 00:03:43.360 --> 00:03:45.599 the staff that work in the developing countries. 00:03:47.400 --> 00:03:54.080 Another product that the World Vision staff produced 00:03:54.080 --> 00:03:57.560 was case studies that were completed and disseminated 00:03:57.560 --> 00:04:01.080 to... gender transforming nutrition, including 00:04:01.080 --> 00:04:04.819 budgeting and monitoring, and also women's rights 00:04:04.819 --> 00:04:07.479 groups in these developing countries, another 00:04:07.479 --> 00:04:10.360 target group. They were trained on how to advocate 00:04:10.360 --> 00:04:13.219 for nutrition and sexual reproductive health 00:04:13.219 --> 00:04:16.800 services. Another target group were health officials 00:04:16.800 --> 00:04:20.019 and health facility managers in these developing 00:04:20.019 --> 00:04:22.439 countries. They were trained to supervise health 00:04:22.439 --> 00:04:25.360 workers on proper gender responsive services, 00:04:25.540 --> 00:04:28.660 budgeting and monitoring. Health systems was 00:04:28.660 --> 00:04:32.079 another target group. I don't know what human 00:04:32.079 --> 00:04:35.100 beings are there, but reports were produced on 00:04:35.100 --> 00:04:38.819 gaps in health services. Ministry of Health staff 00:04:38.819 --> 00:04:41.800 were also a target group in these developing 00:04:41.800 --> 00:04:44.759 countries that were trained on how to plan for 00:04:44.759 --> 00:04:49.800 anemia prevention and treatment and malnutrition 00:04:49.800 --> 00:04:54.529 and family planning. Teachers were another target 00:04:54.529 --> 00:04:57.009 group in these developing countries, training 00:04:57.009 --> 00:05:01.310 on adolescent health. Health facilities also 00:05:01.310 --> 00:05:04.730 received equipment and supplies for nutrition 00:05:04.730 --> 00:05:08.610 and to deliver sexual reproductive health services, 00:05:08.649 --> 00:05:11.750 as well as equipment and supplies to deliver 00:05:11.750 --> 00:05:15.589 water and sanitation. Farmers were another target 00:05:15.589 --> 00:05:17.709 group in these developing countries. They were 00:05:17.709 --> 00:05:21.329 given seeds as part of this $41 million to grow 00:05:21.329 --> 00:05:25.449 food. Faith leaders in these developing countries 00:05:25.449 --> 00:05:27.970 were also another target group that were trained 00:05:27.970 --> 00:05:31.910 on how to reduce barriers for women to access 00:05:31.910 --> 00:05:35.129 nutrition and sexual reproductive health services. 00:05:35.370 --> 00:05:39.329 Men and women were another target group that 00:05:39.329 --> 00:05:42.310 were also trained on how to reduce barriers for 00:05:42.310 --> 00:05:46.389 women to access nutrition and sexual health services. 00:05:46.750 --> 00:05:51.370 Boys and girls were another. Target group. They 00:05:51.370 --> 00:05:54.970 were trained on gender equality. And finally, 00:05:55.069 --> 00:05:57.949 there were peer networks in these developing 00:05:57.949 --> 00:06:00.410 countries. They were trained on how to deliver 00:06:00.410 --> 00:06:04.870 food and hygiene demonstrations, as well as how 00:06:04.870 --> 00:06:07.930 to develop messages specific to sexual reproductive 00:06:07.930 --> 00:06:10.769 health services. So that gives you an idea of 00:06:10.769 --> 00:06:15.589 what they spent the $41 million on. Now, lots 00:06:15.589 --> 00:06:18.290 of training, seeds, all sorts of things, equipment, 00:06:18.430 --> 00:06:21.930 supplies, as I've just described. Now the question 00:06:21.930 --> 00:06:23.750 is, if we look at their performance measurement 00:06:23.750 --> 00:06:28.569 framework, they have some outcomes. In fact, 00:06:28.569 --> 00:06:34.470 they have outcome statements, 13 outcomes that 00:06:34.470 --> 00:06:37.490 they expect to achieve after spending that $41 00:06:37.490 --> 00:06:42.209 million. And out of those 13 outcomes they expect 00:06:42.209 --> 00:06:44.949 to achieve, In the performance measurement framework, 00:06:45.250 --> 00:06:49.649 they developed 38 outcome indicators. So we're 00:06:49.649 --> 00:06:52.290 going to go through not all of them, but as many 00:06:52.290 --> 00:06:55.850 of them as we can in the next 30 or 40 minutes. 00:06:56.069 --> 00:06:58.769 And then I'll give you an idea of what I came 00:06:58.769 --> 00:07:01.490 up to as to whether or not those outcome indicators 00:07:01.490 --> 00:07:05.370 are garbage and they can't support any of the 00:07:05.370 --> 00:07:07.829 claims that they've achieved those outcomes because 00:07:07.829 --> 00:07:10.329 the indicators don't even measure what they're 00:07:10.329 --> 00:07:13.899 trying to claim they're achieving. maybe there's 00:07:13.899 --> 00:07:15.899 some good outcome indicators in there. It's a 00:07:15.899 --> 00:07:19.360 bit of both, as you'll find out. But before I 00:07:19.360 --> 00:07:23.839 do that, and you live in the Global South, or 00:07:23.839 --> 00:07:27.480 you hopefully are planning to visit the Global 00:07:27.480 --> 00:07:31.180 South to hopefully properly evaluate this project, 00:07:31.379 --> 00:07:34.560 or for that matter, any project, you may want 00:07:34.560 --> 00:07:39.379 to check out some sunglasses at akila .la, A 00:07:39.379 --> 00:07:43.319 -K -I -L -A dot L -A. In the episode notes, I'll 00:07:43.319 --> 00:07:47.139 give you that website. If you buy them, prescription 00:07:47.139 --> 00:07:50.959 lenses included in those sunglasses, all made 00:07:50.959 --> 00:07:54.399 from 50 % recycled material, you will get a 10 00:07:54.399 --> 00:07:59.379 % discount if you use my code, WANDCOOLT -T -SHIRTS, 00:07:59.399 --> 00:08:02.279 or you can just click on the link that I have 00:08:02.279 --> 00:08:07.019 in the episode notes, and you'll get a 10 % discount. 00:08:07.220 --> 00:08:09.480 And when you do that, I get a 15 % commission. 00:08:09.980 --> 00:08:12.560 when you purchase any product using my promo 00:08:12.560 --> 00:08:17.139 code. So you have your sunglasses to protect 00:08:17.139 --> 00:08:19.740 you from the sun and the global south, but you 00:08:19.740 --> 00:08:21.980 need the right clothes to be comfortable in that 00:08:21.980 --> 00:08:26.759 hot and humid climate. So what do you want to 00:08:26.759 --> 00:08:30.139 do there? You want to check out hypernaturalstyle 00:08:30.139 --> 00:08:34.960 .com. Some men's polos, tees, and button -ups. 00:08:35.059 --> 00:08:38.419 You get 20 % off any product when you use my 00:08:38.419 --> 00:08:42.419 promo code WANDCOOLT -SHIRTS. And I get a 20 00:08:42.419 --> 00:08:45.500 % commission of the total purchase you make when 00:08:45.500 --> 00:08:50.500 you use my promo code WANDCOOLT -SHIRTS. So what 00:08:50.500 --> 00:08:53.700 is so cool about these t -shirts? Well, they 00:08:53.700 --> 00:08:56.500 have a patent -pending HyperCool Jade technology. 00:08:57.340 --> 00:09:00.080 that when it's hot, it lowers your body temperature 00:09:00.080 --> 00:09:03.580 by three to five degrees. They are also breathable 00:09:03.580 --> 00:09:07.080 thanks to Supina cotton, and they are sold in 00:09:07.080 --> 00:09:11.059 premium retailers like Nordstrom, and were voted 00:09:11.059 --> 00:09:15.759 the best men's polo shirts in 2024 in Men's Journal, 00:09:15.960 --> 00:09:20.340 Forbes, and Esquire. So let's get on with the 00:09:20.340 --> 00:09:23.220 outcomes and the outcome indicators. So let's 00:09:23.220 --> 00:09:27.340 start with this first. Outcome statement increased 00:09:27.340 --> 00:09:29.960 engagement and awareness of the Canadian public 00:09:29.960 --> 00:09:33.100 on the impact of official development assistance 00:09:33.100 --> 00:09:37.279 in the context of gender equitable and responsive 00:09:37.279 --> 00:09:40.460 nutrition, health and sexual reproductive health 00:09:40.460 --> 00:09:43.039 programming. They have two indicators for there. 00:09:43.320 --> 00:09:46.779 The first one is number of Canadians aged 18 00:09:46.779 --> 00:09:50.580 to 65 years of age who are engaging with content 00:09:50.580 --> 00:09:53.889 related. to nutrition, health, sexual reproductive 00:09:53.889 --> 00:09:57.070 health, gender issues in international development 00:09:57.070 --> 00:10:02.350 contexts. So what's interesting here is that 00:10:02.350 --> 00:10:04.129 sounds like a pretty good indicator, but then 00:10:04.129 --> 00:10:07.269 you have to look at the PMF to see what exactly 00:10:07.269 --> 00:10:12.169 they do to measure it. Well, bad news, dudes. 00:10:12.250 --> 00:10:15.429 They do not use a survey of the Canadian public. 00:10:15.649 --> 00:10:18.690 So they can't even claim. that they're achieving 00:10:18.690 --> 00:10:20.610 their target group, which is raised awareness 00:10:20.610 --> 00:10:25.129 of the Canadian public. All they talk about is 00:10:25.129 --> 00:10:29.210 the number of likes and comments online and the 00:10:29.210 --> 00:10:31.389 number of in -person conversations at events. 00:10:31.690 --> 00:10:33.830 And they do it at baseline, midline, headline, 00:10:33.929 --> 00:10:36.570 and annually. And it's going to show an increase 00:10:36.570 --> 00:10:39.669 in engagement and awareness. But they do it from 00:10:39.669 --> 00:10:42.730 the project database. But that's not a sample 00:10:42.730 --> 00:10:45.830 survey of the, quote, Canadian public. So they 00:10:45.830 --> 00:10:48.120 can't make the claim. that the Canadian public 00:10:48.120 --> 00:10:51.659 has experienced an increase in awareness specific 00:10:51.659 --> 00:10:54.639 to gender equity nutrition, sexual reproductive 00:10:54.639 --> 00:10:57.840 health, based on Canadian foreign aid has increased. 00:10:58.080 --> 00:11:00.820 They can't. They've got the wrong target group. 00:11:00.899 --> 00:11:03.559 They can claim that the project people that engage 00:11:03.559 --> 00:11:06.080 with the project, but they haven't done a sample 00:11:06.080 --> 00:11:09.980 of the general Canadian public population. So 00:11:09.980 --> 00:11:12.360 they've got the wrong outcome. So they can't 00:11:12.360 --> 00:11:15.039 make that claim. They could change it. They could 00:11:15.039 --> 00:11:18.049 reduce it down and just say, You know, if people 00:11:18.049 --> 00:11:20.809 increase over time their engagement with us, 00:11:20.909 --> 00:11:25.570 whoever shows up to these events or send comments 00:11:25.570 --> 00:11:28.090 in online, fine, we could show there's been an 00:11:28.090 --> 00:11:30.250 increase in engagement. But they can't make the 00:11:30.250 --> 00:11:34.230 claim that the Canadian public at large has increased 00:11:34.230 --> 00:11:38.049 their awareness. Right? And it's the same for 00:11:38.049 --> 00:11:40.889 the specific to the awareness of official development 00:11:40.889 --> 00:11:44.370 assistance impact has increased. They've got 00:11:44.370 --> 00:11:46.639 the wrong target group. Because they haven't 00:11:46.639 --> 00:11:50.139 done a sample survey, right? The good news is, 00:11:50.159 --> 00:11:53.019 as you'll find out later, some of the other indicators 00:11:53.019 --> 00:11:55.639 they've got, they do a very good job because 00:11:55.639 --> 00:12:00.799 they actually do do a household survey. So that's 00:12:00.799 --> 00:12:05.159 an example of two indicators that really are 00:12:05.159 --> 00:12:08.240 not doing it properly. So they're not good. So 00:12:08.240 --> 00:12:11.759 if we keep going, we can look at another outcome. 00:12:12.720 --> 00:12:16.480 expected to achieve increased utilization of 00:12:16.480 --> 00:12:20.440 gender transformative framework for nutrition 00:12:20.440 --> 00:12:23.639 by key national and subnational stakeholders 00:12:23.639 --> 00:12:27.379 for mainstreaming of gender equality into the 00:12:27.379 --> 00:12:30.139 planning budgeting monitoring and learning processes 00:12:30.139 --> 00:12:33.860 of multi -sectoral nutrition programs and policies 00:12:33.860 --> 00:12:37.940 they've got a few indicators for that you'll 00:12:37.940 --> 00:12:40.870 see well you can't see you can only hear First 00:12:40.870 --> 00:12:44.330 one is percent of national, subnational government, 00:12:44.490 --> 00:12:49.509 civil society, NGO stakeholders endorsing GTFN, 00:12:49.529 --> 00:12:52.750 gender transformative framework network, mainstreaming 00:12:52.750 --> 00:12:56.230 within multisectoral programs and policies. So 00:12:56.230 --> 00:12:58.570 that's one indicator they have. Another indicator 00:12:58.570 --> 00:13:02.909 they have is this framework, gender transformative 00:13:02.909 --> 00:13:07.070 framework nutrition used at least in one subnational 00:13:07.070 --> 00:13:10.279 level government sector. in the project site 00:13:10.279 --> 00:13:13.639 for mainstreaming of gender equality into the 00:13:13.639 --> 00:13:15.419 planning, budgeting, monitoring, and learning 00:13:15.419 --> 00:13:18.620 processes of multisectoral nutrition program. 00:13:19.120 --> 00:13:22.240 But if you can, if you look at their indicators, 00:13:22.460 --> 00:13:27.539 the first one, they talk about endorsement, endorsing 00:13:27.539 --> 00:13:30.759 mainstreaming of GTFN. That's not the same as 00:13:30.759 --> 00:13:33.419 increased utilization. They could say, oh yeah, 00:13:33.460 --> 00:13:35.600 we've endorsed it, but you've got to look to 00:13:35.600 --> 00:13:37.799 see if they've actually used it. So it's not 00:13:37.799 --> 00:13:40.460 a valid indicator. It doesn't validly measure 00:13:40.460 --> 00:13:44.039 whether actual utilization is increased. If they 00:13:44.039 --> 00:13:46.419 want to go back and change the outcome statement 00:13:46.419 --> 00:13:50.500 to endorsed, fine, but they didn't. So they can't 00:13:50.500 --> 00:13:52.899 make that claim. The other problem is with the 00:13:52.899 --> 00:13:57.620 second indicator used in at least one subnational 00:13:57.620 --> 00:14:00.419 level government sector. If you look at the data 00:14:00.419 --> 00:14:03.840 source, this is where the PMF is important because 00:14:03.840 --> 00:14:05.799 you wouldn't know this unless you got the PMF. 00:14:06.379 --> 00:14:09.600 They look at the project database regularly to 00:14:09.600 --> 00:14:13.360 see if it's been used, right? Okay. You have 00:14:13.360 --> 00:14:16.360 to just trust that whoever's entered the information 00:14:16.360 --> 00:14:20.139 in the database, they've just said, yeah, they've 00:14:20.139 --> 00:14:22.659 used it. But it's not really a good measure. 00:14:22.720 --> 00:14:24.960 I think it would be better that they actually 00:14:24.960 --> 00:14:29.059 go in to the government sector to see if they've 00:14:29.059 --> 00:14:32.080 actually used it. You know, developing some criteria. 00:14:32.799 --> 00:14:35.220 Did they use it or not? They shouldn't be going 00:14:35.220 --> 00:14:39.279 into a secondary data source as the project database. 00:14:39.399 --> 00:14:42.940 So I don't trust it. It's a bias, right? You 00:14:42.940 --> 00:14:46.080 should actually just show up a third party evaluator 00:14:46.080 --> 00:14:48.919 and pick a government sector, go in and say, 00:14:49.000 --> 00:14:53.600 did you use the GTFN? Did you use it? And they 00:14:53.600 --> 00:14:56.340 could check by seeing. actual documents rather 00:14:56.340 --> 00:14:58.220 than just going into the database. So that's 00:14:58.220 --> 00:15:00.919 another two indicators that are not well designed. 00:15:01.059 --> 00:15:03.440 They can't support the claim that utilization 00:15:03.440 --> 00:15:07.360 has been increased. Another outcome, increased 00:15:07.360 --> 00:15:11.000 capacity of local community structures such as 00:15:11.000 --> 00:15:13.779 citizen voice and action groups, community health 00:15:13.779 --> 00:15:17.840 committees to engage in evidence -based advocacy 00:15:17.840 --> 00:15:21.259 and policy dialogue at national and sub -national 00:15:21.259 --> 00:15:24.720 levels on gender equitable rights. to nutrition, 00:15:24.899 --> 00:15:27.000 health, and sexual reproductive health rights. 00:15:27.419 --> 00:15:30.600 So they've got three indicators here. First one 00:15:30.600 --> 00:15:33.519 is percent of community -based structures that 00:15:33.519 --> 00:15:36.779 are monitoring gender -responsive and adolescent 00:15:36.779 --> 00:15:39.580 -friendly service delivery at health facilities 00:15:39.580 --> 00:15:44.620 disaggregated by country. Again, if you read 00:15:44.620 --> 00:15:48.519 that, monitoring, they're equating their ability 00:15:48.519 --> 00:15:52.240 to monitor with their increased capacity to advocate. 00:15:52.419 --> 00:15:55.639 That's wrong. They're not the same thing. Now, 00:15:55.639 --> 00:15:57.519 monitoring is, you could argue, is an indirect 00:15:57.519 --> 00:16:00.919 measure that they're actually capable of advocating. 00:16:01.279 --> 00:16:04.500 You could argue that. So I've given it a green, 00:16:04.700 --> 00:16:09.059 meaning I say it's okay. They do an assessment 00:16:09.059 --> 00:16:11.440 of the health staff and look at the project database. 00:16:11.840 --> 00:16:15.039 We can assume that this assessment looks at the 00:16:15.039 --> 00:16:18.480 ability to monitor. So it's an indirect measure. 00:16:18.639 --> 00:16:20.740 I've given it a green light. I would say that's 00:16:20.740 --> 00:16:24.039 a good indicator for that. outcome of increased 00:16:24.039 --> 00:16:28.019 capacity. The second one indicator is percent 00:16:28.019 --> 00:16:30.720 of trained members who understand basic health 00:16:30.720 --> 00:16:34.220 rights and how these rights are articulated under 00:16:34.220 --> 00:16:37.740 local law disaggregated by country. And if you 00:16:37.740 --> 00:16:41.340 take a look there, they actually have an objective 00:16:41.340 --> 00:16:44.720 testing measure of this understanding on basic 00:16:44.720 --> 00:16:46.980 health rights and how to articulate these rights. 00:16:47.179 --> 00:16:50.809 So I checked for the existence of an actual test 00:16:50.809 --> 00:16:54.509 there is a test so that's excellent this is a 00:16:54.509 --> 00:16:57.110 good this is a green light good indicator again 00:16:57.110 --> 00:17:01.149 the only issue there is and i again i haven't 00:17:01.149 --> 00:17:04.309 said it's bad enough to give it a red light is 00:17:04.309 --> 00:17:08.630 would the increase in that percentage who passed 00:17:08.630 --> 00:17:12.869 the test be significantly greater than a group 00:17:12.869 --> 00:17:15.569 of other individuals who didn't participate in 00:17:15.569 --> 00:17:20.430 the project meaning what i call impact Some people 00:17:20.430 --> 00:17:22.150 disagree with that definition, but that's the 00:17:22.150 --> 00:17:24.769 one I use for the podcast. Not only has it been 00:17:24.769 --> 00:17:27.309 achieved, effective, but it's been achieved so 00:17:27.309 --> 00:17:30.730 greatly that the increase on the indicator is 00:17:30.730 --> 00:17:33.849 so great, it is statistically significant, and 00:17:33.849 --> 00:17:37.430 someone can argue it's not due to chance. But 00:17:37.430 --> 00:17:39.269 we're not looking at that right now. But that's 00:17:39.269 --> 00:17:42.930 a good indicator. So the last indicator on that 00:17:42.930 --> 00:17:46.170 outcome is extent of engagement of women and 00:17:46.170 --> 00:17:49.440 men. community leaders in targeted villages with 00:17:49.440 --> 00:17:52.539 local government on nutrition, health, sexual 00:17:52.539 --> 00:17:54.440 reproductive health rights, and gender issues. 00:17:55.420 --> 00:18:00.539 Now, for that one, if you look again at the PMF, 00:18:00.599 --> 00:18:03.059 if you don't have the PMF, you can't find this. 00:18:03.160 --> 00:18:05.299 But then you look at the data source, and this 00:18:05.299 --> 00:18:07.460 is where it gets, tell me about your feelings. 00:18:08.059 --> 00:18:10.380 Engagement is measured with the community leaders 00:18:10.380 --> 00:18:15.059 themselves. So there's a bias. And using focus 00:18:15.059 --> 00:18:19.289 group discussions. So it's going to lead to self 00:18:19.289 --> 00:18:21.430 -reporting bias. So for this indicator, I gave 00:18:21.430 --> 00:18:24.670 them a red light. Not a good measure. Focus groups. 00:18:25.250 --> 00:18:28.430 They need to get a better measure of engagement 00:18:28.430 --> 00:18:31.990 externally to show that the level of engagement 00:18:31.990 --> 00:18:35.470 is actually quantitatively increased. So that's 00:18:35.470 --> 00:18:40.150 not good. So not bad. Three indicators. Two of 00:18:40.150 --> 00:18:43.390 them really good. One of them bad. So if we look 00:18:43.390 --> 00:18:47.089 at another outcome, improved. capacity of sub 00:18:47.089 --> 00:18:49.829 -national health management teams for effective 00:18:49.829 --> 00:18:53.369 gender equitable and responsive nutrition health 00:18:53.369 --> 00:18:55.970 and sexual reproductive health systems governance 00:18:55.970 --> 00:19:00.529 including planning budgeting reporting so improved 00:19:00.529 --> 00:19:04.730 capacity again they have three indicators the 00:19:04.730 --> 00:19:08.170 first one is called percent of health facility 00:19:08.170 --> 00:19:11.849 scoring greater than 80 on their health facility 00:19:11.849 --> 00:19:15.680 planning assessment by the team This indicator 00:19:15.680 --> 00:19:18.839 is good. We can only assume that the test validly 00:19:18.839 --> 00:19:21.740 measures the technical capacity of the team to 00:19:21.740 --> 00:19:24.099 do, quote, planning, budgeting, and reporting. 00:19:24.900 --> 00:19:28.559 But again, I've given it a green light, but also 00:19:28.559 --> 00:19:31.380 the impact issue is, could the team, without 00:19:31.380 --> 00:19:34.400 this $40 million, been able to teach themselves 00:19:34.400 --> 00:19:37.460 how to do this planning, budgeting, and reporting? 00:19:37.839 --> 00:19:41.460 So that should be put in the PMF. is how are 00:19:41.460 --> 00:19:43.720 they going to show that that increase in percent 00:19:43.720 --> 00:19:48.740 is statistically significant, right? Now, it 00:19:48.740 --> 00:19:51.019 could be as simple as just measuring repeatedly 00:19:51.019 --> 00:19:54.740 over time, showing the percent going up, and 00:19:54.740 --> 00:19:56.819 that's good enough. You don't need a control 00:19:56.819 --> 00:20:00.240 group, but there should be some sort of quantitative 00:20:00.240 --> 00:20:05.119 analysis to show that it's been going up, and 00:20:05.119 --> 00:20:08.289 not only going up, scoring greater than 80. but 00:20:08.289 --> 00:20:11.049 statistically significantly going up. So that's 00:20:11.049 --> 00:20:14.390 another way to look at it. But I gave them a 00:20:14.390 --> 00:20:16.230 green light for that indicator. The next two 00:20:16.230 --> 00:20:20.210 indicators, I gave red lights. So the next indicator 00:20:20.210 --> 00:20:23.849 is percent of health workers who report they 00:20:23.849 --> 00:20:27.029 trust the leadership of the management teams 00:20:27.029 --> 00:20:30.470 in ensuring effective gender equitable services. 00:20:31.470 --> 00:20:33.569 Some people will say, oh, we're doing that to 00:20:33.569 --> 00:20:36.730 see if we can check against. improved capacity 00:20:36.730 --> 00:20:39.529 of those health management teams i gave it a 00:20:39.529 --> 00:20:42.309 red light because it's if you look at the data 00:20:42.309 --> 00:20:44.569 source it's the health team members themselves 00:20:44.569 --> 00:20:48.970 so there's self -reporting bias there and also 00:20:48.970 --> 00:20:51.609 measuring trust has nothing to do with increased 00:20:51.609 --> 00:20:54.609 technical capacity to deliver responsive budgeting 00:20:54.609 --> 00:20:57.970 etc right so that that's not a good indicator 00:20:57.970 --> 00:21:01.670 for those two reasons self -reporting bias because 00:21:01.670 --> 00:21:04.210 they're asking themselves If they trust themselves, 00:21:04.329 --> 00:21:08.210 you have to look at the data source, right? When 00:21:08.210 --> 00:21:10.809 they say health workers, according to the data 00:21:10.809 --> 00:21:13.390 source, it's not health workers. It's the health 00:21:13.390 --> 00:21:16.230 team members themselves. But even if I'm wrong 00:21:16.230 --> 00:21:19.789 there, I wouldn't use it. I'd throw it out. The 00:21:19.789 --> 00:21:24.349 third indicator for this outcome was percent 00:21:24.349 --> 00:21:27.849 of team members who have access to planning, 00:21:28.009 --> 00:21:30.829 budgeting, and governance data. And here we go 00:21:30.829 --> 00:21:35.180 again. who report by themselves using the data 00:21:35.180 --> 00:21:37.440 to make decisions about the provision of gender 00:21:37.440 --> 00:21:41.059 equitable services. So instead, we've got self 00:21:41.059 --> 00:21:43.700 -reporting bias again. What I would do instead, 00:21:43.880 --> 00:21:46.519 this is not a good indicator, is I would actually 00:21:46.519 --> 00:21:50.240 sample the reports to see if the data was actually 00:21:50.240 --> 00:21:54.440 used to, quote, make decisions of gender equitable 00:21:54.440 --> 00:21:58.640 services, right? I'd do that by an external evaluator 00:21:58.640 --> 00:22:01.859 instead. Self -reporting bias. So out of those 00:22:01.859 --> 00:22:04.220 three indicators, only one was good. The other 00:22:04.220 --> 00:22:09.339 two were bad. Another outcome, increased health, 00:22:09.500 --> 00:22:12.240 infrastructure, information technology, and capacity 00:22:12.240 --> 00:22:15.660 for prevention and management of acute malnutrition 00:22:15.660 --> 00:22:19.059 in most marginalized communities. The problem 00:22:19.059 --> 00:22:23.039 here is three indicators. Number of school -going 00:22:23.039 --> 00:22:25.720 adolescent girls who received the recommended 00:22:25.720 --> 00:22:29.079 scheme of weekly iron folic acid supplementation. 00:22:29.559 --> 00:22:32.460 Second indicator, percent of children admitted 00:22:32.460 --> 00:22:35.279 treated for moderate or severe acute malnutrition. 00:22:35.680 --> 00:22:40.220 And third indicator was percent of health facilities 00:22:40.220 --> 00:22:43.880 in remote catchment areas rehabilitated received 00:22:43.880 --> 00:22:48.000 support for HMIS strengthening to improve gender 00:22:48.000 --> 00:22:51.720 equitable nutrition. Now, they're claiming here 00:22:51.720 --> 00:22:53.880 that if you just deliver those services to the 00:22:53.880 --> 00:22:57.500 girls and the children, that that in turn can 00:22:57.500 --> 00:23:01.200 be equated with increased capacity for prevention 00:23:01.200 --> 00:23:04.980 and management. You could argue that. I'm changing 00:23:04.980 --> 00:23:08.220 my mind here. I'm thinking these are just outputs. 00:23:08.960 --> 00:23:11.779 But what they're trying to tell you here is that 00:23:11.779 --> 00:23:14.660 if we deliver that folic acid and we deliver 00:23:14.660 --> 00:23:17.400 that treatment to the children for malnutrition, 00:23:17.500 --> 00:23:20.400 and if we deliver to those health facilities 00:23:20.400 --> 00:23:24.460 support, whatever that means, then we could argue... 00:23:24.730 --> 00:23:28.509 that that in turn is the same as increased capacity 00:23:28.509 --> 00:23:31.470 to prevent and manage. So I'm going to change 00:23:31.470 --> 00:23:34.789 my mind here and give this a green light instead 00:23:34.789 --> 00:23:40.009 of a red light, because it is a good way of saying 00:23:40.009 --> 00:23:42.230 we're delivering these services and those services, 00:23:42.269 --> 00:23:44.910 just getting them delivered. It is charity, but 00:23:44.910 --> 00:23:47.670 that's okay. That's another issue. So I'll leave 00:23:47.670 --> 00:23:51.069 it at that as a valid measure of achieving that 00:23:51.069 --> 00:23:54.240 outcome. Another one is increased knowledge and 00:23:54.240 --> 00:23:57.440 skills of health care providers on gender equitable 00:23:57.440 --> 00:24:00.920 and responsive nutrition, health and sexual reproductive 00:24:00.920 --> 00:24:05.099 health services. Now, again, this is the FGD 00:24:05.099 --> 00:24:08.380 problem. The first they've got, how many indicators 00:24:08.380 --> 00:24:12.000 have they got for this outcome? They've got three. 00:24:12.559 --> 00:24:16.440 So the first one, extent of change in knowledge 00:24:16.440 --> 00:24:19.000 and skills among health care providers after 00:24:19.000 --> 00:24:22.599 receiving training. Fantastic. Problem is, go 00:24:22.599 --> 00:24:25.579 to the data source. They're not measuring them 00:24:25.579 --> 00:24:27.480 objectively on their knowledge and skill levels. 00:24:27.640 --> 00:24:30.500 They're holding a focus group discussion. Instead, 00:24:30.799 --> 00:24:33.160 they're holding key informant interviews. Instead, 00:24:33.440 --> 00:24:40.440 no need to do it. So it's not important. Only 00:24:40.440 --> 00:24:42.880 use focus group discussions, key informant interviews, 00:24:42.980 --> 00:24:46.000 if your quantitative indicator is not achieving 00:24:46.000 --> 00:24:49.259 your outcome. Then you can go back to the group 00:24:49.259 --> 00:24:51.470 and say, why? What's going on here? What's the 00:24:51.470 --> 00:24:54.470 problem? So if you look at the second indicator, 00:24:54.809 --> 00:24:58.269 it's excellent. Percent of health workers who 00:24:58.269 --> 00:25:02.009 have passed knowledge competency tests following 00:25:02.009 --> 00:25:04.990 training. It's great. It's a pre and post test. 00:25:05.509 --> 00:25:10.609 So that's very good. Love it. Fantastic. Third 00:25:10.609 --> 00:25:13.569 one indicator for increased knowledge and skills 00:25:13.569 --> 00:25:16.569 is percent of households received at least one 00:25:16.569 --> 00:25:19.079 visit. from a community health worker in the 00:25:19.079 --> 00:25:22.720 last month. Well, they could show up and, like 00:25:22.720 --> 00:25:25.740 me, when I was in Sierra Leone, have a jug of 00:25:25.740 --> 00:25:28.619 palm wine and have no knowledge and no skills 00:25:28.619 --> 00:25:32.400 and get a free meal. Cassava leaf and rice, my 00:25:32.400 --> 00:25:36.119 favorite dish. Right? So this indicator, in my 00:25:36.119 --> 00:25:38.859 opinion, is useless. Third indicator, percent 00:25:38.859 --> 00:25:41.299 of households receiving a visit has nothing to 00:25:41.299 --> 00:25:43.460 do with increased knowledge of health worker 00:25:43.460 --> 00:25:46.640 you just trained. This is an output. So we don't 00:25:46.640 --> 00:25:49.940 know. That health care provider actually has 00:25:49.940 --> 00:25:52.660 increased levels of knowledge and skill. Unless 00:25:52.660 --> 00:25:54.640 one of the measures of knowledge and skill is 00:25:54.640 --> 00:25:56.880 how to find the health center, which on a motorcycle 00:25:56.880 --> 00:25:59.500 in Sierra Leone can be a bit of a challenge. 00:25:59.700 --> 00:26:01.619 Took me about six months to figure that out. 00:26:02.039 --> 00:26:05.640 But no, in this case, it's not good. They should 00:26:05.640 --> 00:26:08.200 just use that previous indicator, pre and post 00:26:08.200 --> 00:26:11.460 test, good enough. Another outcome, increased 00:26:11.460 --> 00:26:15.910 gender equitable access to Nutrition -sensitive 00:26:15.910 --> 00:26:20.230 interventions in wash, water, sanitation, and 00:26:20.230 --> 00:26:22.710 agriculture, including production and consumption 00:26:22.710 --> 00:26:26.450 of biofortified crops and nutrient -dense local 00:26:26.450 --> 00:26:30.130 vegetable and fruits. This one's interesting. 00:26:30.309 --> 00:26:32.589 The indicator for this in terms of increased 00:26:32.589 --> 00:26:37.970 access is the percent of mothers planted biofortified 00:26:37.970 --> 00:26:41.390 crops after receiving planting material directly. 00:26:42.069 --> 00:26:45.279 Now, you'd assume... They're going to have access 00:26:45.279 --> 00:26:48.599 to this food after they plant the seeds, right? 00:26:48.779 --> 00:26:51.720 But they ask the mothers. Don't ask the mothers. 00:26:51.720 --> 00:26:54.000 Just say to the mothers, excuse me, show me your 00:26:54.000 --> 00:26:57.140 plot. So don't survey the mothers. Instead, they 00:26:57.140 --> 00:26:59.240 should survey the crops to see if they were planted 00:26:59.240 --> 00:27:03.500 and see if the crops survived, right? So that's 00:27:03.500 --> 00:27:06.500 a no -go. Now, they do have project progress 00:27:06.500 --> 00:27:09.400 reports. They might show this, but we do not 00:27:09.400 --> 00:27:11.960 know. All we can see is the indicator is useless. 00:27:12.970 --> 00:27:17.430 Same for the second one. Percent of mothers who 00:27:17.430 --> 00:27:20.490 planted local vegetables after receiving the 00:27:20.490 --> 00:27:22.789 seeds. Again, they surveyed the mothers. They 00:27:22.789 --> 00:27:24.650 shouldn't. They should just survey the vegetables 00:27:24.650 --> 00:27:27.849 to see if one, they were planted and two, to 00:27:27.849 --> 00:27:30.670 see if the vegetables survived, right? And the 00:27:30.670 --> 00:27:34.569 third one, same problem, planted fruit trees. 00:27:35.150 --> 00:27:37.769 So I'm assuming from the data source, they're 00:27:37.769 --> 00:27:41.430 asking the mothers instead of going. Now, maybe 00:27:41.430 --> 00:27:43.750 it's in the survey manual. When you ask the mother, 00:27:43.789 --> 00:27:47.170 she says yes. Then go outside and check the plot 00:27:47.170 --> 00:27:50.390 and to see if the fruit trees were planted and 00:27:50.390 --> 00:27:53.589 survived. So right from that reading, they've 00:27:53.589 --> 00:27:56.170 got the wrong data source. Maybe World Vision, 00:27:56.289 --> 00:27:58.509 if they have the courage to show up, they'll 00:27:58.509 --> 00:28:01.569 say, David, we can tell you by fact that in the 00:28:01.569 --> 00:28:04.190 manual, when they did the survey, they went outside 00:28:04.190 --> 00:28:07.589 and checked. But I don't know. It's funny because 00:28:07.589 --> 00:28:10.210 the last indicator, there's another one. percent 00:28:10.210 --> 00:28:13.130 of households with hygienic toilets after receiving 00:28:13.130 --> 00:28:16.329 information on appropriate water and sanitation 00:28:16.329 --> 00:28:19.829 and hygiene. Wash. We can assume that the household 00:28:19.829 --> 00:28:22.990 survey includes a question. Is there a hygienic 00:28:22.990 --> 00:28:25.849 toilet? So that one, they can actually go and 00:28:25.849 --> 00:28:29.150 see it. So they should do the same for the vegetables, 00:28:29.349 --> 00:28:32.529 for the fruits, and for the fortified crops, 00:28:32.849 --> 00:28:36.349 right? So out of the four indicators, the hygienic 00:28:36.349 --> 00:28:38.869 toilet is a good one. The other three are not. 00:28:39.289 --> 00:28:41.990 So let's see, what are we running out of time 00:28:41.990 --> 00:28:43.910 here? How much time have we got here? Well, we're 00:28:43.910 --> 00:28:46.890 at 30 minutes. We can keep going for a bit. I 00:28:46.890 --> 00:28:49.589 think I'll stop around 40. Here's another one. 00:28:50.549 --> 00:28:53.509 Increase knowledge of key community -level influencers 00:28:53.509 --> 00:28:57.750 to challenge discriminatory social norms that 00:28:57.750 --> 00:29:00.990 limit women and adolescent girls from accessing 00:29:00.990 --> 00:29:04.470 gender -equitable, nutrition -specific health, 00:29:04.630 --> 00:29:07.849 WASH, and sexual reproductive health services. 00:29:08.619 --> 00:29:10.759 And they've got, I think they got three indicators 00:29:10.759 --> 00:29:13.720 for this outcome. And the first one, percent 00:29:13.720 --> 00:29:16.799 of mothers of children under five who know at 00:29:16.799 --> 00:29:20.019 least three modern family planning methods after 00:29:20.019 --> 00:29:22.720 receiving the information from religious leaders 00:29:22.720 --> 00:29:26.920 and lead fathers. It's a good indicator. It checks 00:29:26.920 --> 00:29:30.059 on whether that knowledge transfer from the influencer 00:29:30.059 --> 00:29:35.000 to the mother actually happened. Only issue again 00:29:35.000 --> 00:29:37.920 is, as I've mentioned before, is could that have 00:29:37.920 --> 00:29:40.859 happened without spending $41 million? Could 00:29:40.859 --> 00:29:43.579 they have figured that out, modern family planning 00:29:43.579 --> 00:29:47.640 methods, without spending $41 million? By talking 00:29:47.640 --> 00:29:50.619 to other mothers, going down the street, whatever. 00:29:50.880 --> 00:29:56.640 So what they should insert in the PMF is, over 00:29:56.640 --> 00:29:59.339 time, was that percent that increased, was it 00:29:59.339 --> 00:30:02.619 statistically significant, one -tailed, right 00:30:02.619 --> 00:30:08.119 -tailed, P less than .05, right? Or they could 00:30:08.119 --> 00:30:10.940 use a comparison group, but they don't have to, 00:30:11.000 --> 00:30:14.359 of a group of mothers. But they should be putting 00:30:14.359 --> 00:30:17.460 that in the PMF to show, yeah, the percent not 00:30:17.460 --> 00:30:20.779 only went up, but it went up statistically significantly. 00:30:21.440 --> 00:30:24.779 Therefore, using my definition, it has, quote, 00:30:24.900 --> 00:30:28.160 impact. It's not just a difference, effective, 00:30:28.500 --> 00:30:31.420 but the difference was so big, they could say, 00:30:31.500 --> 00:30:34.940 wow, it's an impact. I know that DAC uses impact 00:30:34.940 --> 00:30:39.039 broader. They say unintended and intended effects 00:30:39.039 --> 00:30:42.740 above and beyond the project in other areas. 00:30:42.900 --> 00:30:47.019 That's not my definition. Second indicator, changes 00:30:47.019 --> 00:30:50.319 in perception and opinions among key community 00:30:50.319 --> 00:30:53.660 influencers to challenge discriminatory social 00:30:53.660 --> 00:30:57.599 norms that limit women and adolescent girls from 00:30:57.599 --> 00:31:01.039 accessing gender -equitable, nutrition -specific 00:31:01.039 --> 00:31:04.640 health, wash, and sexual reproductive health 00:31:04.640 --> 00:31:09.079 services. The problem there is the mother's knowledge 00:31:09.079 --> 00:31:12.059 is first dependent upon the influencer that gives 00:31:12.059 --> 00:31:15.319 it to them, also having that knowledge. Where 00:31:15.319 --> 00:31:18.440 is that test of the influencer's knowledge, like 00:31:18.440 --> 00:31:20.480 the knowledge test they gave to the health worker 00:31:20.480 --> 00:31:23.619 previously? This is missing and should be done. 00:31:24.059 --> 00:31:27.799 Also, qualitative focus group discussions with 00:31:27.799 --> 00:31:31.000 these influencers is not a valid measure of perception 00:31:31.000 --> 00:31:34.440 change. Need to include in the test on knowledge. 00:31:35.019 --> 00:31:37.619 that perception change. You can do that quantitatively. 00:31:38.079 --> 00:31:43.220 So it's not a good indicator. The last one, percent 00:31:43.220 --> 00:31:45.720 of mothers of children under five who know at 00:31:45.720 --> 00:31:48.380 least three benefits of exclusive breastfeeding 00:31:48.380 --> 00:31:51.259 after receiving the information from religious 00:31:51.259 --> 00:31:54.500 leaders and lead fathers. Again, excellent indicator. 00:31:56.099 --> 00:31:58.619 The only issue again is, is it statistically 00:31:58.619 --> 00:32:02.160 significant compared to if we didn't spend that 00:32:02.160 --> 00:32:05.549 $41 million at all? Could that percent go up 00:32:05.549 --> 00:32:08.690 anyways? So we've got to make sure if we're going 00:32:08.690 --> 00:32:13.690 to blow $41 million that our intervention is 00:32:13.690 --> 00:32:16.950 increasing that percent way, way, way, way better 00:32:16.950 --> 00:32:22.190 of those mothers that know about exclusive breastfeeding 00:32:22.190 --> 00:32:24.750 and how it's beneficial. That percent of mothers 00:32:24.750 --> 00:32:28.150 goes way up higher because of the project rather 00:32:28.150 --> 00:32:31.690 than if we didn't have the project at all. So 00:32:31.690 --> 00:32:35.589 we'll move on to... A few more indicators, improved 00:32:35.589 --> 00:32:38.990 knowledge and self -efficacy of poor and most 00:32:38.990 --> 00:32:41.990 marginalized women and adolescent girls to negotiate 00:32:41.990 --> 00:32:46.289 access to and control over nutrition -specific 00:32:46.289 --> 00:32:48.890 and sensitive health and sexual reproductive 00:32:48.890 --> 00:32:51.970 health rights services. We got three indicators 00:32:51.970 --> 00:32:57.410 there. But again, the indicator, first one, percent 00:32:57.410 --> 00:33:00.769 of women who decided to use family planning alone 00:33:00.769 --> 00:33:04.630 or jointly with their partners. Sounds like a 00:33:04.630 --> 00:33:07.109 pretty good indicator. But what they do, if you 00:33:07.109 --> 00:33:09.470 look at the PMF, is they indicate they've got 00:33:09.470 --> 00:33:12.329 target areas. So that means that there's areas 00:33:12.329 --> 00:33:15.849 where they're not operating to increase this 00:33:15.849 --> 00:33:21.170 knowledge and self -efficacy to negotiate access, 00:33:21.490 --> 00:33:26.130 right? So they need to show that in the non -project 00:33:26.130 --> 00:33:29.450 areas that that indicator, percent of women who 00:33:29.450 --> 00:33:33.440 decided to use family planning, is statistically 00:33:33.440 --> 00:33:37.579 significantly lower than the target areas. It's 00:33:37.579 --> 00:33:41.940 got to show impact, right? Because if the project 00:33:41.940 --> 00:33:46.039 $41 million spent doesn't really make a statistically 00:33:46.039 --> 00:33:50.000 significant difference in that percent, it brings 00:33:50.000 --> 00:33:53.099 into question the efficacy and impact of the 00:33:53.099 --> 00:33:56.539 project. Second indicator, percent of school 00:33:56.539 --> 00:33:59.319 -going adolescent girls and boys with appropriate 00:33:59.319 --> 00:34:03.420 knowledge. on adolescent nutrition, sexual reproductive 00:34:03.420 --> 00:34:07.539 health, et cetera. Now, what have we got here? 00:34:07.740 --> 00:34:11.320 I think we don't know. I must confess, I don't 00:34:11.320 --> 00:34:13.480 have the answer here because I'd have to look 00:34:13.480 --> 00:34:16.179 in the PMF and that will take a bit of time. 00:34:16.679 --> 00:34:19.980 So we'll look at the third indicator. Again, 00:34:20.099 --> 00:34:22.639 the same issue, percent of mothers with appropriate 00:34:22.639 --> 00:34:25.559 knowledge on good maternal nutrition before, 00:34:25.719 --> 00:34:30.309 during, and after pregnancy. Again, The indicator 00:34:30.309 --> 00:34:34.010 is excellent, but the percent is measured annually, 00:34:34.329 --> 00:34:38.750 just annually. That suggests for this indicator 00:34:38.750 --> 00:34:41.909 that throughout that year, before they measure 00:34:41.909 --> 00:34:46.469 it, the group of mothers that aren't exposed 00:34:46.469 --> 00:34:50.449 to the project, they also could figure out the 00:34:50.449 --> 00:34:52.590 knowledge on good maternal nutrition before, 00:34:52.710 --> 00:34:55.110 during, and after pregnancy. So they have to 00:34:55.110 --> 00:34:58.840 put in the PMF how their project... They just 00:34:58.840 --> 00:35:01.300 got to mention control group, comparison group, 00:35:01.380 --> 00:35:06.340 or statistical significance analysis on the percent 00:35:06.340 --> 00:35:09.719 over time, right? They just got to put that in 00:35:09.719 --> 00:35:12.340 there. As far as I know, it's not in the PMF. 00:35:12.420 --> 00:35:14.639 So I'm going to assume they're not looking at 00:35:14.639 --> 00:35:17.039 it and they need to look at it. They want to 00:35:17.039 --> 00:35:19.739 make the claim that their project increased the 00:35:19.739 --> 00:35:22.780 percent statistically significantly with impact. 00:35:24.099 --> 00:35:27.130 Next outcome. improved effectiveness of local 00:35:27.130 --> 00:35:31.550 stakeholders in target countries and Canada on 00:35:31.550 --> 00:35:34.110 gender equitable local and international nutrition 00:35:34.110 --> 00:35:37.190 specific and sexual reproductive health rights 00:35:37.190 --> 00:35:42.070 activities advocacy and policy dialogue so the 00:35:42.070 --> 00:35:45.190 outcome indicators there are Canadians reached 00:35:45.190 --> 00:35:48.889 through the project on nutrition health sexual 00:35:48.889 --> 00:35:51.710 reproductive health and gender equality issues 00:35:51.710 --> 00:35:56.710 so it's a little vague How do they define improved 00:35:56.710 --> 00:36:01.469 effectiveness? They don't. Effective in doing 00:36:01.469 --> 00:36:04.090 what? Increase in knowledge? But its outcome 00:36:04.090 --> 00:36:08.269 indicator is clear, specific to increasing reach. 00:36:09.130 --> 00:36:12.230 Canadians reached. The measure of the number 00:36:12.230 --> 00:36:14.349 reached is clear, based on counting the number 00:36:14.349 --> 00:36:17.110 online, who have read, watched, commented, shared, 00:36:17.250 --> 00:36:19.619 donated through their website. awareness campaign 00:36:19.619 --> 00:36:22.659 but there's no data on whether their quote awareness 00:36:22.659 --> 00:36:26.179 campaign in this project was responsible for 00:36:26.179 --> 00:36:29.119 a significant increase in this number so i gave 00:36:29.119 --> 00:36:33.119 it a negative there not very good in showing 00:36:33.119 --> 00:36:37.340 their project increased reach more than if they 00:36:37.340 --> 00:36:41.400 didn't do it same problem with the second indicator 00:36:41.400 --> 00:36:44.519 extent of engagement of local health facilities 00:36:45.340 --> 00:36:48.360 on gender equitable nutrition, health, water, 00:36:48.460 --> 00:36:52.760 sanitation, health, and sexual reproductive health 00:36:52.760 --> 00:36:55.960 rights issues. Data source and data analysis 00:36:55.960 --> 00:36:59.519 is a focus group discussion, which is a poor 00:36:59.519 --> 00:37:02.059 way to measure whether levels of engagement of 00:37:02.059 --> 00:37:06.340 stakeholders with health facilities, specific 00:37:06.340 --> 00:37:09.360 to if services have been delivered equitably. 00:37:09.679 --> 00:37:13.380 They need to better define and measure improved 00:37:13.380 --> 00:37:16.980 effectiveness. specific to levels of engagement 00:37:16.980 --> 00:37:22.099 and the last one is my favorite degree of effectiveness 00:37:22.099 --> 00:37:24.920 of local women and adolescent girls rights groups 00:37:24.920 --> 00:37:29.699 and associations feel using a liquored five point 00:37:29.699 --> 00:37:32.900 scale in ensuring gender equitable nutrition 00:37:32.900 --> 00:37:35.900 health and sexual reproductive health services 00:37:35.900 --> 00:37:39.239 do not ask them how they feel about their effectiveness 00:37:39.239 --> 00:37:42.989 in ensuring that services are delivered equitably. 00:37:43.110 --> 00:37:46.030 Measure their ability to advocate for services 00:37:46.030 --> 00:37:51.429 to be delivered equitably. For example, evaluate 00:37:51.429 --> 00:37:54.329 written advocacy products they have written or 00:37:54.329 --> 00:37:56.710 their presentation of those advocacy products. 00:37:57.030 --> 00:38:00.210 That would be a better measure of how effective 00:38:00.210 --> 00:38:03.949 they are in ensuring gender equitable services 00:38:03.949 --> 00:38:07.909 are delivered, right? So on that one, those three 00:38:07.909 --> 00:38:12.309 indicators get a red light, unfortunately. So 00:38:12.309 --> 00:38:15.570 if we go to another one, strengthened delivery 00:38:15.570 --> 00:38:18.710 of gender equitable and responsive nutrition 00:38:18.710 --> 00:38:21.269 health and sexual reproductive health services 00:38:21.269 --> 00:38:23.829 for the poorest and the most marginalized women, 00:38:23.949 --> 00:38:26.650 adolescent girls and children. The indicator, 00:38:26.769 --> 00:38:30.610 percent of adolescent girls who report that they 00:38:30.610 --> 00:38:34.210 were offered services without judgment by providers. 00:38:34.889 --> 00:38:38.070 Excellent, valid indicator. And it's using a 00:38:38.070 --> 00:38:41.860 household survey. That's awesome. But again, 00:38:42.280 --> 00:38:45.500 they have to show, using that household survey, 00:38:45.679 --> 00:38:51.159 if they used a right -tailed, one -tailed statistical 00:38:51.159 --> 00:38:54.679 significance test to show that their $40 million 00:38:54.679 --> 00:38:58.739 project increased that percent statistically 00:38:58.739 --> 00:39:03.480 and significantly over time compared to where 00:39:03.480 --> 00:39:06.320 the project was not operating or compared to 00:39:06.320 --> 00:39:09.750 the same target group over time. Excellent indicator, 00:39:09.969 --> 00:39:12.650 but there's no indication in the PMF that they're 00:39:12.650 --> 00:39:16.090 doing any statistical analysis to show that it's 00:39:16.090 --> 00:39:19.170 had a statistically significant difference in 00:39:19.170 --> 00:39:21.889 the right direction, percent increase over time. 00:39:23.150 --> 00:39:25.610 But it's great. It's expensive to do a household 00:39:25.610 --> 00:39:28.690 survey. So you would think they're doing that 00:39:28.690 --> 00:39:31.550 analysis, and maybe they are. It's just not in 00:39:31.550 --> 00:39:33.489 the performance measurement framework, but it 00:39:33.489 --> 00:39:36.170 should be, especially when the performance measurement 00:39:36.170 --> 00:39:40.550 framework is trying to show If the outcomes have 00:39:40.550 --> 00:39:43.630 been effective and have been achieved. The second 00:39:43.630 --> 00:39:46.289 indicator, percent of health facilities promoting 00:39:46.289 --> 00:39:49.869 gender equitable and responsive nutrition as 00:39:49.869 --> 00:39:51.750 measured by a modified tool in the baseline. 00:39:53.469 --> 00:39:55.909 Now, again, without the performance measurement 00:39:55.909 --> 00:39:57.869 framework, you'd think, ah, that's pretty good. 00:39:58.090 --> 00:40:01.070 But then if you look at the PMF and you look 00:40:01.070 --> 00:40:04.409 at the data source, it's quote, the head of the 00:40:04.409 --> 00:40:06.960 health facility. So obviously the head's going 00:40:06.960 --> 00:40:11.760 to say, yeah, it's great. We did it. We promoted 00:40:11.760 --> 00:40:14.760 gender equitable services. We did it. Yes or 00:40:14.760 --> 00:40:17.880 no? Of course he or she's going to say yes. Conflict 00:40:17.880 --> 00:40:20.440 of interest. They should have an objective measure 00:40:20.440 --> 00:40:23.219 of determining if the health facility is actually 00:40:23.219 --> 00:40:25.980 promoting gender equitable services by looking 00:40:25.980 --> 00:40:29.119 for, quote, gender equitable products at the 00:40:29.119 --> 00:40:33.119 facility. Or they could take out the syndicator. 00:40:33.440 --> 00:40:36.179 Since the survey of the patients visiting the 00:40:36.179 --> 00:40:39.360 facility is covered in the household facility. 00:40:42.420 --> 00:40:45.920 Household facility survey. If they, the girls, 00:40:46.139 --> 00:40:48.820 the name of the health facility they visited 00:40:48.820 --> 00:40:51.420 to get health services. They could actually in 00:40:51.420 --> 00:40:54.300 the survey of the girls, in the household survey, 00:40:54.400 --> 00:40:56.800 ask them which health facility did you go to? 00:40:56.920 --> 00:40:59.420 Could have a whole list in the region and they 00:40:59.420 --> 00:41:02.260 could just tick it off. And that way they could 00:41:02.260 --> 00:41:04.960 show. If there's been an increase in the health 00:41:04.960 --> 00:41:07.599 facilities, that would be a better measure rather 00:41:07.599 --> 00:41:09.719 than just asking the head of the health facility. 00:41:10.320 --> 00:41:12.960 The last one indicator changes in perception 00:41:12.960 --> 00:41:17.340 of female and male clients on the gender responsiveness 00:41:17.340 --> 00:41:20.460 of health facilities in providing nutrition, 00:41:20.639 --> 00:41:23.460 health, wash, and sexual reproductive health 00:41:23.460 --> 00:41:27.860 rights services. So again, the problem here is 00:41:27.860 --> 00:41:33.710 if you look at the data source, It's a focus 00:41:33.710 --> 00:41:36.550 group discussion. It's a poor measure of tracking 00:41:36.550 --> 00:41:39.849 changes in perception. Ask any social psychologist. 00:41:41.289 --> 00:41:46.269 You can easily do that quantitatively. Again, 00:41:46.329 --> 00:41:48.969 even changes in perception in the desired direction 00:41:48.969 --> 00:41:52.710 could happen anyway outside of the project. So 00:41:52.710 --> 00:41:54.789 what they need to do there is do a quantitative 00:41:54.789 --> 00:41:59.409 and ideally do some stat significance to see 00:41:59.409 --> 00:42:02.570 if the percent has gone up. in perception in 00:42:02.570 --> 00:42:06.210 the right direction, right? So how are we doing 00:42:06.210 --> 00:42:09.710 here? Oh, we're almost done. We're getting to 00:42:09.710 --> 00:42:12.590 the global indicators now. So here's another 00:42:12.590 --> 00:42:15.670 outcome. Improved adoption of gender equitable 00:42:15.670 --> 00:42:19.230 practices in nutrition, health, and sexual reproductive 00:42:19.230 --> 00:42:22.610 rights at individual, household, and community 00:42:22.610 --> 00:42:25.769 levels. And if you look at the indicators here, 00:42:26.280 --> 00:42:29.119 These are definitely globally recognized. So 00:42:29.119 --> 00:42:31.699 the challenge is connecting the $40 million project 00:42:31.699 --> 00:42:34.900 with whether or not that $40 million project 00:42:34.900 --> 00:42:37.780 achieved these three indicators. The first one, 00:42:38.039 --> 00:42:41.059 percent of mothers of children who attended at 00:42:41.059 --> 00:42:44.219 least four antenatal visits during their last 00:42:44.219 --> 00:42:48.519 pregnancy. Project outputs clearly deliver services. 00:42:50.179 --> 00:42:53.440 The only challenge is to show if the change in 00:42:53.440 --> 00:42:56.559 those percents on the indicators is due to the 00:42:56.559 --> 00:42:59.559 project to show the significant impact of the 00:42:59.559 --> 00:43:02.780 project. Therefore, the PMF needs to show in 00:43:02.780 --> 00:43:05.699 the data collection method a one -tailed, right 00:43:05.699 --> 00:43:08.400 -tailed hypothesis where the null hypothesis 00:43:08.400 --> 00:43:11.519 is that the observed percent increase was not 00:43:11.519 --> 00:43:14.360 statistically significant and the percent increase 00:43:14.360 --> 00:43:17.480 could have happened anyway in the absence of 00:43:17.480 --> 00:43:20.519 the $41 million project or in the presence of 00:43:20.519 --> 00:43:27.050 the $41 million project. But the percent was 00:43:27.050 --> 00:43:29.769 negligible, small, not worth the $41 million. 00:43:30.469 --> 00:43:33.710 Same goes for the next two indicators. Percent 00:43:33.710 --> 00:43:36.750 of women married who are currently using or whose 00:43:36.750 --> 00:43:41.130 sexual partner is using at least one modern contraceptive 00:43:41.130 --> 00:43:44.690 method. And the third indicator, percent of children 00:43:44.690 --> 00:43:48.909 who receive minimum dietary diversity and minimum 00:43:48.909 --> 00:43:52.940 meal frequency disaggregated. Right? Good indicators, 00:43:53.079 --> 00:43:56.380 global indicators. Just got to make sure that 00:43:56.380 --> 00:43:59.300 the $41 million that Canadian taxpayers spent 00:43:59.300 --> 00:44:03.559 resulted in a statistically significant different 00:44:03.559 --> 00:44:07.260 increase due to the project, right? And finally, 00:44:07.280 --> 00:44:10.940 the ultimate outcome, same issue. This is a global 00:44:10.940 --> 00:44:14.559 indicator. Improved nutrition, nutrition -related 00:44:14.559 --> 00:44:18.260 rights, and gender equality for the poorest and 00:44:18.260 --> 00:44:21.280 most marginalized, especially women. adolescent 00:44:21.280 --> 00:44:24.659 girls and children under five years of age in 00:44:24.659 --> 00:44:29.000 Bangladesh, Kenya, Somalia, and Tanzania. Right? 00:44:29.079 --> 00:44:32.900 Three indicators? Very popular, common indicators. 00:44:33.639 --> 00:44:36.860 Global indicators. Percent of households achieved 00:44:36.860 --> 00:44:40.579 gender equality. Got some funky measure of gender 00:44:40.579 --> 00:44:43.699 equality. Nothing wrong with the indicator, but 00:44:43.699 --> 00:44:48.019 we've got to show the percent. has gone up statistically 00:44:48.019 --> 00:44:50.880 and significantly right in the pmf same with 00:44:50.880 --> 00:44:54.420 anemia prevalence among adolescent girls and 00:44:54.420 --> 00:44:57.179 the third indicator percent of stunted children 00:44:57.179 --> 00:45:01.099 disaggregated by sex and country so it raises 00:45:01.099 --> 00:45:05.119 the overall issue of okay let's hope that they 00:45:05.119 --> 00:45:08.780 can show that the 41 million dollars was statistically 00:45:08.780 --> 00:45:12.599 significantly producing these increases in these 00:45:12.599 --> 00:45:15.900 percents in the right direction So the $41 million 00:45:15.900 --> 00:45:20.480 really did have impact, right? So that's what 00:45:20.480 --> 00:45:22.619 we want to look at. And even when the indicators 00:45:22.619 --> 00:45:26.460 are global and therefore we know their proper 00:45:26.460 --> 00:45:30.219 valid measures of the outcomes where you just 00:45:30.219 --> 00:45:33.219 copy and paste, it's all the other outcome indicators 00:45:33.219 --> 00:45:37.599 and the analysis to make sure that it's statistically 00:45:37.599 --> 00:45:41.440 significant. That's assuming that the indicator, 00:45:41.579 --> 00:45:43.949 and as I've mentioned here, Many of the indicators 00:45:43.949 --> 00:45:46.909 are not valid measures of the outcome. So I've 00:45:46.909 --> 00:45:49.590 led to the conclusion, and I think I can add 00:45:49.590 --> 00:45:52.469 three more now. I can cheat a bit. I can say 00:45:52.469 --> 00:45:57.389 that out of the 38 indicators, 10 of them are 00:45:57.389 --> 00:46:00.489 really good, valid outcome indicators, not seven, 00:46:00.550 --> 00:46:03.449 because I added those three. And the remaining, 00:46:03.670 --> 00:46:08.050 what is that, 28? Not good for the reasons of 00:46:08.050 --> 00:46:10.889 statistical significance or for other reasons 00:46:10.889 --> 00:46:15.059 that I stated previously. So now this will go 00:46:15.059 --> 00:46:18.260 off, this episode recording with the PMF plus 00:46:18.260 --> 00:46:22.480 the PMF critique that summarizes all of the indicators, 00:46:22.679 --> 00:46:25.119 whether they're valid or not, will go off to 00:46:25.119 --> 00:46:28.199 the Secretary of State for International Development, 00:46:28.280 --> 00:46:31.960 along with the shadow critics for the conservative 00:46:31.960 --> 00:46:37.619 and NDP parties and the Bloc Québécois. And with 00:46:37.619 --> 00:46:39.820 the recommendation that these improvements be 00:46:39.820 --> 00:46:42.369 made. It's a judgment call as to whether or not 00:46:42.369 --> 00:46:46.570 the 41 million should be stopped. We don't know, 00:46:46.650 --> 00:46:52.150 but I'm not going to go that far. And also, I'm 00:46:52.150 --> 00:46:55.469 trying to think now. Yes, we're going to invite 00:46:55.469 --> 00:46:59.110 World Vision Canada to come on to the podcast 00:46:59.110 --> 00:47:03.090 if they want and respond to my critique of their 00:47:03.090 --> 00:47:06.690 PMF. Thank you for listening. And we'll be back 00:47:06.690 --> 00:47:09.550 soon as I have another performance measurement 00:47:09.550 --> 00:47:12.590 framework. from another Canadian organization. 00:47:12.969 --> 00:47:16.829 So my next episode will follow shortly. Bye for 00:47:16.829 --> 00:47:16.969 now.