Welcome to the Improving Development Evaluation
Podcast. I'm your host David Wand and welcome
to this episode where we feature World Vision
Canada. You can learn more about World Vision
Canada at their website worldvisioncanada .ca.
This project we're going to talk about cost the
Canadian taxpayer 41 million dollars and it is
delivering services and products in Kenya, Somalia,
Bangladesh, Tanzania, and Cambodia, as well as
Canada as a country. And it is delivering a variety
of services and products. The title of the project
is Realizing Gender Equality, Attitudinal Change,
and Transformative Systems in Nutrition. But
before I get into the details of the project,
I want to give you a little bit of background
as to why it's taken me so long to do another
episode. My episodes depend on receiving performance
measurement frameworks, PMFs, that list the outcomes
that they are claiming they're going to achieve
with the project, as well as, more importantly,
the outcome indicators, because we need to know
if they're valid or not. As I've said before
on this podcast, before the money can be dispersed,
the $41 million, they have to produce this performance
measurement framework to the government of Canada,
and the government of Canada has to approve it.
If you go to the Project Browser website, which
I'll provide in the episode notes where the project
is, you'll notice they've been receiving this
$41 million since May of 2023. So the PMF has
already been done since 2023. I requested this
PMF in October of 2024, but now it's January
of 2026 when I finally received the PMF, even
though I'd been complaining for several months,
even sent a complaint to the Independent Information
Commissioner. It wasn't until I sent an email
to my local member of parliament who happens
to be a liberal in the government, happens to
be the minister of immigration. And I said to
her, look, I'm not getting this PMF. Somebody's
stalling. Come on. It's been over almost two
years since I've received this PMF. What's going
on? And I kind of jokingly said, I'm not going
to vote for you the next time. Within a week,
I received an email from, quote, her team. saying
her team would get back to me. And they did.
Next, about a week later, in January of 2026,
after initially requesting the PMF in October
of 2024, I finally received it. So that's some
background for you about the politics of information.
So I'll shut up now and get to the details. To
give you a bit of background about the target
groups and the products and services that this
project spent $41 million delivering in these
various developing countries. One of the target
groups was Canada and the developing countries
I mentioned, their populations. And one of the
products they produced was art to raise awareness
on gender equality for those populations. Another
target group was World Vision staff themselves.
They were trained on how to develop message products
to raise awareness on gender equality. Probably
the World Vision offices in the developing countries,
the staff that work in the developing countries.
Another product that the World Vision staff produced
was case studies that were completed and disseminated
to... gender transforming nutrition, including
budgeting and monitoring, and also women's rights
groups in these developing countries, another
target group. They were trained on how to advocate
for nutrition and sexual reproductive health
services. Another target group were health officials
and health facility managers in these developing
countries. They were trained to supervise health
workers on proper gender responsive services,
budgeting and monitoring. Health systems was
another target group. I don't know what human
beings are there, but reports were produced on
gaps in health services. Ministry of Health staff
were also a target group in these developing
countries that were trained on how to plan for
anemia prevention and treatment and malnutrition
and family planning. Teachers were another target
group in these developing countries, training
on adolescent health. Health facilities also
received equipment and supplies for nutrition
and to deliver sexual reproductive health services,
as well as equipment and supplies to deliver
water and sanitation. Farmers were another target
group in these developing countries. They were
given seeds as part of this $41 million to grow
food. Faith leaders in these developing countries
were also another target group that were trained
on how to reduce barriers for women to access
nutrition and sexual reproductive health services.
Men and women were another target group that
were also trained on how to reduce barriers for
women to access nutrition and sexual health services.
Boys and girls were another. Target group. They
were trained on gender equality. And finally,
there were peer networks in these developing
countries. They were trained on how to deliver
food and hygiene demonstrations, as well as how
to develop messages specific to sexual reproductive
health services. So that gives you an idea of
what they spent the $41 million on. Now, lots
of training, seeds, all sorts of things, equipment,
supplies, as I've just described. Now the question
is, if we look at their performance measurement
framework, they have some outcomes. In fact,
they have outcome statements, 13 outcomes that
they expect to achieve after spending that $41
million. And out of those 13 outcomes they expect
to achieve, In the performance measurement framework,
they developed 38 outcome indicators. So we're
going to go through not all of them, but as many
of them as we can in the next 30 or 40 minutes.
And then I'll give you an idea of what I came
up to as to whether or not those outcome indicators
are garbage and they can't support any of the
claims that they've achieved those outcomes because
the indicators don't even measure what they're
trying to claim they're achieving. maybe there's
some good outcome indicators in there. It's a
bit of both, as you'll find out. But before I
do that, and you live in the Global South, or
you hopefully are planning to visit the Global
South to hopefully properly evaluate this project,
or for that matter, any project, you may want
to check out some sunglasses at akila .la, A
-K -I -L -A dot L -A. In the episode notes, I'll
give you that website. If you buy them, prescription
lenses included in those sunglasses, all made
from 50 % recycled material, you will get a 10
% discount if you use my code, WANDCOOLT -T -SHIRTS,
or you can just click on the link that I have
in the episode notes, and you'll get a 10 % discount.
And when you do that, I get a 15 % commission.
when you purchase any product using my promo
code. So you have your sunglasses to protect
you from the sun and the global south, but you
need the right clothes to be comfortable in that
hot and humid climate. So what do you want to
do there? You want to check out hypernaturalstyle
.com. Some men's polos, tees, and button -ups.
You get 20 % off any product when you use my
promo code WANDCOOLT -SHIRTS. And I get a 20
% commission of the total purchase you make when
you use my promo code WANDCOOLT -SHIRTS. So what
is so cool about these t -shirts? Well, they
have a patent -pending HyperCool Jade technology.
that when it's hot, it lowers your body temperature
by three to five degrees. They are also breathable
thanks to Supina cotton, and they are sold in
premium retailers like Nordstrom, and were voted
the best men's polo shirts in 2024 in Men's Journal,
Forbes, and Esquire. So let's get on with the
outcomes and the outcome indicators. So let's
start with this first. Outcome statement increased
engagement and awareness of the Canadian public
on the impact of official development assistance
in the context of gender equitable and responsive
nutrition, health and sexual reproductive health
programming. They have two indicators for there.
The first one is number of Canadians aged 18
to 65 years of age who are engaging with content
related. to nutrition, health, sexual reproductive
health, gender issues in international development
contexts. So what's interesting here is that
sounds like a pretty good indicator, but then
you have to look at the PMF to see what exactly
they do to measure it. Well, bad news, dudes.
They do not use a survey of the Canadian public.
So they can't even claim. that they're achieving
their target group, which is raised awareness
of the Canadian public. All they talk about is
the number of likes and comments online and the
number of in -person conversations at events.
And they do it at baseline, midline, headline,
and annually. And it's going to show an increase
in engagement and awareness. But they do it from
the project database. But that's not a sample
survey of the, quote, Canadian public. So they
can't make the claim. that the Canadian public
has experienced an increase in awareness specific
to gender equity nutrition, sexual reproductive
health, based on Canadian foreign aid has increased.
They can't. They've got the wrong target group.
They can claim that the project people that engage
with the project, but they haven't done a sample
of the general Canadian public population. So
they've got the wrong outcome. So they can't
make that claim. They could change it. They could
reduce it down and just say, You know, if people
increase over time their engagement with us,
whoever shows up to these events or send comments
in online, fine, we could show there's been an
increase in engagement. But they can't make the
claim that the Canadian public at large has increased
their awareness. Right? And it's the same for
the specific to the awareness of official development
assistance impact has increased. They've got
the wrong target group. Because they haven't
done a sample survey, right? The good news is,
as you'll find out later, some of the other indicators
they've got, they do a very good job because
they actually do do a household survey. So that's
an example of two indicators that really are
not doing it properly. So they're not good. So
if we keep going, we can look at another outcome.
expected to achieve increased utilization of
gender transformative framework for nutrition
by key national and subnational stakeholders
for mainstreaming of gender equality into the
planning budgeting monitoring and learning processes
of multi -sectoral nutrition programs and policies
they've got a few indicators for that you'll
see well you can't see you can only hear First
one is percent of national, subnational government,
civil society, NGO stakeholders endorsing GTFN,
gender transformative framework network, mainstreaming
within multisectoral programs and policies. So
that's one indicator they have. Another indicator
they have is this framework, gender transformative
framework nutrition used at least in one subnational
level government sector. in the project site
for mainstreaming of gender equality into the
planning, budgeting, monitoring, and learning
processes of multisectoral nutrition program.
But if you can, if you look at their indicators,
the first one, they talk about endorsement, endorsing
mainstreaming of GTFN. That's not the same as
increased utilization. They could say, oh yeah,
we've endorsed it, but you've got to look to
see if they've actually used it. So it's not
a valid indicator. It doesn't validly measure
whether actual utilization is increased. If they
want to go back and change the outcome statement
to endorsed, fine, but they didn't. So they can't
make that claim. The other problem is with the
second indicator used in at least one subnational
level government sector. If you look at the data
source, this is where the PMF is important because
you wouldn't know this unless you got the PMF.
They look at the project database regularly to
see if it's been used, right? Okay. You have
to just trust that whoever's entered the information
in the database, they've just said, yeah, they've
used it. But it's not really a good measure.
I think it would be better that they actually
go in to the government sector to see if they've
actually used it. You know, developing some criteria.
Did they use it or not? They shouldn't be going
into a secondary data source as the project database.
So I don't trust it. It's a bias, right? You
should actually just show up a third party evaluator
and pick a government sector, go in and say,
did you use the GTFN? Did you use it? And they
could check by seeing. actual documents rather
than just going into the database. So that's
another two indicators that are not well designed.
They can't support the claim that utilization
has been increased. Another outcome, increased
capacity of local community structures such as
citizen voice and action groups, community health
committees to engage in evidence -based advocacy
and policy dialogue at national and sub -national
levels on gender equitable rights. to nutrition,
health, and sexual reproductive health rights.
So they've got three indicators here. First one
is percent of community -based structures that
are monitoring gender -responsive and adolescent
-friendly service delivery at health facilities
disaggregated by country. Again, if you read
that, monitoring, they're equating their ability
to monitor with their increased capacity to advocate.
That's wrong. They're not the same thing. Now,
monitoring is, you could argue, is an indirect
measure that they're actually capable of advocating.
You could argue that. So I've given it a green,
meaning I say it's okay. They do an assessment
of the health staff and look at the project database.
We can assume that this assessment looks at the
ability to monitor. So it's an indirect measure.
I've given it a green light. I would say that's
a good indicator for that. outcome of increased
capacity. The second one indicator is percent
of trained members who understand basic health
rights and how these rights are articulated under
local law disaggregated by country. And if you
take a look there, they actually have an objective
testing measure of this understanding on basic
health rights and how to articulate these rights.
So I checked for the existence of an actual test
there is a test so that's excellent this is a
good this is a green light good indicator again
the only issue there is and i again i haven't
said it's bad enough to give it a red light is
would the increase in that percentage who passed
the test be significantly greater than a group
of other individuals who didn't participate in
the project meaning what i call impact Some people
disagree with that definition, but that's the
one I use for the podcast. Not only has it been
achieved, effective, but it's been achieved so
greatly that the increase on the indicator is
so great, it is statistically significant, and
someone can argue it's not due to chance. But
we're not looking at that right now. But that's
a good indicator. So the last indicator on that
outcome is extent of engagement of women and
men. community leaders in targeted villages with
local government on nutrition, health, sexual
reproductive health rights, and gender issues.
Now, for that one, if you look again at the PMF,
if you don't have the PMF, you can't find this.
But then you look at the data source, and this
is where it gets, tell me about your feelings.
Engagement is measured with the community leaders
themselves. So there's a bias. And using focus
group discussions. So it's going to lead to self
-reporting bias. So for this indicator, I gave
them a red light. Not a good measure. Focus groups.
They need to get a better measure of engagement
externally to show that the level of engagement
is actually quantitatively increased. So that's
not good. So not bad. Three indicators. Two of
them really good. One of them bad. So if we look
at another outcome, improved. capacity of sub
-national health management teams for effective
gender equitable and responsive nutrition health
and sexual reproductive health systems governance
including planning budgeting reporting so improved
capacity again they have three indicators the
first one is called percent of health facility
scoring greater than 80 on their health facility
planning assessment by the team This indicator
is good. We can only assume that the test validly
measures the technical capacity of the team to
do, quote, planning, budgeting, and reporting.
But again, I've given it a green light, but also
the impact issue is, could the team, without
this $40 million, been able to teach themselves
how to do this planning, budgeting, and reporting?
So that should be put in the PMF. is how are
they going to show that that increase in percent
is statistically significant, right? Now, it
could be as simple as just measuring repeatedly
over time, showing the percent going up, and
that's good enough. You don't need a control
group, but there should be some sort of quantitative
analysis to show that it's been going up, and
not only going up, scoring greater than 80. but
statistically significantly going up. So that's
another way to look at it. But I gave them a
green light for that indicator. The next two
indicators, I gave red lights. So the next indicator
is percent of health workers who report they
trust the leadership of the management teams
in ensuring effective gender equitable services.
Some people will say, oh, we're doing that to
see if we can check against. improved capacity
of those health management teams i gave it a
red light because it's if you look at the data
source it's the health team members themselves
so there's self -reporting bias there and also
measuring trust has nothing to do with increased
technical capacity to deliver responsive budgeting
etc right so that that's not a good indicator
for those two reasons self -reporting bias because
they're asking themselves If they trust themselves,
you have to look at the data source, right? When
they say health workers, according to the data
source, it's not health workers. It's the health
team members themselves. But even if I'm wrong
there, I wouldn't use it. I'd throw it out. The
third indicator for this outcome was percent
of team members who have access to planning,
budgeting, and governance data. And here we go
again. who report by themselves using the data
to make decisions about the provision of gender
equitable services. So instead, we've got self
-reporting bias again. What I would do instead,
this is not a good indicator, is I would actually
sample the reports to see if the data was actually
used to, quote, make decisions of gender equitable
services, right? I'd do that by an external evaluator
instead. Self -reporting bias. So out of those
three indicators, only one was good. The other
two were bad. Another outcome, increased health,
infrastructure, information technology, and capacity
for prevention and management of acute malnutrition
in most marginalized communities. The problem
here is three indicators. Number of school -going
adolescent girls who received the recommended
scheme of weekly iron folic acid supplementation.
Second indicator, percent of children admitted
treated for moderate or severe acute malnutrition.
And third indicator was percent of health facilities
in remote catchment areas rehabilitated received
support for HMIS strengthening to improve gender
equitable nutrition. Now, they're claiming here
that if you just deliver those services to the
girls and the children, that that in turn can
be equated with increased capacity for prevention
and management. You could argue that. I'm changing
my mind here. I'm thinking these are just outputs.
But what they're trying to tell you here is that
if we deliver that folic acid and we deliver
that treatment to the children for malnutrition,
and if we deliver to those health facilities
support, whatever that means, then we could argue...
that that in turn is the same as increased capacity
to prevent and manage. So I'm going to change
my mind here and give this a green light instead
of a red light, because it is a good way of saying
we're delivering these services and those services,
just getting them delivered. It is charity, but
that's okay. That's another issue. So I'll leave
it at that as a valid measure of achieving that
outcome. Another one is increased knowledge and
skills of health care providers on gender equitable
and responsive nutrition, health and sexual reproductive
health services. Now, again, this is the FGD
problem. The first they've got, how many indicators
have they got for this outcome? They've got three.
So the first one, extent of change in knowledge
and skills among health care providers after
receiving training. Fantastic. Problem is, go
to the data source. They're not measuring them
objectively on their knowledge and skill levels.
They're holding a focus group discussion. Instead,
they're holding key informant interviews. Instead,
no need to do it. So it's not important. Only
use focus group discussions, key informant interviews,
if your quantitative indicator is not achieving
your outcome. Then you can go back to the group
and say, why? What's going on here? What's the
problem? So if you look at the second indicator,
it's excellent. Percent of health workers who
have passed knowledge competency tests following
training. It's great. It's a pre and post test.
So that's very good. Love it. Fantastic. Third
one indicator for increased knowledge and skills
is percent of households received at least one
visit. from a community health worker in the
last month. Well, they could show up and, like
me, when I was in Sierra Leone, have a jug of
palm wine and have no knowledge and no skills
and get a free meal. Cassava leaf and rice, my
favorite dish. Right? So this indicator, in my
opinion, is useless. Third indicator, percent
of households receiving a visit has nothing to
do with increased knowledge of health worker
you just trained. This is an output. So we don't
know. That health care provider actually has
increased levels of knowledge and skill. Unless
one of the measures of knowledge and skill is
how to find the health center, which on a motorcycle
in Sierra Leone can be a bit of a challenge.
Took me about six months to figure that out.
But no, in this case, it's not good. They should
just use that previous indicator, pre and post
test, good enough. Another outcome, increased
gender equitable access to Nutrition -sensitive
interventions in wash, water, sanitation, and
agriculture, including production and consumption
of biofortified crops and nutrient -dense local
vegetable and fruits. This one's interesting.
The indicator for this in terms of increased
access is the percent of mothers planted biofortified
crops after receiving planting material directly.
Now, you'd assume... They're going to have access
to this food after they plant the seeds, right?
But they ask the mothers. Don't ask the mothers.
Just say to the mothers, excuse me, show me your
plot. So don't survey the mothers. Instead, they
should survey the crops to see if they were planted
and see if the crops survived, right? So that's
a no -go. Now, they do have project progress
reports. They might show this, but we do not
know. All we can see is the indicator is useless.
Same for the second one. Percent of mothers who
planted local vegetables after receiving the
seeds. Again, they surveyed the mothers. They
shouldn't. They should just survey the vegetables
to see if one, they were planted and two, to
see if the vegetables survived, right? And the
third one, same problem, planted fruit trees.
So I'm assuming from the data source, they're
asking the mothers instead of going. Now, maybe
it's in the survey manual. When you ask the mother,
she says yes. Then go outside and check the plot
and to see if the fruit trees were planted and
survived. So right from that reading, they've
got the wrong data source. Maybe World Vision,
if they have the courage to show up, they'll
say, David, we can tell you by fact that in the
manual, when they did the survey, they went outside
and checked. But I don't know. It's funny because
the last indicator, there's another one. percent
of households with hygienic toilets after receiving
information on appropriate water and sanitation
and hygiene. Wash. We can assume that the household
survey includes a question. Is there a hygienic
toilet? So that one, they can actually go and
see it. So they should do the same for the vegetables,
for the fruits, and for the fortified crops,
right? So out of the four indicators, the hygienic
toilet is a good one. The other three are not.
So let's see, what are we running out of time
here? How much time have we got here? Well, we're
at 30 minutes. We can keep going for a bit. I
think I'll stop around 40. Here's another one.
Increase knowledge of key community -level influencers
to challenge discriminatory social norms that
limit women and adolescent girls from accessing
gender -equitable, nutrition -specific health,
WASH, and sexual reproductive health services.
And they've got, I think they got three indicators
for this outcome. And the first one, percent
of mothers of children under five who know at
least three modern family planning methods after
receiving the information from religious leaders
and lead fathers. It's a good indicator. It checks
on whether that knowledge transfer from the influencer
to the mother actually happened. Only issue again
is, as I've mentioned before, is could that have
happened without spending $41 million? Could
they have figured that out, modern family planning
methods, without spending $41 million? By talking
to other mothers, going down the street, whatever.
So what they should insert in the PMF is, over
time, was that percent that increased, was it
statistically significant, one -tailed, right
-tailed, P less than .05, right? Or they could
use a comparison group, but they don't have to,
of a group of mothers. But they should be putting
that in the PMF to show, yeah, the percent not
only went up, but it went up statistically significantly.
Therefore, using my definition, it has, quote,
impact. It's not just a difference, effective,
but the difference was so big, they could say,
wow, it's an impact. I know that DAC uses impact
broader. They say unintended and intended effects
above and beyond the project in other areas.
That's not my definition. Second indicator, changes
in perception and opinions among key community
influencers to challenge discriminatory social
norms that limit women and adolescent girls from
accessing gender -equitable, nutrition -specific
health, wash, and sexual reproductive health
services. The problem there is the mother's knowledge
is first dependent upon the influencer that gives
it to them, also having that knowledge. Where
is that test of the influencer's knowledge, like
the knowledge test they gave to the health worker
previously? This is missing and should be done.
Also, qualitative focus group discussions with
these influencers is not a valid measure of perception
change. Need to include in the test on knowledge.
that perception change. You can do that quantitatively.
So it's not a good indicator. The last one, percent
of mothers of children under five who know at
least three benefits of exclusive breastfeeding
after receiving the information from religious
leaders and lead fathers. Again, excellent indicator.
The only issue again is, is it statistically
significant compared to if we didn't spend that
$41 million at all? Could that percent go up
anyways? So we've got to make sure if we're going
to blow $41 million that our intervention is
increasing that percent way, way, way, way better
of those mothers that know about exclusive breastfeeding
and how it's beneficial. That percent of mothers
goes way up higher because of the project rather
than if we didn't have the project at all. So
we'll move on to... A few more indicators, improved
knowledge and self -efficacy of poor and most
marginalized women and adolescent girls to negotiate
access to and control over nutrition -specific
and sensitive health and sexual reproductive
health rights services. We got three indicators
there. But again, the indicator, first one, percent
of women who decided to use family planning alone
or jointly with their partners. Sounds like a
pretty good indicator. But what they do, if you
look at the PMF, is they indicate they've got
target areas. So that means that there's areas
where they're not operating to increase this
knowledge and self -efficacy to negotiate access,
right? So they need to show that in the non -project
areas that that indicator, percent of women who
decided to use family planning, is statistically
significantly lower than the target areas. It's
got to show impact, right? Because if the project
$41 million spent doesn't really make a statistically
significant difference in that percent, it brings
into question the efficacy and impact of the
project. Second indicator, percent of school
-going adolescent girls and boys with appropriate
knowledge. on adolescent nutrition, sexual reproductive
health, et cetera. Now, what have we got here?
I think we don't know. I must confess, I don't
have the answer here because I'd have to look
in the PMF and that will take a bit of time.
So we'll look at the third indicator. Again,
the same issue, percent of mothers with appropriate
knowledge on good maternal nutrition before,
during, and after pregnancy. Again, The indicator
is excellent, but the percent is measured annually,
just annually. That suggests for this indicator
that throughout that year, before they measure
it, the group of mothers that aren't exposed
to the project, they also could figure out the
knowledge on good maternal nutrition before,
during, and after pregnancy. So they have to
put in the PMF how their project... They just
got to mention control group, comparison group,
or statistical significance analysis on the percent
over time, right? They just got to put that in
there. As far as I know, it's not in the PMF.
So I'm going to assume they're not looking at
it and they need to look at it. They want to
make the claim that their project increased the
percent statistically significantly with impact.
Next outcome. improved effectiveness of local
stakeholders in target countries and Canada on
gender equitable local and international nutrition
specific and sexual reproductive health rights
activities advocacy and policy dialogue so the
outcome indicators there are Canadians reached
through the project on nutrition health sexual
reproductive health and gender equality issues
so it's a little vague How do they define improved
effectiveness? They don't. Effective in doing
what? Increase in knowledge? But its outcome
indicator is clear, specific to increasing reach.
Canadians reached. The measure of the number
reached is clear, based on counting the number
online, who have read, watched, commented, shared,
donated through their website. awareness campaign
but there's no data on whether their quote awareness
campaign in this project was responsible for
a significant increase in this number so i gave
it a negative there not very good in showing
their project increased reach more than if they
didn't do it same problem with the second indicator
extent of engagement of local health facilities
on gender equitable nutrition, health, water,
sanitation, health, and sexual reproductive health
rights issues. Data source and data analysis
is a focus group discussion, which is a poor
way to measure whether levels of engagement of
stakeholders with health facilities, specific
to if services have been delivered equitably.
They need to better define and measure improved
effectiveness. specific to levels of engagement
and the last one is my favorite degree of effectiveness
of local women and adolescent girls rights groups
and associations feel using a liquored five point
scale in ensuring gender equitable nutrition
health and sexual reproductive health services
do not ask them how they feel about their effectiveness
in ensuring that services are delivered equitably.
Measure their ability to advocate for services
to be delivered equitably. For example, evaluate
written advocacy products they have written or
their presentation of those advocacy products.
That would be a better measure of how effective
they are in ensuring gender equitable services
are delivered, right? So on that one, those three
indicators get a red light, unfortunately. So
if we go to another one, strengthened delivery
of gender equitable and responsive nutrition
health and sexual reproductive health services
for the poorest and the most marginalized women,
adolescent girls and children. The indicator,
percent of adolescent girls who report that they
were offered services without judgment by providers.
Excellent, valid indicator. And it's using a
household survey. That's awesome. But again,
they have to show, using that household survey,
if they used a right -tailed, one -tailed statistical
significance test to show that their $40 million
project increased that percent statistically
and significantly over time compared to where
the project was not operating or compared to
the same target group over time. Excellent indicator,
but there's no indication in the PMF that they're
doing any statistical analysis to show that it's
had a statistically significant difference in
the right direction, percent increase over time.
But it's great. It's expensive to do a household
survey. So you would think they're doing that
analysis, and maybe they are. It's just not in
the performance measurement framework, but it
should be, especially when the performance measurement
framework is trying to show If the outcomes have
been effective and have been achieved. The second
indicator, percent of health facilities promoting
gender equitable and responsive nutrition as
measured by a modified tool in the baseline.
Now, again, without the performance measurement
framework, you'd think, ah, that's pretty good.
But then if you look at the PMF and you look
at the data source, it's quote, the head of the
health facility. So obviously the head's going
to say, yeah, it's great. We did it. We promoted
gender equitable services. We did it. Yes or
no? Of course he or she's going to say yes. Conflict
of interest. They should have an objective measure
of determining if the health facility is actually
promoting gender equitable services by looking
for, quote, gender equitable products at the
facility. Or they could take out the syndicator.
Since the survey of the patients visiting the
facility is covered in the household facility.
Household facility survey. If they, the girls,
the name of the health facility they visited
to get health services. They could actually in
the survey of the girls, in the household survey,
ask them which health facility did you go to?
Could have a whole list in the region and they
could just tick it off. And that way they could
show. If there's been an increase in the health
facilities, that would be a better measure rather
than just asking the head of the health facility.
The last one indicator changes in perception
of female and male clients on the gender responsiveness
of health facilities in providing nutrition,
health, wash, and sexual reproductive health
rights services. So again, the problem here is
if you look at the data source, It's a focus
group discussion. It's a poor measure of tracking
changes in perception. Ask any social psychologist.
You can easily do that quantitatively. Again,
even changes in perception in the desired direction
could happen anyway outside of the project. So
what they need to do there is do a quantitative
and ideally do some stat significance to see
if the percent has gone up. in perception in
the right direction, right? So how are we doing
here? Oh, we're almost done. We're getting to
the global indicators now. So here's another
outcome. Improved adoption of gender equitable
practices in nutrition, health, and sexual reproductive
rights at individual, household, and community
levels. And if you look at the indicators here,
These are definitely globally recognized. So
the challenge is connecting the $40 million project
with whether or not that $40 million project
achieved these three indicators. The first one,
percent of mothers of children who attended at
least four antenatal visits during their last
pregnancy. Project outputs clearly deliver services.
The only challenge is to show if the change in
those percents on the indicators is due to the
project to show the significant impact of the
project. Therefore, the PMF needs to show in
the data collection method a one -tailed, right
-tailed hypothesis where the null hypothesis
is that the observed percent increase was not
statistically significant and the percent increase
could have happened anyway in the absence of
the $41 million project or in the presence of
the $41 million project. But the percent was
negligible, small, not worth the $41 million.
Same goes for the next two indicators. Percent
of women married who are currently using or whose
sexual partner is using at least one modern contraceptive
method. And the third indicator, percent of children
who receive minimum dietary diversity and minimum
meal frequency disaggregated. Right? Good indicators,
global indicators. Just got to make sure that
the $41 million that Canadian taxpayers spent
resulted in a statistically significant different
increase due to the project, right? And finally,
the ultimate outcome, same issue. This is a global
indicator. Improved nutrition, nutrition -related
rights, and gender equality for the poorest and
most marginalized, especially women. adolescent
girls and children under five years of age in
Bangladesh, Kenya, Somalia, and Tanzania. Right?
Three indicators? Very popular, common indicators.
Global indicators. Percent of households achieved
gender equality. Got some funky measure of gender
equality. Nothing wrong with the indicator, but
we've got to show the percent. has gone up statistically
and significantly right in the pmf same with
anemia prevalence among adolescent girls and
the third indicator percent of stunted children
disaggregated by sex and country so it raises
the overall issue of okay let's hope that they
can show that the 41 million dollars was statistically
significantly producing these increases in these
percents in the right direction So the $41 million
really did have impact, right? So that's what
we want to look at. And even when the indicators
are global and therefore we know their proper
valid measures of the outcomes where you just
copy and paste, it's all the other outcome indicators
and the analysis to make sure that it's statistically
significant. That's assuming that the indicator,
and as I've mentioned here, Many of the indicators
are not valid measures of the outcome. So I've
led to the conclusion, and I think I can add
three more now. I can cheat a bit. I can say
that out of the 38 indicators, 10 of them are
really good, valid outcome indicators, not seven,
because I added those three. And the remaining,
what is that, 28? Not good for the reasons of
statistical significance or for other reasons
that I stated previously. So now this will go
off, this episode recording with the PMF plus
the PMF critique that summarizes all of the indicators,
whether they're valid or not, will go off to
the Secretary of State for International Development,
along with the shadow critics for the conservative
and NDP parties and the Bloc Québécois. And with
the recommendation that these improvements be
made. It's a judgment call as to whether or not
the 41 million should be stopped. We don't know,
but I'm not going to go that far. And also, I'm
trying to think now. Yes, we're going to invite
World Vision Canada to come on to the podcast
if they want and respond to my critique of their
PMF. Thank you for listening. And we'll be back
soon as I have another performance measurement
framework. from another Canadian organization.
So my next episode will follow shortly. Bye for
now.