Welcome to the Improving Development Evaluation podcast.
I'm your host David Wand and welcome to season two,
episode two, where we're going to feature
the international development organization
called Doctors of the World.
And if I recall correctly,
they are located in Montreal, Canada.
They have received $14,886,968 according to the Government of Canada Global Affairs Canada
project browser website.
For a project they are delivering in the West African country of Burkina Faso.
And the name of that project is called Access to Health Rights in Northern Burkina Faso.
And you can learn more about Doctors of the World, the international development organization
at their website www.medecinsdumonde.ca.
And in this project of $14 million, courtesy of the Canadian taxpayer,
there were eight expected outcomes to be achieved from delivering services in Burkina Faso
for that cost of $14 million.
And for those eight outcomes, they had 15 outcome indicators that they used
from the performance measurement framework to see whether or not those eight expected
outcomes had been achieved.
My review of those 15 outcome indicators indicated that 13 out of the 15 were not valid measures
of the expected outcomes and therefore Doctors of the World cannot make the claim that even
if those outcomes were achieved, they cannot claim that the project costing $14 million
was responsible for the achievement of those expected outcomes.
Before I go into detail about those outcome indicators and the outcomes, I'm going to
give a brief description of what the project from the performance measurement framework
actually delivered.
And at any time, if you're interested in getting the performance measurement framework plus
the critique in an Excel sheet that I have, you can email me at evaluatecanadaaid.gmail.com
and I would be happy to email those documents to you.
So let's begin with a brief description of the project.
The project in Burkina Faso delivered by Doctors of the World has roughly 16 target groups
and each one of those target groups receives a certain amount of services.
And with those services, they expect some of those target groups to achieve certain
expected outcomes.
So starting with target group number one, International Dialogue Forum participants.
They are supposed to receive from Doctors of the World presentations at these international
forums to advocate for access to health.
Second target group, authorities and stakeholders in Burkina Faso at local and national level
and they are to receive from Doctors of the World presentations to this target group to
advocate for access to health.
Target group number three are community and religious leaders and they are to receive
support in quote, facilitating awareness sessions.
Target group number four, peer educators.
They're supposed to receive trained sessions to promote to men and boys their responsibility
on their right to health, sexual reproductive health and gender based violence.
Target group number five, men and boys.
They are receiving training on the right to health, sexual reproductive health and fighting
gender based violence in quote, targeted areas.
Target group number six, civil society organizations.
They are trained on how to promote health rights, sexual reproductive health and fighting
gender based violence.
Target group number seven are women and their allies and they're trained and mobilized for
the right to health, sexual reproductive health and the fight against gender based violence.
Target group number eight are women and girls and they are sensitized and supported on health
and gender based violence in quote, targeted areas.
Target group number nine are families with children under five and in targeted areas
they are trained on the mid upper arm circumference measure to enable them to detect nutritional
edema and make referrals.
Target group 10 are communities, women, men and adolescents in these communities.
They are to be sensitized on primary health care services, sexual reproductive health
and gender based violence.
Target group 11 are community based health agents and community relays and they are to
be trained on prevention, care and referral for primary health care, sexual reproductive
health and gender based violence.
Target group number 12, it's not clear who exactly they are but these are people that
are accepting referrals to treat women and girls and they're trained on referrals treated
with no financial charge to patients.
So I guess what that means is they're treating patients that they've received referrals from
women and girls where they're not going to be charging them for services.
Target group number 13 are health providers.
They're trained in epidemic management and rapid response during a humanitarian emergency.
Target group number 14 is remote areas and this is where mobile clinics go to those remote
areas deployed to deliver free medical services including sexual reproductive health services.
Target group 15 are supervision teams and they're trained on quote provider supervision
techniques and finally target group 16, I don't know what the acronym stands for FOSA,
FOSA.
They are supervised and supported to deliver primary health care and treat gender based
violence cases.
So that gives you an idea of the target groups and the services delivered at a cost of roughly
$14 million.
So now we're going to look at the eight outcomes and for those eight outcomes the 15 indicators
that were used to measure whether or not those eight outcomes were achieved and there could
even be more but from my review of the performance measurement framework this is what I figured
out were the eight expected outcomes.
The first one was and I quote increased mobilization of actors to take into account gender inequalities
in particular sexual reproductive health issues in emergency response.
The outcome indicator number one, actors involved is measured enough and also there's no comparison
or control group outside of the targeted areas for this project so we don't know if the number
of actors outside of the project also were involved in issues of gender inequality and
access to health in the response emergency.
So keeping with the trailer for season two this would violate the development assistance
committee criteria on impact because there's no comparison group or control group so they
can't make a claim that any difference they see in this indicator such as an increase
in the number of actors it could happen anyways regardless of the project so it violates the
impact criteria for the development assistance committee and this is a big thing in international
development where you'll see they often go to these five questions or issues of evaluation
in the development assistance committee criteria one relevance two coherence three effectiveness
four efficiency and five impact and six sustainability.
So in this performance measurement framework in this project most of the violations are
on impact they just simply can't claim that the project is responsible for achieving the
outcomes so indicator number two for this same outcome is the number of advocacy priorities
taking into account gender inequalities in particular sexual reproductive health issues
in the emergency response presented to the authorities for consideration in local and
national policies again problem is it's not measured frequently enough and there's no
comparison group if we look at the performance measurement framework I can't see it right
here I'm claiming it's not measured very frequently it's probably measured only once every it's
measured up every six months the first one that we just covered so anything could happen
in six months between those measures actors in other communities outside of the project
could also be getting mobilized and it's the same for this indicator here indicator number
two they only measuring it every six months and the other thing that's interesting here
and it's it's been done before is presented to the authorities for consideration in local
and national policies a policy is just a piece of paper what you could do instead is just
see if expenditures addressing gender gender inequalities has gone up actually expenditures
not the budget that they claim they're going to spend but after what have they actually
follow the money and see if it's been spent so for example did they spend money to increase
enrollment of girls in primary schools or did the number of girls or women with access
to sexual reproductive health services the number of visits free of charge actually go
up thanks to maybe an increase in expenditures in that area by the government so that would
be better measures rather than this advocacy priority taking into account gender inequalities
to see if after the advocacy did they actually result in increased expenditures and again
you could still argue even if it goes up the expenditures it may have nothing to do with
the project that's why you need to have some sort of comparison group or control group
the next outcome is called increased co-responsibility of men and adolescents in favor of access
for all to the right to health in the targeted areas and the outcome indicator for this outcome
this is outcome indicator number three for outcome number two proportion of men and adolescents
who participated in project activities who report having adopted behaviors in favor of
equitable gender norms obviously there's self-reporting bias there they're going to say sure sure
I adopted the right behaviors because it's hard to measure the behaviors directly so
self-reporting is going to have a bias so even more importantly here you need another
comparison group outside of the project where men and boys weren't being engaged with to
change their attitudes right so that is what you need to do here and again this violates
the impact criteria the DAC development existence committee criteria because we don't have a
control group we can't claim that the men and boys outside of the quote targeted areas
also had a preferable shift in attitude we just don't know so we need to measure men
and boys in areas outside of where the project is going on and this is a challenge of course
but it should be at least attempted so again for that increased co-responsibility outcome
there's another indicator they use outcome indicator number four proportion of men and
adolescents participating in activities that have quote increased their knowledge of the
right to health including sexual reproductive health and the fight against gender-based
violence now here you're measuring knowledge levels which is fine but again what you need
to do is have a comparison group because their knowledge levels could go up anyways in other
areas where they're not operating the project so again this violates the impact criteria
for the development assistance committee criteria on evaluation the next outcome number three
is strengthened capacities of women and adolescent girls affected by the conflict to exercise
their right to health in particular with regard to sexual reproductive health in the targeted
areas and the outcome indicator number five for that is proportion of women and adolescent
girls participating in the empowerment program who have quote increased their knowledge of
sexual reproductive health so again if you want to show an increase in knowledge you
have to show due to the project so ideally what you'd like to do is find a comparison
group of women and adolescent girls outside of the project and see how their knowledge
levels are going before during and after the project and then you could show again that
the project is having some statistically significant one-tail positive effect right just to find
some women and girls not in the project areas another option is if you don't want to use
control groups you could just measure more frequently and just show that the knowledge
levels are going up rather than just a pre and a post test level I again I'd have to
look at the performance measurement framework in detail to see how frequently they measure
this another outcome indicator for this strength and capacity of women and girls specific to
their knowledge levels is quote number of new initiatives in favor of sexual reproductive
health or the right to health taken by influential women and their allies so this is a separate
group of women who take it up and again there's no comparison group here so this violates
the impact criteria and here I have made a note according to the performance measurement
framework this is only measured once a year so we could experience this increase anyways
in between the time that the women and girls are trained on their capacity for their right
to health in targeted areas and whether or not these new initiatives have been taken
up by influential women and their allies so it's only measured once a year anyways so
even without the control group once a year is not frequent enough outcome number four
women and adolescent girls sensitized and supported in terms of health and gender-based
violence in targeted areas and outcome indicator number seven for that outcome number four
is proportion of women and adolescent girls in the community who have completed the health
sexual reproductive health and gender-based violence empowerment program in targeted areas
that's just attendance showing up but that doesn't indicate how many quote got sensitized
they showed up for the program but did they get sensitized whatever that means from receiving
that empowerment program so it's just attendance so what they need to do is find a better indicator
to actually measure that sensitization due to the empowerment program that they have
received and I also looked at the performance measurement framework and noticed the annual
empowerment program activity reports once a year so again it's not very frequent if
they want to show that their sensitivity levels have gone up or their empowerment levels have
gone up they have to measure more frequently than just once a year and then get a better
indicator rather than just showing up attendance who completed the program which is what they
currently have and also maybe have a comparison group if they can here is it's a good example
of how the indicator fails to validly measure the outcome in the first place where they're
just taking attendance rather than measuring any levels of sensitivity or empowerment so
outcome number five community and religious leaders quote strengthened and mobilized for
the promotion of the right to health for all including sexual reproductive health including
the fight against gender-based violence this was initially reported as an output but strengthening
suggests the ability to advocate and promote so I considered it an outcome the outcome
indicator for this is the number of community and religious leaders supported in facilitating
awareness sessions this would suggest just attendance they're not really looking at their
ability to mobilize their ability to advocate and their ability to promote so again the
indicator fails to measure any increases in the ability to promote health rights for all
it only shows the number who were supporting in facilitating awareness sessions but no
measure showing that their ability to promote health rights for all has been strengthened
so they should introduce a measure that shows that those who attended the community and
religious leaders who delivered the awareness sessions experienced increases in awareness
preferably an indicator and measurement tool that has been shown to be a valid measure
of the ability to promote health rights for all and increase awareness plus measuring
those who attended on their awareness levels against a non-project target group so outcome
number six increased capacity of conflict affected communities especially women and
adolescent girls to utilize primary health care services including sexual reproductive
health outcome indicator nine for this outcome number six proportion of women men and adolescents
targeted by the project who can name at least three types of modern contraceptive methods
again this is a knowledge indicator so if you want to show that the project services
targeting these particular individuals has increased their knowledge on contraception
you need to have a comparison group or a control group so this again violates the impact criteria
for the development assistance committee criteria on evaluation the performance measurement
framework shows that the percent is measured at the start middle and end of the project
but no mention of a controller comparison group so with a control group and the statistical
sampling they mention in the performance measurement framework they actually mention it they could
complete if we follow their interest in statistics a one-tailed hypothesis test and see with
95 percent confidence if they can claim that the percent increases were statistically significant
and due to the project training so that's what they have to do otherwise right now it
violates the DAC criteria on impact because they're not showing that their project group
is statistically significantly greater in terms of the percentages who can name at least
three types of modern contraceptive methods another outcome indicator for that outcome
number six this is outcome number indicator number ten proportion of women men and adolescents
quote who have improved their knowledge during training and awareness raising on the use
of primary health care services including sexual reproductive health same issue no comparison
or control group and they according to the performance measurement framework they only
measure this once before the training and after the training so that could show an increase
due to the training but if we don't have a control group or a comparison group you could
show also another group of women who didn't show up for the training somewhere else in
the country who also just over time improved increased their knowledge levels and awareness
levels so that has to be taken into account and once again that violates the DAC development
assistance committee criteria on impact evaluation outcome number seven improving the social
community and political environment in favor of the right to health in particular sexual
and reproductive rights and gender equality for women and adolescent girls affected by
the conflict in northern Burkina Faso outcome indicator 11 for this outcome number seven
proportion of women men and adolescents targeted by the project having improved their behavior
in favor of sexual and reproductive rights this is tricky because I'm assuming having
improved their behavior this has to be self-reporting which is biased and again no control group
or comparison group outside of the project so even if these percentages increase you
need a control group where these women men and boys have not received the various trainings
etc from this project their behaviors could also increase so the DAC impact criteria is
not met here also same in outcome number seven we have another outcome indicator number 12
number of sexual reproductive health initiatives undertaken by civil society organizations
that seek to mitigate traditional practices harmful to women and adolescent girls according
to the performance measurement framework this is only measured once per year and again there's
no control group so even if you don't have a control group or a comparison group outside
of the project you could measure it more frequently and that might help in allaying concerns that
these initiatives would happen anyways regardless of the trainings that these women men and
adolescents or CSO civil society organizations receive to uptake these initiatives so that's
the other problem there again no control group no comparison group so the DAC criteria of
impact cannot be claimed to be achieved another indicator for this same outcome outcome indicator
number 13 number of national policies or guidelines adopted or revised taking into account gender
inequalities in particular sexual reproductive health issues in the emergency response outcome
indicator third now again you could argue here that the number of national policies
adopted or revised is really not related to the outcome of improving the political environment
in favor of the right to health in particular the right to access to sexual reproductive
health services because as I mentioned earlier a policy is just a piece of paper a better
measure would be an actual increase in the amount and percentage of the government budget
that was actually spent on women and girls specific to accessing sexual reproductive
health and gender based violence services so even though they have a policy on a piece
of paper do they actually put their money where their policy is this is a common sort
of indicator that many governance organizations when monitoring governance and government
institutional performance look at because they can have all the policies in the world
but really what you need to do is follow the money especially when it comes to in this
project they are doing a large portion of this 14 million dollars in advocacy so really
what you should be doing is tracking government expenditures outcome number eight increased
enjoyment of rights relating to health in particular sexual and reproductive health
of women men girls and boys affected by the conflicts in northern Burkina Faso here we
have two indicators and I look at these indicators and I think they're fairly universal in this
area and the first indicator outcome indicator number 14 of the project is percentage age
15 to 19 who have given birth to their first child who or who are pregnant with their first
child and the idea clearly there is to try to reduce this percentage and I think it's
a universal global indicator that's used which is fine the indicator is a valid measure of
the outcome of increased access and enjoyment of rights relating to health and what is tricky
here is this is the only indicator where you could argue it is a valid measure but again
all of the services I described for those 16 target groups that this 14 million dollars
is paying for in the delivery of services to those target groups and you're hoping that
this indicator will go down that's fine but to show that you need to have some sort of
comparison group or ideally in the demographic health survey they call it where they measure
this one of the questions on that demographic health survey should be by the way did you
participate in any way shape or form in the doctors of the world project paid for by global
affairs Canada and right there you could do a split and see without getting too methodologically
complex here because I don't claim I'm a statistician happy to get email saying sorry that's not
how you do it but at least we could compare the two groups when we do the survey and those
who say no what was their rate of pregnancy between age 15 to 19 compared to the group
that said yeah I did show up at those awareness sessions on contraception and then you could
do a comparison but right now I don't see that in the performance measurement framework
it's a great indicator but we don't know if the project for 14 million dollars actually
contributed to a reduction in that percentage age 15 to 19 that reported being pregnant
we just don't know and finally the last outcome indicator number 15 proportion of women who
have decided to use family planning alone or jointly with their husband or partner again
an excellent indicator that suggests if this project is successful and has an impact quote
unquote this indicator would be going up the proportion of women who have decided to use
family planning would be going up and it would be going up statistically significantly compared
to a group of women who were not participating in the project so ideally what you want to
know if this question is being asked in the demographic health survey did they participate
in this doctors of the world Burkina Faso project in any way and then you could see
if the project was in any way responsible for contributing to this increased improvement
and performance on that indicator so that's my summary and as I'll repeat I would say
in its current form right now the performance measurement framework is flawed and the project
and doctors of the world cannot make the claim that their 14 million dollar project is achieving
the outcomes that they claim they're achieving in the project so now I'm going to send this
episode to the Minister for International Development along with the shadow critics
and I'm also going to invite doctors of the world to provide a written response to this
critique and also an invitation if they wish to be on the podcast to respond to the critique
and if that happens we'll be doing another episode thank you for listening and stay tuned
for a further episode and I do have some good news that we have located in the United States
the actual grant for an American USAID project that was funded to an implementing partner
and so hopefully that will in turn result in getting the activity monitoring evaluation
learning plan and then we can critique that which is similar to these Canadian performance
measurement frameworks bye for now