In this episode we're going to introduce you to an international development organization.
That international development organization is the United Nations Population Fund.
You can learn more about the United Nations Population Fund if you go to their website
at www.unfpa.org.
Their headquarters is located in New York, USA.
The project we're going to be talking about, that the UNFPA has been funded by the Government
of Canada to deliver, is called Human Resources for Sexual Reproductive Health Rights and
Gender-Based Violence.
This project is being delivered in South Sudan and the cost of this project to the Canadian
taxpayer is $37 million.
And in this part one I'm going to give you a brief description of the services delivered
in this project and a critique of the United Nations Population Fund, South Sudan's Performance
Measurement Framework.
The UNFPA have developed this performance measurement framework to measure the project's
performance using the outcome indicators in that performance measurement framework.
I obtained this performance measurement framework through an access to information request.
I also obtained the United Nations Population Fund Evaluation Plan for this project.
However, this evaluation plan provides no additional detail as to how the outcome indicators
will be used to measure the project outcomes for this project.
Therefore I will not be discussing the evaluation plan as part of this episode.
But before I describe the project services that are being delivered for this project
by the UNFPA in South Sudan as well as the expected outcomes to be achieved from the
delivery of those services, I'm going to go to my summary and conclude that in this
performance measurement framework for this project there are 16 outcome indicators that
are measuring 7 outcomes for this project.
My review of all 16 outcome indicators has led me to conclude that all of these indicators,
except one, are not properly measuring the outcomes.
So, the organization, the United Nations Population Fund, South Sudan, country office, cannot make
the claim that it's achieving its project outcomes with the services they're delivering
in the project simply because their outcome indicators are not properly measuring the
outcomes for this project.
As you probably know from the trailer for this podcast, those 16 outcome indicators
fall into one of five problem areas.
In this project, eight of those 16 outcome indicators fall into the no comparison group
problem, which is where there's no comparison group of people outside of the project that
are also measured on the same outcome.
Which means even if the people in the project are achieving the outcome, we don't know if
they're achieving them any better than another group of people that could also achieve the
outcome outside of the project without receiving the project services.
Another seven of the outcome indicators out of the 16 fall into the problem area where
the outcome indicator is not measuring the outcome at all and is measuring something
not related to the outcome.
So, and finally, we have one outcome indicator that is properly measuring the outcome.
So, before I give you three examples where the outcome indicator fails to properly measure
the project outcome, I'm going to briefly describe the services the project delivers
and the target groups that receive those services.
And then I'm going to list the seven outcomes that the project expects to achieve from delivering
those services to those target groups.
There are 11 target groups in this project.
The first target group is staff at targeted health facilities and the services being delivered
to them are training on a maternal and perinatal death surveillance and response system so
that these health facilities have a functional system to report on maternal and perinatal
deaths.
Staff at these health facilities are also provided training on how to repair and maintain
solar and water supply systems since these health facilities are provided essential medicines,
solar panels and water supply.
These health facilities are also provided with health care workers, especially health
care workers trained in delivering gender responsive sexual reproductive health services
and being able to respect the rights of those receiving these services.
A second target group are health care personnel.
This group receives training in how to mentor and supervise students completing their practicums.
A third target group are staff at one stop center family protection centers who are provided
training on how to deliver information and services for victims of gender based violence,
including how to take cases to court.
A fourth target group are women and girls and men and boys who are delivered messages
on the importance and availability of using sexual reproductive health and gender based
violence services.
A fifth target group are community action groups and religious and civic leaders that
receive messages to promote women's use of sexual reproductive health and gender based
violence services.
It is not clear from the performance measurement framework, but we can assume that either training
or messages are also delivered to these community action groups on how to prevent and respond
to gender based violence and child marriage.
Because the outcome in the PMF is and I quote, community structures are mobilized to take
actions to prevent and respond to gender based violence and harmful practices.
A sixth target group are men and boys who are oriented on their role as men and boys
to prevent and respond to gender based violence and other harmful practices, as well as promote
women and girls rights to access sexual reproductive health and gender based violence services.
A seventh target group are service providers and community based organizations, state and
national coordination bodies who receive training on sexual reproductive health and gender based
violence prevention and response.
An eighth target group are health training institutions, including tutors within these
institutions where these tutors are trained on how to train health professionals in gender
sensitive sexual reproductive health rights and gender based violence services.
These institutions are also provided with educational materials and solar power.
These health training institutions include health sciences institutes that receive quality
assurance regular field supervision visits.
A ninth target group are health professional associations that are provided with technical
support on how to advocate for and monitor sexual reproductive health and gender based
violence services.
A tenth target group are Ministry of Health staff at national and state level who are
provided with technical support on how to coordinate and manage sexual reproductive
health and gender based violence services.
Finally, the Performance Measurement Framework, PMF states an expected outcome is of laws
and policies being developed, revised and enacted with the Minister of Health and Parliament
passing those acts.
Those acts are the Midwife Act and the Anti-Gender Based Violence Act.
Therefore the 11th target group would be the Minister of Health and Parliament.
However, the PMF does not indicate if the Minister of Health and Members of Parliament
need training to achieve this expected outcome of passing these acts.
They probably are assuming in this project that the Ministry of Health staff and the
Health Professional Association staff, not the politicians, through their training received
from this project is sufficient to convince these politicians to pass these acts.
So that gives you an idea of the services this $37 million project delivers in South
Sudan by the United Nations Population Fund.
Now after those services are delivered they expect to achieve for these target groups
seven outcomes.
And those outcomes are, outcome number one, improved national and state government capacities
to develop and implement sexual reproductive health, gender based violence, HIV policies,
legislation and coordination mechanisms.
Outcome number two, strengthened capacities of health training institutions to produce
quality human resources for sexual reproductive health rights, gender based violence and HIV
services.
Outcome number three, increased awareness among women and girls to utilize gender responsive
sexual reproductive health, gender based violence and HIV services.
Outcome number four, strengthened capacity of targeted national and state level health
facilities to provide quality integrated sexual reproductive health rights, gender based violence
and HIV services.
Outcome number five, strengthened enabling environment at the national state and local
level for women and girls access to sexual reproductive health rights and gender based
violence services.
Outcome number six, increased use of gender responsive sexual reproductive health rights
and sexual gender based violence services by women and girls, especially those living
in vulnerable situations.
And finally, outcome number seven, increased realization of sexual and reproductive health
and rights of women and girls in South Sudan.
Now we're going to look at three outcomes and three outcome indicators for each of those
outcomes and explain how they do not properly measure the outcome.
So let's start with outcome number five, strengthened enabling environment at the national state
and local level for women and girls access to sexual reproductive health rights and gender
based violence services.
One of the two outcome indicators in the PMF that is used to measure strengthened enabling
environment is proportion of targeted health science institutions that meet the minimum
educational standard for training health professionals.
We're going to assume that if these health science institutions are training health professionals
up to a minimum educational standard, that this is good for women and girls who receive
sexual reproductive health and gender based violence services from these students who
graduate from these institutions and start delivering these services to these women and
girls.
So this is the same as saying we have a strengthened enabling environment for women and girls when
they access these services.
That seems to be a good match between the outcome indicator and the outcome statement.
But when we look at the performance measurement framework for this outcome indicator, we see
that an assessment is done every three months at the institution along with a progress report.
Upon reviewing the performance measurement framework as well as the evaluation plan,
I cannot find a measurement tool on how this proportion of health science institutions
that meet this minimum educational standard for training health professionals is calculated.
The PMF shows that the people inside of this institution who are responsible for training
health professionals will be assessed as to whether they meet this minimum educational
standard.
The outcome indicator is excellent, but there is no explanation in the performance measurement
framework as to how they calculate this percentage.
We need to ask the United Nations Population Fund, South Sudan representative, where is
the method and calculation that comes up with this percentage?
Also in the PMF, the staff at the health science institution itself, the staff at the Ministry
of Health and the United Nations population staff are responsible for coming up with this
percentage.
It would be better if an external third party would be responsible for assessing these staff
in these health science institutions to see what percentage of them have reached this
minimum educational standard for training health professionals.
Then this could be repeated across all health science institutions to get the proportion
of all health science institutions that achieve this minimum educational standard for the
indicator.
That would remove any bias since right now the United Nations Population Fund staff and
these health science institution staff are assessing themselves on how well they train
the health professionals to reach this minimum educational standard.
You may recall from the project services I described earlier that tutors at these health
science institutions are trained by the project on how to train health professionals to be
quote gender sensitive in their delivery of sexual reproductive health rights and gender
based violence services.
The project also delivers quality assurance regular field supervision visits at these
health science institutions which should or could include supervision of these tutors
in how they train midwives in delivering gender sensitive sexual reproductive health rights
and gender based violence services.
So what is the minimum percentage of tutors at these health science institutions that
first need to reach this minimum educational standard based on this assessment that's listed
in the PMF before that health science institution as a whole is declared to have reached this
minimum educational standard for training health professionals.
And then we have to figure that out and then we can move on to the next health science
institution and repeat it so that for all health institutions we figure out what the
percentages that meet this minimum educational standard.
In the performance measurement framework there is a target where they're trying to achieve
80% of health science institutions reaching this minimum educational standard.
But if only one tutor let's say out of a hundred tutors in one of these institutions shows
up for the assessment it passes it.
Is that enough to say that there's a sufficient number of tutors that have passed the minimum
requirements to train the midwives before we move on to the next health science institution?
So the percentage could be misleading we just don't know what's going on inside the institution
to come up with this magical percent of health science institutions that have reached this
minimum educational standard.
So again we need to ask the United Nations population fund what percent of tutors do
you require to assess before you conclude that the whole health science institution
of all the tutors is good enough and it's reached the minimum educational standard.
And then what percent of those tutors do you have as a minimum requirement to say ah we
got 50% of them met the assessment minimum educational standard therefore we conclude
that the whole health science institution is good enough we'll go on to the next health
science institution before we get to this target of 80%.
So we need to ask United Nations population fund about that and what's the assessment
tool that they actually use to say oh that tutor you're up you're ready you have reached
the minimum educational standard when you're training midwives on delivering gender-based
violence and sexual reproductive health services you're okay and then they move on to the next
tutor and assess.
So we see there's progress reports we see that they do an assessment but there's no
mention of how they come up with this percentage and that should be included in the performance
measurement framework and even in the evaluation plan which is there's nothing in there it's
just a generic evaluation plan for any project it's not specific to this project so that
is a problem.
Let's look at outcome number six increased use of gender responsive sexual reproductive
health rights and sexual gender based violence services by women and girls especially those
living in vulnerable situations.
One of the outcome indicators measuring this increased use of gender responsive sexual
reproductive health rights and gender based violence services is proportion of women aged
15 to 49 who receive four or more antenatal visits as per targeted locations.
This indicator has two problems first there is no mention of the proportion of women who
receive four or more antenatal visits where those antenatal visits were specific to gender
responsive sexual reproductive health rights as reported by the women and girls who receive
these antenatal services.
Remember this project trained tutors at health institutions to quote train health professionals
in quote gender sensitive sexual reproductive health rights and gender based violence services.
The indicator should include the same statement in the outcome on gender responsive sexual
health reproductive rights to read proportion of women aged 15 to 49 who receive four or
more antenatal visits as per targeted locations and where those visits were reported by these
women as being quote gender responsive and respecting their sexual reproductive health
rights.
So we need to ask United Nations population fund why did they drop the increased use of
gender responsive sexual reproductive health rights from the outcome indicator when it
is expected to be achieved as stated in the outcome.
The second problem is that the outcome indicator is only measured in quote targeted locations.
So even if the proportion of women who receive four or more antenatal visits and who report
those antenatal rights as being gender responsive and respecting their sexual reproductive health
rights is going up thanks to the projects men and boys promoting women and girls right
to access those services and the projects training of tutors to train health professionals
on being gender responsive and respecting sexual reproductive health rights.
It could be going up even more in another part of the country where the project is not
delivering those training services right.
So we need a comparison group of women not located in the project area that are also
receiving antenatal services to also ask them if the antenatal services they received also
respected their sexual reproductive health rights and were gender responsive.
Then we compare their percentage against the percentage of women in the project who report
those antenatal services.
We would expect the project percent to be higher.
Finally, outcome number seven increased realization of sexual and reproductive health and rights
of women and girls in South Sudan.
One of the outcome indicators for this outcome is the maternal mortality ratio.
This indicator is a universally agreed upon indicator and could be seen as a proper measure
of increased realization of sexual reproductive health for women and girls.
If these women and girls are experiencing lower rates of death due to pregnancy after
receiving sexual reproductive health services in this project.
But there are two points I need to make clear here.
First, the project delivers these necessary sexual reproductive health services to women
and girls in quote targeted locations.
For example, from the performance measurement framework, health facilities in these targeted
locations are provided with health care workers.
This suggests there are health facilities outside of these targeted locations that do
not receive these health care workers along with the sexual reproductive health services
they deliver.
Secondly, this project also delivers other services other than sexual reproductive health
services to women and girls such as training for tutors and mentors of midwife students
on how to deliver gender responsive sexual reproductive health services and how to respect
the sexual reproductive health rights of women and girls when delivering these services.
There's also training to men and boys to promote sexual reproductive health rights for
women and girls.
There's also training to Ministry of Health staff on how to manage the delivery of sexual
reproductive health services to these women and girls.
There's also training to health profession associations on how to advocate for and monitor
sexual reproductive health services to these women and girls.
So even if the percent of deaths due to pregnancy goes down, we need to show that this $37 million
project did it better with greater reductions in maternal deaths than in locations where
this project was not operating.
And we also need to show that all these trainings delivered to all these other target groups
worked better than these same target groups in locations where the project was not delivering
these trainings.
Once again, we need comparison groups for each of these target groups.
So now what I'm going to do is email this episode, part one, to the United Nations Population
Fund representatives, inviting them to attend part two of this podcast where they can respond
to this critique of this performance measurement framework.
I will also send to the UNFPA representatives this performance measurement framework as
well as my summary critique of all of the 16 outcome indicators for the project.
And I'm also going to give the United Nations Population Fund representatives the option
of sending me a written response to my critique if they are unable to attend the podcast.
An evaluation expert will be attending part two of this podcast.
And in part two of this podcast, we will be covering all of the seven outcomes for the
project with at least one outcome indicator for each of the seven outcomes.
If you have any comments or suggestions for this episode or wish to receive a copy of
the performance measurement framework and my summary critique of all 16 outcome indicators
for this UNFPA project, you can email me at evaluatecanadaaid at gmail.com.
And if there are any evaluation experts who'd like to be on the podcast to look at other
performance measurement frameworks, you can email me and I'd be happy to send you the
remaining performance measurement frameworks that we have lined up for season one.
Thank you for listening and stay tuned for episode three, part two, featuring the United
Nations Population Fund.
And there's $37 million project in South Sudan called the Human Resources for Sexual Reproductive
Health Rights and Gender-Based Violence.