Welcome to the Improving Development Evaluation Podcast.
I'm your host David Wand and welcome to Episode 3, Part 2, where we continue our discussion
about the United Nations Population Fund South Sudan Project, which is called Human Resources
for Sexual Reproductive Health in South Sudan.
As you may recall from Part 1, I reviewed 16 outcome indicators covering 7 outcomes
and came to the conclusion that the South Sudan UNFPA Country Office could not make
the claim that they were achieving their project outcomes simply because their outcome indicators
were not properly measuring their outcomes.
On November 18th, I sent to the UNFPA South Sudan Country Office through their contact
form an invitation to attend the podcast today on December 14th.
But as of today, we have not received a reply from UNFPA.
I also sent it the same invitation to the headquarters in New York, HQ at UNFPA.org,
as well as the East South African Regional Office at isarro.info at UNFPA.org.
And I also sent it through a contact form at UNFPA.org in New York, indicating it should
go to the evaluation office in New York.
Even despite those four contacts, I haven't received a reply from any of them regarding
this invitation.
But fortunately, we do have an evaluation expert with us today from Accra, Ghana.
His name is Benjamin Serbo and he holds a master's degree in political science from
the University of Zululand in South Africa.
And he is a monitoring and evaluation specialist.
And we're glad to have him on the podcast to provide more insight into the flaws in
the performance measurement framework for this project that the UNFPA is delivering
in South Sudan.
Welcome, Benjamin.
How are you?
Yes, I'm good.
Thank you.
And yourself?
I'm fine.
I'm very happy that you could join us.
So what we're going to do now is we're going to go through each of the seven outcomes
and to make sure we cover the whole spectrum of the project.
And then if we have additional time, we'll go back to some of the other outcome indicators.
So starting with the outcome in this project, improved national and state government capacities
to develop and implement sexual reproductive health, gender-based violence, HIV policies,
legislation and coordination mechanisms.
And Benjamin, you're going to be taking a look at an outcome indicator for that outcome.
Okay, you're right.
Thank you very much.
So the outcome indicator for the number of state coordinating mechanisms and coordination
plan.
David, is that correct?
That's correct.
So first of all, let me say this, when you talk about the state coordinating mechanisms,
it is not only the project which is going to develop the state coordinating mechanisms.
And the number of guests who are going to develop this mechanism.
And so the project cannot claim that.
If you're not paying those state mechanisms for it, then you can sue the project.
So then you need to look at the kind of contribution that the project made.
So for instance, when you look at having an hour with black, that's not necessarily mean
you supported or you did it, you supported it.
As with either some of the areas that the project will have to look at.
Now, one thing we need to know is that when you talk about having a state coordinating
mechanism, and I know it will not be able to justify how the project was continued,
improvement of state capacities.
Rather, I would like to see the number of project officials who contributed in the development
and the implementation of those annual coordinating.
Because when you are conducting an evaluation of the project and you know how the project
contributed to improving state and national capacities, you have to assess how the beneficiaries
of your project played a direct role in the development of those mechanisms.
So one of the indicators we can look at in this, for instance, the number of project
beneficiaries who participated in the implementation of the annual coordinating.
So assuming the five of your project beneficiaries participated, then we can get some qualitative
information that can support the provided to the development.
So in that case, we'll be able to know the contribution that our project made to strengthen
the state.
So these are some things that we can look out for.
But if we just did the number of state mechanisms, annual coordination, yes, we can have the
number.
The project will not be able to clearly demonstrate how it contributed to the development of the
mechanisms or development of the annual.
That's a good point.
And also it seems to be misplaced, the indicator.
It's not quite right, let you say, is focusing on who there's no who in here.
And so, yeah, it should absolutely be specific to the project participants who were trained
on developing these mechanisms.
Yes, that's a good point.
And maybe you're going to say the next part.
So they also, maybe I will just add, if you look at the outcome, it's very clear that
they want to improve capacity.
And just saying number is not good enough.
You have to show that these mechanisms and coordination plans have actually been improved
in terms of quality.
So once you get the project participants involved, then you have to show that from the trainings
that the project participants received, they actually improve the quality of these coordination
mechanisms or annual coordination plans.
Perhaps you were going to say that next.
Yeah.
Okay.
So that's great.
So it really actually shows two things that are missing.
It's misplaced, the indicator, and it's not measuring increased levels of capacity.
That's great.
So let's move to the next outcome, which is strengthened capacities of health training
institutions to produce quality human resources for sexual reproductive health rights, gender
based violence, and HIV services.
And Benjamin, I think you were going to look at an indicator there.
Okay.
So with this indicator, with this outcome, they indicate in the outcome proportion of
targeted health training institutions, MOUs signed with targeted practical site.
And the number of supportive machines conducted at a health teaching.
With this outcome also, now let's indicate the proportion of targeted health institutions,
MOUs.
Strengthening MOU does not necessarily translate into strengthening the health institution.
It doesn't show any, because MOUs now by health institutions has strengthened.
So then when we talk about strengthened capacities, we are looking at how the training institute
participating in the project have improved their structure, process, human resources
to deliver quality training to various healthcare providers.
So just finding MOUs conducting supportive visits, that's not the most ridiculous.
So we're looking at these two indicators.
We cannot use it to measure the strength and capacity of the health institute.
So and liberal you rightly mentioned, we can measure the achievement by looking at the
percentage of faculty that's reached certain level of technical capacity.
So in that case, the UNFPA considered a freeway index.
Number one, can you get a percentage of teachers or trainers of beneficial health, who meet
the required national or international certification for expertise.
Usually we have this certification which can be used by the national.
So in order for this to be strengthened, we need the people working in this institution
to find all the expertise.
So then in that case, we can say our institution, the institution has strengthened with the
staff achieving these issues.
Even that's one indicator.
Now another indicator you can number health training institutions, who are currently meet
the required industry standards.
So I believe the training curriculum for the HRHRN keeping as opposed to meet certain standards.
Obviously there will be criteria to determine it meets the standards.
So the institute meeting the standards be a very good indication of its strength and
capacity to deliver quality training and produce quality human resources.
Actually this is another indicator, those can be considered.
And then other areas that can be considered are the satisfaction of the health care that
will receive training.
And even the beneficiary of the gender based fund is and then sexual and reproductive health
service from this health care by this because when you get a fee from the institute, they
will satisfy the kind of service that we provide.
It's going to help you to mention or to determine whether the institutions are strengthened
and whether these institutions are providing quality, GVVN, HRHRN services to the brand
and to the people.
So these are the indicators.
Thank you, Benjamin.
That's great.
And so what we know here is that the indicator in its current form is not measuring the outcome
at all.
At all.
It is not measuring the outcome.
That's right.
Yeah, an MOU signed is clearly not a measure of strength and capacity.
Great.
Okay.
So we'll move on to the next outcome, which is increased awareness among women and girls
to utilize gender responsive, sexual reproductive health, gender based violence and HIV services.
And there are two outcome indicators for this outcome.
And I chose the second one, proportion of women, girls from targeted communities who
report making their own decision for seeking sexual health services disaggregated by age.
I correct myself, Benjamin.
I've seen my notes.
This is yours.
Sorry.
Go ahead.
That is mine, right?
The outcome indicator is proportion of women, girls from targeted communities who report
making their own decision for seeking sexual health services disaggregated by age.
So we are talking about increased awareness here.
So when you talk about it, increased awareness, it will mean the procedures and processes
they need to give you the HRH services.
So then this could lead to reported cases.
And this is something I will see.
It was awareness to resolve a number of cases reported.
They understand the processes, you ensuring that their awareness has been treated.
There are GBV cases.
So now let's look at this.
With the original indicator, you don't have any comparison in there.
So now your project is getting its awareness.
It can happen to the natural population somewhere else.
And then they also go in a student.
So then if you have a comparison, you will be able to extend to which your project made
the difference.
So with it, current state without Paris, it will be difficult for the project to make
the contribution to creating awareness.
So in order to ensure that we are to be confident, the project is strong, patient awareness.
We need to make sure that the patient is aware of the patient's awareness.
Yes, thank you, Benjamin.
And if you look at the performance measurement framework for this indicator, they do measure
it three times, baseline, midline, and I think they call it end of the project.
And you'd think, oh, well, if we can show that the percent of women reporting this has
gone up, that's good enough, but not necessarily like you pointed out.
So here we have in the performance measurement framework, three measures for this indicator,
a baseline, midline, and end of project.
But even if the percentage of women who report utilizing services has gone up, which is good,
it could be going up even greater somewhere else in the country, like you said, where
the awareness campaign and money spent by this project to increase awareness like radio
ads, brochures.
I'd have to go back to part one to find out exactly what I said, how they increased awareness.
But still, you need a comparison group to show that the project is doing it better.
So yeah, that's a good point.
Okay, so we can move on to the next outcome, which is strengthening capacity of targeted
national and state level health facilities to provide quality, integrated sexual reproductive
health, gender-based violence, and HIV services.
And the indicator I looked at here, there's only really one, it's a cluster of, I think,
three.
And those three are number of visits for family planning services in targeted locations, number
of visits for antenatal clinics in targeted locations, and number of deliveries by skilled
birth attendants in targeted locations.
And the problem here with these indicators for this outcome is they're not really measuring,
as we've discussed earlier on another outcome, they're not measuring strength and capacity
of the health facilities.
And if you look at part one or listen to part one and look at the project, the number needs
to be changed to the number and percent of targeted facilities that have solar power,
have water supply, have adequate staff trained to maintain the solar and water supply that
have sufficient essential health workers that can deliver gender responsive sexual reproductive
health services, gender-based violence services, and HIV services.
So the number is just inadequate.
You need to go a step further and figure out the percent of staff that actually have the
capacity to provide the services in a minimum quality way, because it says quality integrated
services.
So just counting the number of visits or the number of deliveries or the number of visits
for family planning or antenatal services is not good enough.
And that's why it falls into the problem area of not related to the outcome of showing strengthened
capacity of these targeted health facilities.
And the other point is, even if they modify the indicator to have a criteria of what is
an acceptable minimum percent of staff trained at each facility that knows how to repair
and maintain the solar power panels and the water supply, as well as having a minimum
number of staff that have adequate skill levels to deliver the services, you could argue that
in another part of the country where the project's not operating, they've got even a higher percentage
of health facilities that have strengthened their capacity to deliver services.
So that's basically the issue for that outcome indicator.
Now the next one is the next outcome is strengthened enabling environment at the national, state,
and local level for women and girls access to sexual reproductive health and gender based
violence services.
And we've talked about this earlier, so it kind of overlaps.
But it's the same issue.
The indicator here is the proportion of targeted health science institutions that meet the
minimum educational standard for training health professionals.
Now that's an excellent indicator.
But the problem is we don't know inside of these institutions what's going on.
And in the performance measurement framework, they actually talk about the staff that train
these health professionals being assessed.
And there actually is progress reports based on those assessments being produced.
But we don't know what sort of measurement tool they're using to measure these staff,
to say that they've reached a minimum educational standard, and then for that one health institution,
we don't know if that's sufficient to conclude that the health science institution all by
itself has reached the minimum educational standard.
And like you had mentioned earlier, Benjamin, another indicator could be even looking at
the curriculum content to make sure it's up to a minimum educational standard.
So unfortunately, UNFPA are not here to explain.
And I've looked at their evaluation plan, and there's no measurement tool there talking
about how they come up with this percent.
This is despite the fact that they have a target of 80% of health science institutions
reaching this minimum educational standard.
So we need to ask the UNFPA, how are they measuring the staff?
And coming up with this calculation to conclude that this is the percent of targeted health
facilities that meet the targeted health science institutions that meet this minimum educational
standard for training health professionals.
We simply don't know how they calculate that.
So it's not adequate or related to the outcome of saying we have strengthened the enabling
environment for women and girls access to services, because we assume when they graduate
these midwives, that they're up to standard, and that the women and girls who get services
from them have experienced a strengthened enabling environment.
But we just don't know.
The next outcome is increased use of gender responsive sexual reproductive health rights
and gender based violence services by women and girls, especially those living in vulnerable
situations.
And the outcome indicator for that is the proportion of women aged 15 to 49 who receive
four or more antenatal visits.
Now this again has issues here where they clearly have targeted sites per targeted locations.
So we need to measure these same indicators for areas in the country of South Sudan where
the project is not operating so that we can show that the proportion of women who receive
four or more antenatal visits is higher in the project area for this project, $37 million,
compared to health facilities that haven't been targeted.
Because one of the services is actually to deploy midwives from these institutions to
these targeted health facilities.
So we would expect the percentage of women in that catchment area for that targeted health
facility to get greater access.
So the only way to know that is to have a comparison group, and we don't have any comparison
group mentioned in this performance measurement framework.
So that's something that's important to recognize is that it's really important to have comparison
groups when they are even explicitly saying we are targeting particular locations, which
means other locations we're not targeting.
So the project should be showing that in the targeted locations, they're getting greater
rates of service where higher percentage of women are receiving four or more antenatal
visits.
And the final outcome-
And David, if I can-
Yes, jump in.
I mean, yeah, usually when we get to the outcome level, I usually want to see some kind of
feedback from whoever is benefiting.
So for instance, we are looking at proportional age 15 to 14 who receive or antenatal visit.
Assuming they receive this for more than one, so then on the outcome level, I would like
to see those women who are receiving this, they can observe it.
They receive from this, but to what extent are they satisfied with this kind of service?
So in a matter of fact, they receive the services that normally are not required to function.
Then we want to high number of women or patients receiving this to express satisfaction.
So if we have like only 40% of them being satisfied, then we have a problem.
So then on the outcome level, these are areas that we can look at.
Those who are receiving the service, they are at a level of satisfaction with the service.
Yeah, David, back to you.
That's an excellent point because it reminds me of a point I missed.
The indicator actually says increased use of gender responsive sexual reproductive health
and gender-based violence services.
So that implies that they not only are receiving the service, like you said, but they're satisfied
that the services being delivered are gender responsive.
And that's a big focus of the current government of Canada in a lot of these projects they
fund is not only getting them trained up to capacity to deliver the service, but that
it's gender responsive.
And that's very precise in the outcome.
And you're quite right.
In the outcome indicator, they have missed that.
So that's a very good point.
And that's also needs to be noted in the indicator, not only just the proportion of women who
report getting the service, but the proportion of women who report satisfaction with the
service being delivered being gender responsive.
Yeah, good point.
And the last outcome of the project is as we go up the results chain is the increased
realization of sexual and reproductive health and rights of women and girls in South Sudan.
And the indicators there are excellent.
They're universally agreed upon.
The two indicators are maternal mortality ratio and the unmet need for family planning.
And UNFPA is very, very good at measuring these.
They do large surveys using standard public health surveys.
But the problem here is the one we mentioned earlier is how do we show that the $37 million
that the Canadian taxpayer paid for this project has reduced the maternal deaths at a lower
rate than women and girls who unfortunately did not benefit from this project.
Now UNFPA may come back and say, it's a universal project.
It goes all across the country.
But my understanding is no, because they mentioned targeted locations.
So that means there's an opportunity to pick a non-project group of women and girls and
measure their maternal mortality ratios also.
So otherwise you don't have evaluation.
You just have a charity project delivering necessary services, but we don't know if they
made a statistically significant difference in reducing the maternal death ratio compared
to a group of women and girls that did not receive the project services.
So yeah.
Any comment you want to make Benjamin before we?
No, and I really agree with you on this point.
Because when you are looking at maternal mortality ratio, the project is supposed to reduce maternal
mortality ratio.
So while we are having the project, we expect that by the end of the project we will start
seeing a reduction in maternal mortality ratio.
What about the other areas where the project is not being implemented?
Is it the same?
Maybe they will even do a better project.
So then you ask people how they want to support that, right?
Even though they are not receiving any support or they are not receiving any intervention,
but we are still having a reduced maternal mortality ratio.
And that's why it's always important to have the reason.
Because it can help you to think about the strategies that you are using to implement
your project.
If there is something that you should have done, which is done some way, so I agree with
you.
That's why it's always important to have a comparison group.
It helps you to learn the project.
Thank you, Benjamin.
I mean, to be fair to UNFPA, another option is they may say we don't have the funds to
do a large scale comparison group measuring these people outside of the project.
The other thing they could do is measure more frequently just the project group, before,
before, before, during, during, during, during, after, after, after.
So this is a standard thing that people try to do, as I mentioned in my trailer of this
podcast, where you can sort of get around the problem of not using comparison groups.
It's quasi-experimental, they call it.
So there's a whole bunch of ways you can maybe measure more frequently rather than just,
you know, like they mentioned, baseline, midline end of project.
And you could maybe even do small samples of those women and girls more frequently throughout
the project.
But there's really never any discussion in the evaluation plan about this problem and
how to get around it.
It's always the same cookie cutter approach, which is pre-test, post-test only when they're
training people, which we know is the same problem, it's not good enough.
Or when they get to larger scale measurements like maternal mortality, they just do baseline,
midline end of project.
And they just assume that they can assume that's $37 million is why the maternal deaths
have gone down.
So they have options.
And the other option I've discussed on other podcasts is just be a charity.
Just say, look, these women and girls need these services, full stop.
We're not going to evaluate them.
This is an emergency humanitarian situation.
And we're going to deliver these services.
And we don't care about evaluation because we know we're saving lives by giving them
access to these services.
And that may explain why there's limited evaluation.
A lot of people are just assuming they're a charity, but unfortunately they're making
these claims that they are achieving all these outcomes, but then they're not prepared to
evaluate.
So in the wider scheme of things, we get it.
And we understand that's why I have the podcast to sort of say, maybe you can make a choice
here and just be a charity and not make these broad claims of achieving outcomes.
That's your other option.
So yeah.
Okay.
Before we go, are there any other, I think I was going to say we could add, we could
talk about other indicators, but I think we've done pretty well here.
Did you want to talk about some other indicators or should we just go to the chase, which is
sort of to conclude from you at least, what you think of the PMF in its current form?
Or do you want to talk about some other indicators?
What would you like to do?
Because there is another indicator.
There's the number of policy recommendations to government and key stakeholders on health
and gender-based violence from that first indicator, first outcome, which is improved
national and state government capacities to develop and implement sexual reproductive
health, gender-based violence, HIV policies, legislation, coordination mechanisms.
There is another indicator there.
Number of policy recommendations.
Yes.
I think I have them.
So policy recommendation, you know, so then is it a recommendation project from evaluation
report or my project beneficiary or under development darkness?
I mean, the way it has been stated now is to open and for me when I pick, you know,
all three, I will not know where do I get the information to measure the number of policy
recommendations.
Who is making this recommendation?
So then it needs to be, yeah, because there are a number of policy recommendations that
are made to that.
So then the project will need to be specific.
So you can see a number of recommendations made, the project, a result of our evaluation
findings or monitoring findings.
So of course, we have them planned to monitor and monitor the project.
So probably as a result of findings from our evaluation or monitoring, we made this recommendation
to the government.
But then when I pick, I will not know, so this indicator is beneficiary, as a result
the knowledge they have by making this recommendation to the government.
All the evaluation we want, the monitoring of the assessment, as a result of making this
recommendation to them.
So then this is something that needs to be considered.
Actually it's a good point, because if you look at the performance measurement framework,
the outcome indicator is buried in the outputs.
It's misplaced.
Actually have a number of functional, national, and state coordination mechanisms.
So they actually have a good indicator of increased capacity, but it's buried in the
outputs.
So it's good, but the problem is they just have project reports which report on the quality
of these recommendations based on the training that the project delivers to these Ministry
of Health staff.
The problem is there's no attribution that the project training has contributed to this
increased quality of the policy recommendations.
It just says technical support, but it's only done annually, and that's not good enough
to show that the training, or the technical support they call it, is responsible for the
improved policy recommendations.
So it's buried in the PMF, but it's lower down, and even it's there, even though it's
a good indicator, as I mentioned in the summary report that the listeners can get a copy of,
they don't adequately measure it to claim that the project training was responsible.
It's just an annual report, and they could figure this stuff out all on their own at
the same time or in between the trainings.
So that's one of the issues that's common in a lot of these performance measurement
frameworks is they do testing to show increased skill knowledge acquisition, but they don't
do it frequently enough, or they don't use comparison groups.
So there's no way of knowing if the project was responsible, like you were talking about
contribution.
And a lot of the times they'll say contribution instead of attribution, because attribution
is a dirty word, but they're not even there in this case.
So that's a good point.
I think we're good.
So I wanted to ask you, completely biased of course, the performance measurement framework
and its current form, what do you think?
Is it adequate for $37 million UNFPA South Sudan project to make the claim that they're
achieving their outcomes?
In its current state, we say no.
So the performance management framework needs to be revised.
Because when you look at the indicators in this case, the original indicators, they had
the rain and the rain was that day.
So even using the performance measurement framework says your contribution was held in
South Sudan.
I think it needs to be revised, I mean it needs to be revised.
All the indicators, or most of the indicators, needs to be revised again.
And those were good for probably the M&A office, the representative from the M&A office of
the UNFPA to be here.
I'm not going to say it myself, but you will understand it very well.
So it's a minute to us.
So at this time, for even the PIAF, we denounce to raise other contributions.
Yeah, and even though they're required by Global Affairs Canada to produce an evaluation
plan, all of the evaluation plans I've received, including the one for this project, they're
all generic in nature.
They don't go through each outcome indicator like we have just done and explain how they're
going to show the project is achieving the outcome.
And that's an issue.
Yeah, that's good to know.
So what we're going to do now, and I should point out that the Global Affairs Canada,
their own guidelines on designing performance measurement frameworks makes this clear, that
you have to have outcome indicators that reliably and validly measure the outcomes.
So they're not even following their own guidelines.
This is not some thing I've cooked up myself.
I'm just bringing it out into the public domain.
So what we're going to do now is I will send part one and part two to the Government of
Canada, Minister of International Development, as well as the performance measurement framework
our summary sheet that explains all the indicators and why they have this problem.
And these two episodes, part one and two, and also sending it to the shadow critics
of international development for the Bloc Québécois, the New Democratic Party, and
the Conservative Party.
So everybody knows this is a Canadian problem.
It's not specific to a particular political affiliation.
It should be a concern for all Canadians that there are $4 billion that they spend every
year in developing countries.
It should be properly evaluated.
And we'll leave it at that.
So Benjamin, thank you very much for your time.
Thank you for having me.
And thank you for your expertise and insight.
It was very helpful.
All right, David, thank you too.
Okina.
Ok.
All right.
Bye.
And have some Bangkok for me, please.
In Italy.
Whenever you are coming to Ghana, you know.
That's the same.
All right.
Midasi.
Ok.
Mr. Midasi.
Midasi.
Okina.
Ok.
All right, David.
Bye bye.