Welcome to the Improving Development Evaluation Podcast.
I'm your host David Wand and welcome to season one,
episode four, part one.
The International Development Organization
where you're going to talk about in this episode,
part one is the Ghana Rural Integrated Development
International Organization or SHORTGRID, G-R-I-D.
You can learn more about this
International Development Organization
if you check out their website at www.GRID.org.
Their headquarters is located in Milton, Canada
and the Ghana Rural Integrated Development
is doing a project called
Improving the Well-Being of Rural Women and Girls in Ghana.
And the cost to the Canadian taxpayer for this project
is $1,337,797.
Now before I briefly describe the services
delivered for this project,
I'm going to first talk about the performance measurement framework
that I acquired through an access to information request
for this project.
The PMF or performance measurement framework for this project
has 18 outcome indicators measuring seven outcomes.
And based on my review of this performance measurement framework,
I have come to the conclusion that only two out of these 18 outcome indicators
properly measure the outcomes.
And therefore I cannot support GRID's claim
that they have achieved their project outcomes
with their project services delivered.
In a breakdown of this summary of the 18 outcome indicators,
11 of the outcome indicators do not measure the outcome at all,
but something else not related to the outcome.
Another three outcome indicators do measure the outcome,
but they have no comparison group outside of the project.
So that means women and girls outside of the project
could have also achieved the outcomes at the same amount
or even better than the project itself.
Another outcome indicator asks people how they feel
about achieving the outcome rather than measuring objectively the outcome.
And one other outcome indicator measures the outcome,
but not frequently enough to claim that the services achieved the outcome
rather than other factors.
So that's my conclusion for the performance measurement framework.
And if you want to receive the performance measurement framework
for this project or and or the summary of all of the 18 outcome indicators,
feel free to email me at evaluatecanadaaid at gmail.com
and I would be happy to send those documents to you.
And also if there's any evaluation experts out there
that would like to be on my podcast to also critique
the performance measurement framework for upcoming episodes,
feel free to email me.
I will be happy to send you those performance measurement frameworks
that I have acquired through an access to information request.
Now the services for this project, there were four target groups
that these services were being delivered to.
And those four target groups were number one, women.
And the focus of the project was to train them on how to grow and sell peanuts
and how to raise and sell goats.
In addition to that, they were also trained on safe food preparation,
healthy behaviors and how to access markets to sell those peanuts
and sell those goats.
A second target group were women co-op leaders
and they were trained in leadership, also peanut farming and animal husbandry
and also in acquiring knowledge on women's concerns.
And a third target group were women and men
and they were trained in household financial management.
And a fourth target group were girls and young women
who received cash and scholarships at the secondary school level
and at the post-secondary school level.
So those services for those four target groups when delivered
were expected for the project to achieve one of seven outcomes.
And those seven outcomes taken from the performance measurement framework are,
outcome number one, increased ability of responsibility bearers
to protect the rights of women in rural northern Ghana.
The second outcome, improved ability of women from 30 rural communities
to advocate for their well-being.
Outcome number three, improved access to senior secondary
and post-secondary education by young women in 30 rural communities.
Outcome number four, improved knowledge and skills
in managing sustainable, resilient and productive agricultural livelihoods
by adult women in 30 rural communities.
Number five, improved participation in equitable household
and community decision-making by women in 30 communities in rural northern Ghana.
Outcome number six, increased participation in sustainable productive work
by women in 30 communities in rural northern Ghana.
And outcome number seven, improved well-being of women and girls
in 30 rural communities in Ghana's northern and Brongahafu regions.
So there you have the outcomes and the services.
Now, for this part one, I'm just going to give you three examples
of some outcome indicators for some of the outcomes in the project
where they run into these various problems of poor measurement
or no measurement at all.
And despite the small size of this project, only over a million dollars
if you've been following the podcast, the other projects I've been reviewing go,
you know, anywhere up from five million dollars up to 37 million dollars.
But in this example, I want to give some credit where credit is due.
And as I'll get into detail here, you'll notice that some
of the indicators are quite good as they start to actually measure incomes
for expected immediate outcomes or longer term outcomes for training on how to grow
and sell peanuts and how to raise and sell goats.
So we'll get right into it now.
So let's take a look at the outcome.
Number four, improved knowledge and skills in managing sustainable, resilient
and productive agricultural livelihoods by adult women in 30 rural communities.
One of the outcome indicators for this outcome number four is, and I quote,
number of women able to produce nine bags of peanuts on one acre farms.
Not bad.
They're getting close.
If you learn how to grow and sell peanuts, eventually you're going to make income
from selling those peanuts, but before you make the income from selling the peanuts,
you have to be able to know how to grow and sell the peanuts.
And so they're actually getting close by saying, let's take a look at the bags
of peanuts that you've produced, which is pretty good.
Only problem is those nine bags of peanuts could be empty.
So I would suggest it's going in the right direction, this outcome indicator,
but ideally what you'd like to do is weigh the bags of peanuts to see the weight
of the peanuts that they've actually been able to harvest.
So I would change the number of bags to the weight in kilograms, and please change the number
of women to the median weight of peanuts produced by these women, along with the number
of women in the project that were producing the peanuts that are associated
with the median weight in kilograms of peanuts produced.
The interesting thing about the performance measurement framework is
if you take a closer look at it, you'll notice that the project staff are responsible
for coming up with how many women produced nine bags of peanuts.
Slight conflict of interest there.
They review the project harvest records, as it says in the performance measurement framework.
That should ideally be done by a third party.
They should just go in and go to the women and say, show me your peanuts.
Let's weigh them.
That's a more accountable and transparent way of coming up with this weight
in kilograms of peanuts produced.
So that's a good indication that this organization, GRID, is going in the right direction.
They're saying, show me your peanuts, but you want to get the weight, ideally.
That's what you'd like to do.
So the second outcome, outcome number two, improved ability of women from 30 rural communities
to advocate for their well-being.
The indicator here, one of them, is the percent of women co-op leaders able to identify one
or more things, quote, they plan to lead their co-ops and their communities in doing
to improve quality of life.
The problem with this indicator is the plan to lead does not measure improved ability
to advocate.
It's too soft.
It's too indirect.
Also, a plan to lead is not advocating.
The performance measurement framework shows this is done through focus group discussions
with these women co-op leaders.
That's not good enough.
What you could do is, since you're training these women co-op leaders to be technically
able to advocate, you should test them objectively on their ability to advocate.
So for example, you could look at some advocacy or communication documents that they've produced
and score them.
And you could also look at their presentation skills, because I presume that they have to
advocate to somebody, maybe the government, local government, people, governments that
are going to respond to the interests of these women.
So you have to see what their presentation skills are also.
These are just examples.
But the point is, a plan to lead is not good enough.
If you're training them on how to advocate, you're training them how to lead, then you
should objectively measure their ability to lead and their ability to advocate.
So you could check out their documents that they're producing for advocacy on their quality.
And you could check out their communication presentation skills on advocating.
And you could score those.
In addition, even if their technical ability to advocate does go up after the training,
you need to have either frequent measures to show that, or a comparison group of women
who've been not trained by the project, but are advocating for women in some other part
of the country that are not in the project, because you would expect women who've been
trained in this project to have better levels of advocacy skills.
The same goes for, back to the other outcome, for the weight of the peanuts.
You would expect women peanut farmers in Ghana, for example, outside of the project, to have
lower median levels of weight of peanuts compared to the project where they were actually training
them on how to sell and grow peanuts.
So it would be helpful also to have a comparison group there too.
Another example is outcome number six, increased participation in sustainable, productive work
by women in 30 communities in rural northern Ghana.
The indicator here is the percent of women earning at least $500 per year through productive
work.
As you can tell, the indicator earning income has got nothing to do with increased participation.
Now you could argue that if they are increasing their participation, eventually their incomes
are going to go up from doing that agricultural work, which is fine.
And in fact, it's an excellent start for this organization to start measuring incomes, rather
than asking them how they feel about their incomes, actually starting to measure.
The only thing I would suggest here is, once again, like the number of bags of peanuts,
here instead of just measuring the percent of women who earn $500 or more per year through
the selling of peanuts or the selling of goats, they could simply get the median income for
each of these activities for selling peanuts and for selling goats.
But it's still useful because they're moving in the right direction by looking at income.
And also again, they should have a comparison group of women outside of the project who
are also selling peanuts and raising, selling goats to see if their incomes, their median
incomes are lower.
And that in fact, what we would do is instead of the percent of women, you would just have
the median income for these women and the median income for another group of women in
the country that have not been part of this project and compare their incomes.
Because you would obviously expect the project training to be better at training them and
their incomes would go up and that those income levels would be greater than the median income
of the women who do not have the benefit of the training from the project.
We don't know as you know, if that's going to be the case unless you have a comparison
group.
So those are just some examples of where the outcome indicators do not properly measure
the outcomes.
So what I'm going to do now is share the summary of all of the 18 outcome indicators along
with the performance measurement framework with the representative from the Ghana Rural
Integrated Development Organization and invite them to respond to this critique of part one.
We'll make sure that we go through at least seven outcome indicators covering the seven
outcomes in the project.
But ideally we'd like to go through all 18 to explain why they believe or why they disagree
with these indicators being a problem in supporting their claim that they've achieved their outcomes.
And then once we finish part two, we will send it to the Minister for International Development
as well as the shadow critics for the opposition party and the third party recommending that
again immediately the performance measurement framework for this project and all future
projects for this organization and for all organizations for that matter be included
on the global affairs Canada project browser website so that everybody can see that these
performance measurement frameworks actually show the flaws that we've discussed.
And ideally the second recommendation would be to begin starting to properly measure the
outcomes with better outcome indicators such as three examples I have provided in this
part one.
Thank you for listening and stay tuned for episode four part two featuring the Ghana
rural integrated development international organization.