Welcome to the Improving Development Evaluation Podcast.
I'm your host David Wand,
and welcome to episode four, part two,
where we continue our review and critique
of the Ghana Rural Integrated Development,
that's GRID, G-R-I-D,
International Development Organization,
and their project in Ghana entitled,
Improving the Well-Being of Rural Women and Girls in Ghana.
The cost to the Canadian taxpayer for this project
is $1,337,797,
and it has 18 outcome indicators covering seven outcomes
that we will review each of those 18 outcome indicators
in this part two.
On January 7th, I sent an email
and also left a voicemail with GRID,
inviting them to attend the podcast.
I did not receive a reply.
Then I followed up again inviting them also again
on January 18th with an email and voice message,
but so far have not heard a reply regarding my invitation
for them to attend this podcast in part two,
to respond to my critique.
So I will go through the 18 outcome indicators
covering the seven outcomes,
starting with the first outcome,
which is increased ability of responsibility bearers
to protect the rights of women in Northern Ghana.
For this outcome, they have two outcome indicators.
The first outcome indicator is number of times strategies
have been implemented by responsibility bearers
to protect the rights of women.
And the second indicator is the percent of women
who are fairly or very confident
that community leaders are protecting their rights.
Now the problem with this indicator is
the number of strategies that have been implemented
by responsibility bearers to protect the rights of women
is quite a good indicator.
You would think that as the strategies go up,
this shows an increased ability to advocate.
The problem is there's no comparison group
outside of the project of other responsibility bearers
to see how well they are protecting the rights of women
in Northern Ghana.
So that was the problem with that indicator.
The second indicator, percent of women who are fairly
or very confident that community leaders
are protecting their rights is also quite a good indicator
because you would expect as confidence levels go up
in these women, in their perception of the increased ability
of these responsibility bearers,
this would seem to be a good measure
of their increased ability to protect them.
But unfortunately again, there's no comparison group.
So we don't know if responsibility bearers
in another part of Ghana outside of the project
could also be doing quite well.
So that was the problem with these two indicators
for that outcome.
The next outcome is improved ability of women
from 30 rural communities to advocate for their wellbeing.
And there's three indicators for this outcome.
The first indicator is the number of women farmers
able to successfully advocate for long-term access to land.
The second indicator is the percent of women
who feel they are completely or mostly able to advocate
on behalf of women's concerns.
And the final indicator is percent of women co-op leaders
able to identify one or more things they plan
to lead their co-ops and or communities
in doing to improve quality of life.
The problem with this, the first indicator,
able to successfully advocate for long-term access to land
is that it doesn't directly measure
the ability to advocate.
A better measure would be just to look
at the amount of land that they actually acquired
for the women.
If they're advocating for access to land,
instead of advocating for access, successfully advocate,
they should just look at how much land
they were able to acquire for these women.
You could even argue that it's not related,
the indicator in its current form.
So that would be a better way to improve the indicator
to actually measure their improved ability to advocate.
And that is actually getting the land for the women.
The second indicator, percent of women who feel
they're completely or mostly able to advocate
on behalf of women's concerns, they're asking for feeling.
And instead, as I've mentioned before,
they should measure directly their ability to advocate,
more objectively than just asking them if they feel
they are completely or mostly able to advocate.
It's not a proper objective measure
of the ability to advocate.
The third indicator, the percent of women co-op leaders
able to identify one or more things they plan
to lead their co-ops or communities in doing
to improve quality of life.
They measure this through focus group discussions
and they only do it three times
over four years of the project.
This is not going to show that the training they received
to advocate is actually responsible
for their ability to advocate.
There's no objective measure there.
It's a focus group discussion
and they don't do it frequently enough.
They only do it three times over the four years.
So again, a better measure of their ability to advocate
would be to test them on their ability to advocate
and maybe do that every six months
and even better compare it against another group of women
who are advocating outside of the project
and see on the same measures, maybe a test
or reviewing their advocacy products that they deliver,
such as maybe a pamphlet or a radio announcement
where they're advocating for these women's rights
and they could compare the quality of the products
they've produced against another group of women
and rate and compare the quality
of those products objectively.
The next outcome is improved access to senior secondary
and post-secondary education
by young women in 30 rural communities.
The two indicators for this outcome are
the number of women less than 25 years of age
attending secondary and post-secondary school
and the number of women versus men
completing senior secondary school by age 18.
Now, these outcomes of improved access
actually are quite straightforward
because they're just trying to deliver more education
to these women through cash to pay for their school fees
or scholarships to pay for their post-secondary tuition.
So this is pretty straightforward
in terms of improving access to education.
This would be considered to be good measures.
The problem is it's more a service through cash
and scholarships rather than expected outcomes
after they've attended school.
The next outcome is improved knowledge
and skills in managing sustainable, resilient,
and productive agricultural livelihoods
by adult women in 30 rural communities.
Here we have three outcome indicators for this outcome.
The number of women able to produce nine bags of peanuts
on one acre farms.
Second indicator is the number of women
owning one or more healthy female goats.
And the third indicator is the percentage of women
who are moderately or highly confident
in their knowledge and skills in farm management
for food and crop production.
And the percentage of women who are moderately
or highly confident in their knowledge and skills
in livestock rearing.
Now, I mentioned this in part one,
they're getting close,
number of bags of peanuts on one acre farms,
it's pretty good, but they haven't weighed the bags,
they could be empty.
So we wanna get a third party to get in there
and actually weigh the bags of peanuts.
So we have the total weight of peanuts produced,
which would be a good measure of their ability,
skill and knowledge levels in growing peanuts,
which is good.
And even better, it should be the median weight of peanuts.
And they should compare that against another group
of women peanut farmers that did not get access
to the training through this project.
And hopefully you'll see the median weight
of this group through the project
having a higher median weight
than the women outside of the project
who did not receive any peanut farm training.
And the same goes for the goats.
You would wanna get the median number of goats
and compare that with another group of women
who are raising goats,
but haven't had the opportunity to receive training
on how to raise goats.
And finally, the last outcome indicator
is the confidence levels
in their knowledge and skill levels.
I would say you don't need that.
If you're directly measuring
their ability through how many peanuts
in weight they produced
and how many goats they have produced and sold,
that would be sufficient.
There's no need to ask them on their confidence levels
regarding their knowledge and skills.
So you wouldn't even need that third indicator.
The next outcome is improved participation
in equitable household and community decision-making
by women in 30 communities in rural Northern Ghana.
We have three indicators for that outcome,
and they are number of community members
reporting that women participate
in key household decisions.
Second indicator, number of community members
who are satisfied with women's participation
in community decision-making.
And number three, the third indicator
is the number of girls completing or planning
to complete secondary school and post-secondary education.
So in the first one, community members reporting
that women participate in key household decisions,
that's a bit of a tricky one
because you're asking community members
to report on another group of women.
That's the women that actually are inside of the household.
That is kind of questionable.
I would rather survey the women themselves
inside their households to see how they feel they're doing
when it comes to participating in key household decisions.
So there's the flaw there, there's not a direct measurement.
The second indicator, community members who are satisfied
with women's participation in community decision-making,
that's a pretty good measure.
They could improve it a bit by actually looking
at the actual representation of women
on these community decision-making bodies,
such as boards, school boards, for example,
or even municipal elected councils,
to see the percentage of women that are on these boards.
These are bodies, community decision-making bodies
that actually have power to make decisions.
As for the third one, I would take it out.
Percent of girls completing secondary school
or post-secondary school, yes,
it's in a measure of improved participation,
obviously through attendance,
but it's not related, the completion of educational school
with the equitable household decision-making.
Although you could argue that the more girls
that graduate from secondary school
and post-secondary education,
the more knowledge they have in terms of trying to make
a decision in the household or at the community level,
but the other two indicators, if improved upon,
would be sufficient to measure improved participation.
The next outcome is increased participation
in sustainable productive work by women in 30 communities
in rural northern Ghana.
Here we have three outcome indicators for that outcome
of increased participation in productive work.
They are the percent of women earning $500 or more per year
through productive work,
the percent of women under 25 completing
or expecting to complete secondary
and post-secondary education,
and the third outcome indicator for that outcome
is the percent of young and adult women
who think they are somewhat or very likely to be hired
for a job outside their household.
Now, the first indicator I've mentioned previously
is excellent.
They are trying to get at the incomes that are measured
from learning how to grow and sell peanuts
and learning how to raise and sell goats.
So it's excellent.
They're trying to get in the right direction.
It just needs a slight correction,
and that is instead of 500 per year,
they should measure the median income
of the 750 women in the project
and watch their incomes go up.
It also should be specific,
like earlier in the performance measurement framework,
they should be measuring the median incomes
for those growing peanuts and those raising goats.
So it's not just productive work,
it's precise, specific to the project training,
which is on how to grow peanuts
and how to raise goats and sell them, right?
The second indicator, again,
is talking about completion rates for women
in secondary school and post-secondary education.
Yes, it's a good measure of increased participation
in productive work because they will have the potential
to increase their participation in earning a livelihood
based on the benefit of completing their education.
So yes, that is important.
I think focusing on the incomes is sufficient enough,
and I would eliminate that indicator.
You can keep it into track how many women as an output
completed their education after getting the cash
from the project or getting the scholarship
from the project, that's useful.
But in terms of productive work,
I would look at incomes as a better measure
of whether or not they've increased their participation
in productive work.
The last outcome indicator there,
dealing with the percent of women who think they are somewhat
or very likely to be hired for a job
outside their household, again,
it's not a bad measure, but it's indirect.
I would focus on the incomes.
The next outcome is improved wellbeing of women and girls
in 30 rural communities in Ghana's northern
and Brong Ahafl regions.
And the indicators there, they have, there's two indicators.
The first one is percent of women living on $2 US
or more per day.
And the second indicator is level of perceived wellbeing
on a scale from one to 10 of the participating women.
Now the first indicator is a standard,
I think, World Bank measure.
They don't have to do that.
They can just, again, focus on the incomes over time
for the project, for the training they have received.
They don't have to stretch it and try to claim
that there's this percent of women in the larger community
living on $2 US per day.
There's no connection.
It's too far stretched.
They should just focus on the incomes of the women
in the project to make sure that they have been going up.
And ideally, they should compare it against women
who are not receiving training on how to grow peanuts
and how to raise goats and track their incomes also.
And hopefully the project will have higher median incomes
than those women not in the project.
That will be a better way of evaluating
the effectiveness of the training project
that we're talking about today.
Now the second indicator, the level of perceived wellbeing
of participating women, that's a pretty good indicator.
I think it's okay.
But if we look at the performance measurement framework
in more detail, you'll notice it's only measured once
per year over this four years of this project.
So it would be very difficult to claim
that the perceived wellbeing is going up
if it is in fact going up,
that it's due to all this wonderful training
that these women are receiving on how to grow
and sell peanuts and how to raise and sell goats
if it's only measured once a year.
There could be other factors involved
in why their perceived wellbeing is going up
that has nothing to do with the training
they received in the project.
So ideally what you'd like to do
is measure it more frequently in the project
and again compare it, this perceived level of wellbeing
with other women outside of the project.
Ideally those also involved in peanut farming
and goat farming and then you could show
that the perceived wellbeing is going up greater
in the project compared to the women outside of the project
who are also measured on their perceived wellbeing.
Now in addition to these seven outcomes
and the 18 outcome indicators I've covered,
there's also another eight outcome indicators
that all have the same problem.
They have pre-test before the training
and a post-test just once before they get tested
on the knowledge and skill levels
of the content of the training
and then they get tested once more afterwards.
Now that's pretty good.
They're at least doing some objective testing
which is great.
The problem is they only do it once before and once after
and as I've explained in the trailer,
they need to show more frequent testing
unless it's done during the same day.
Now that's interesting.
750 women, they come in and do a test before the training
and then are tested after the training
and you could show that their skill levels have gone up
from the test, that would be good.
We don't know from the performance measurement framework
if it's a same day training.
Ideally what you also want to have is a comparison group
where they're measuring women
who don't attend the training at all
but are also given the same test before and after.
That's ideally what you want to do
so you also want to have a comparison group
and those content trainings are in the following areas.
Ability to identify justice issues specific to women
along with an appropriate solution.
Ability to identify key actions to maintain health.
Ability to identify best practices
for safe food preparation.
Ability to identify ways of successfully accessing markets
and understanding of basics in sustainable peanut farming
and small ruminant rearing.
And knowledge of peanut farming
and knowledge of small ruminant husbandry.
I guess that's the goats.
And knowledge of women's concerns.
So they do pre and post testing
for these various subject areas
but I think if they had comparison groups
that would be better because then they could show
that the project training is more effective
increasing knowledge and skill levels in these subject areas
compared to a group of similar women
who are not in the project.
So now that I've finished my review and critique
of all the outcome indicators for the project,
if you want to email me at evaluatecanadaaid at gmail.com,
I would be happy to send you
the performance measurement framework for this project
as well as the summary of all of the 18 outcome indicators.
And once again, if there are any evaluation experts
out there who would like to participate in a part two
of a podcast where we go through all the outcome indicators
explaining how they do or do not fit properly
with the measurement of the outcome,
feel free to email me and I'm happy to send you
the remaining performance measurement frameworks
along with my critique
of the performance measurement framework
so that you could participate
as we have a lot more PMFs coming shortly.
Now what I will do is I will email this part two
and part one to the minister for international development
along with the shadow critics for international development
for the conservative party, the new democratic party
and the Bloc Québécois.
And I will also recommend in that email
that performance measurement frameworks
immediately be available to the public
on the Government of Canada,
Global Affairs Canada project browser website.
And obviously also recommend that there be improvements
in development evaluation
by developing better outcome indicators
to measure project outcomes.
Thank you very much and stay tuned for episode five,
part one where we will be introducing once again
another project and the project that we will be doing
in episode five, part one, cost to the Canadian taxpayer,
$19 million.
So stay tuned and thank you for listening.