Welcome to the Improving Development Evaluation Podcast.
I'm your host David Wand and welcome back to Episode 7, Part 2, where we continue our
discussion of the international development organization called Equitas that has their
headquarters in Montreal, Canada, and we are talking about their project worth about 18
million Canadian dollars, courtesy of the Canadian taxpayer, and this is a human rights
project where they're promoting women, gender equality in five different developing countries,
and we're going to go into more detail about their performance measurement framework for
those five developing countries, and they are Burkina Faso, Haiti, Kenya, Senegal, and
Tanzania, but they also use some of the 18 million dollars for promoting gender equality
and human rights in certain regions such as West Africa, and we won't go into detail about
Part 1, we've already done that.
We're going to focus on the outcomes as we usually do in Part 2.
Just to give you some background, on April 23rd, I sent Equitas their performance measurement
framework along with our critique of their performance measurement framework that outlines
why each of the 20 outcome indicators do not adequately measure properly the 10 outcomes
in the project.
Then I also sent them the Part 1 of the podcast along with the PMF and the Excel summary sheet,
and then on April 29th, I received an email back from Equitas where they declined to attend
Part 2, which is today, of the podcast to respond to our critique of their performance
measurement framework, and they also declined to provide a written response as to what their
response would be to our critique, which is unfortunate, but we will proceed as usual
with Part 2, going through each of the 20 outcomes, explaining why they fail to properly
measure the outcomes.
And with me today is an impact evaluation specialist from London, Ontario, Canada.
Her name is Yvonne Okéké.
She has a master's degree in impact evaluation for international development from the University
of East Anglia in the United Kingdom.
Welcome, Yvonne.
How are you?
Thank you, David.
I'm doing absolutely fine.
How are you doing?
I'm fine, as usual, continuing our quest to let people know that we've got some issues
with some international development projects.
So we will start, as usual, with the 20 outcome indicators covering the 10 outcomes, and I'll
start with the first outcome of this performance measurement framework, which is increased
commitment of select women's organizations to engage duty bearers in dialogue on advancing
gender equality in target countries.
And the outcome indicator I'm looking at is, and I quote, number of plans implemented by
select women's organizations to engage duty bearers in dialogue on advancing gender equality
in target countries.
The problem with this outcome indicator, it doesn't adequately measure increased commitment
to engage with the duty bearers because it's just reporting on the number of plans.
And the problem here is the plan has to be of sufficient quality to engage with these
duty bearers.
Just reporting on the number of plans is not adequate.
The plans could be garbage.
We just don't know.
And to be fair to the Equitas, maybe somewhere in the performance measurement framework,
but I cannot find it.
And I want to repeat to the listeners out there, if you want a copy of the performance
measurement framework from Equitas, as well as our critique, feel free to email me at
evaluatecanadaaid at gmail.com, and I would be happy to send it to you.
And you're also welcome to ask for any of the other performance measurement frameworks
and any of the other critiques of those performance measurement frameworks that we've covered
in the podcast so far.
So that's the issue here.
The number of plans is not adequate as a measure of increased commitment.
What they should do instead is measure the percentage of plans that achieve, quote, quality
criteria to engage effectively.
In fact, another indicator in the performance measurement framework that we'll discuss
later actually talks about quality criteria, but they don't seem how to do it here.
Also, a thing to note in the performance measurement framework for this indicator is they also
interview the duty bearers, the ones who receive these women who are lobbying them to promote
gender equality.
And another recommendation here would be to ask them, not the women's organizations,
but go to the duty bearers and ask them, do you think these women who are engaging with
you on advancing dialogue on advancing gender equality, are they doing it adequately?
Because the whole project's based on training them on making sure that they can effectively
lobby these duty bearers.
So that's the end of that one.
So what I'm going to do now is read the next outcome statement and Yvonne's going to look
at the next outcome indicator.
So here the next outcome statement is, and I quote, increased capacity of select women's
organizations to engage duty bearers in better fulfilling their human rights obligations
to advance gender equality in target countries.
And the outcome indicator that you're going to look at, Yvonne?
That is the percentage of intermediaries, men and women, trained in transferring knowledge
to select women's organizations on engaging duty bearers in fulfilling their human rights
obligations to advance gender equality in target countries.
That is the outcome indicator that has been specified.
And this indicator is flawed in that it has the disadvantage of self-reporting bias.
The women's organizations are expected to have increased technical capacity to engage
or lobby the duty bearers based on the training already provided in the previous output.
Hence, they should be tested on this technical ability, just knowing that the knowledge has
been transferred is not really good enough for this particular outcome.
That's right.
And there's even a weird, strange intermediary of the women and men who are trained that
then transfer the knowledge to select women's organizations who then engage with the duty
bearers.
So it's kind of strange.
They're just looking at the percent who are able to transfer.
Now again, at the performance measurement framework, maybe they should have a little
sentence where they say, see the following measurement tool, but they don't.
So we can only assume that they've missed the mark here on properly measuring increased
capacity.
Yeah.
So.
Absolutely.
Yeah.
And the next outcome is increased opportunities for civil society organizations to collaborate
on advancing gender equality in target countries.
And the indicator for this is interesting.
It's quote, type of opportunities available for civil society organizations to collaborate
on advancing gender equality in target countries.
Now initially when you read that, you think, well, if civil society organizations have
increased opportunities to collaborate, that's a good measure, I would think.
You know, when you first read it, you think that's a good measure because they can expand
their opportunities to collaborate, which will over time be a benefit for the country
in advancing gender equality.
Makes sense.
The problem is it's not really defined and it should be connected to the technical capacity
to engage and promote gender equality.
So maybe they do have more opportunities.
So yes, you could argue that's pretty good, but it should be connected to their ability
technically to engage and promote gender equality.
So what exactly will these civil society organizations do when they get the opportunity to engage
to advance gender equality?
They can have all these wonderful opportunities running around, collaborating with other civil
society organizations, but don't forget they have to engage.
They have to promote gender equality.
So that's the problem there is on the surface, it looks good, but we need that type of opportunity
to be clearly defined and assume that that type of opportunity is equated with better
performance on promoting gender equality and then also making sure that that in turn is
connected to a better technical ability to engage and promote gender equality.
So I'm criticizing it, but it's still not a bad indicator to be honest, but it doesn't
go far enough and they don't define it.
Right.
So now we're going to move on to the next outcome, which is, and I quote, increased
capacity of women's and human rights organizations to carry out human rights education to advance
gender equality.
And you're going to be looking at the outcome indicator.
Level of perceived capacity on a four point scale on women's and human rights organizations
to carry out human rights education to advance gender equality.
Now, a measuring capacity is not a bad idea, but level of perceived capacity of those you
train is not a valid measure of technical capacity.
If people have been trained on how to carry out human rights education to advance gender
equality, what should be done to measure the particular outcome is to test them.
They should undergo tests because the perceived measures would need a comparison group, which
is not also provided for in this PMF.
There is no comparison group and also measuring level of perceived capacity for this particular
outcome would not technically tell us whether they did a good job or not in increasing the
capacity of women's and human rights organizations to carry out this education, human rights
education to advance gender equality.
So we've got two points there.
First of all, the perceived capacity, measuring perceived capacity is not a valid measure
of technical capacity.
And if they truly want to test, if a good job has been done in training these women's
and human rights organizations, we need a comparison group.
Thank you.
Sure.
Thank you, Yvonne.
Yeah, that's right.
Even if we say, okay, look, we'll let you do perceived capacity.
You've got to show that the project has improved or increased that perceived capacity, even
though it's self-reporting bias and they're all going to say, oh yeah, much better thanks
to Equitas.
But maybe there's another group of women and human rights organizations out there that
also are experiencing increased capacity to carry out human rights education to advance
gender equality in the country.
And they've had nothing to do with the project.
So yes, need a comparison group if you want to show that your project of $18 million actually
was involved and responsible for improving, increasing that perceived capacity, even though
we know it's not a valid measure of the ability to carry out human rights education.
Right.
So the next outcome is increased opportunities for women and or girls to participate in decision-making
process within local structures in target countries.
Here the outcome indicator measure is level of confidence on a four-point scale of women
and or girls in their ability to participate in decision-making process within local structures.
Again, we have a self-reporting bias in their ability to participate.
That's one issue.
And the other issue is maybe Equitas is going to say, look, we're not necessarily interested
in just increasing technical ability to abdicate.
We also want to increase the confidence levels of women or girls in their ability or just
their confidence to speak up and participate.
And I think that's a fair point.
Problem is they don't have a comparison group.
So they can't even show if their project increased perceived levels of confidence reported by
these women or girls.
So they can't have it both ways.
If they don't want to, if they want to measure technically increased opportunities, for example,
on a decision-making body, do they really have the power to make decisions, which they're
not doing here?
Or if they want to stick with level of confidence, then again, they have to show their project
is doing a better job of increasing levels of confidence than women and girls in the
country that are not in the project.
So next outcome is increased capacity of intermediaries that we mentioned earlier, women and girls,
to mobilize communities to advance gender equality in target countries.
And you're going to be looking at an outcome there, Yvonne.
Yeah, the outcome indicator we're going to be focusing on here is the level of confidence
on a four-point scale, I believe, of intermediaries, men and women, in their ability to mobilize
communities to advance gender equality in target countries.
Beautiful.
We understand that the effort, the resources, the trainings that have involved these women
and men has been targeted at improving their capacity to mobilize their communities, you
know, to advance gender equality.
Beautiful.
However, this particular indicator is suffering from self-reporting bias again.
Here we're talking about perceived capacity.
If you ask people, did your level of confidence increase after our training?
They could say yes, but that's really subjective.
That's really subjective.
And sadly, that might not be a valid measure of technical ability.
It might not be.
Right.
David?
No, I was just going to add, if you read the last part of the outcome indicator in their
ability to mobilize communities to advance gender equality in target countries, ability
to mobilize communities, you could also, as an alternative, just look in the future to
see how many communities they mobilize.
And you could have some sort of crude definition of what do you mean by a community being mobilized?
Maybe a couple of meetings, a criteria that they have to meet.
That's another way you could measure whether they've actually done it.
Exactly.
Forget about asking them if they feel they have the ability.
Go and find out.
Yes, I agree with you, David.
Find out how many communities they actually mobilized and then compare that to another
area in the country where the project is not operating and see how many communities they
mobilized, and that would be a way of showing that communities are actually being mobilized.
But they haven't done that here.
Another way of looking at this outcome indicator is to just look at whether these women and
men actually mobilize these communities, and that would show that they do have the ability
to mobilize, and there could be some minimum criteria, how many communities, and exactly
what is a minimum criteria to reach to say, da, da, da, da, we've mobilized this community.
And then they could compare that with a group of men and women in another part of the country
that are not even involved in the project and how they have mobilized their communities.
And you would expect in this project to be a higher level.
Or if you don't want to compare, just look at the number of communities that have been
mobilized over time for the target men and women in the project.
That would be another way of showing that our project has indeed succeeded.
Either way, but this asking how you feel is unacceptable.
Yes, because that's exactly, that's so, so, so subjective.
And David, I could have, my confidence level could be improved about something, but that
does not actually translate to me actually doing it.
Do you understand?
Because there are other contextual and environmental and socio-economic factors that actually play
into the actual decision to mobilize communities to advance gender equality.
So it's not just about, you know, we've done the training, we've trained 500 people,
we expect them to do this because your confidence levels have improved.
I don't think it's such a linear equation.
Yeah.
That's a good point.
And it raises an issue for those outcome harvesters out there, still trying to get one on the
show to talk about the fact that in outcome harvesting, you could go to the end, see how
many communities have been mobilized, and then work your way backwards to see if the
mobilization of those communities was actually due to the project.
Absolutely.
And that's what I think.
I'm not an expert on this, but outcome harvesting, I think when I read the description of that
method, that's what they try to do because they argue, you know, the whole world is complex
and you can't do a linear connection, which is true.
But there's not even an attempt here, as you can see.
And some of the other projects we've talked about on the podcast, they do use most significant
change outcome harvesting, but so far I haven't got them on the show.
So I'm quite excited.
Hopefully someday we'll get somebody on the show that can explain it and actually show
that in their performance measurement framework, we can show that our outputs are achieving
the outcomes.
But right now we're not there yet.
Good point.
Yeah, yeah, absolutely.
Yeah.
Go ahead.
Yeah.
The next one is increased efforts by select women's organizations to influence duty bearers
in fulfilling their human rights obligations related to gender equality in target countries.
It's the same outcome I think we read before.
So I'm talking about it again here and the outcome indicator is type of efforts made
by select women's organizations to influence duty bearers in fulfilling their human rights
obligations related to gender equality in target countries.
And again, it's like type of opportunities I talked about earlier, type of efforts.
True.
It's in the PMF, if you look at it, it says duty bearers who receive these efforts going
from none up to frequent efforts is not good enough.
That's what I point out here.
Duty bearers are interviewed, which is good.
They not the women organizations who report their frequency effort should be asked not
how many times the women came and engaged with them, but something like this.
Do you believe those efforts were a sufficient quantity and quality to advance gender equality?
So they should get right at it, go right to the chase, ask the duty bearers, we've got
these women coming at you from this project, cost to the taxpayer of 18 million Canadian
dollars.
Have they done their homework?
Do they look like they're ready?
Do they know what they're talking about when they're advancing gender equality, when they
engage with you?
Forget about how many times they show up.
What's the quality like?
The irony here is they interviewed the duty bearers in the performance measurement framework,
but they asked them the wrong question.
They should really probe, are you really doing a good job here?
And they don't.
Now, to be fair to them, maybe it's buried in their measurement tool somewhere, but I
don't see it in the performance measurement framework.
All we see is the scale, none up to frequent efforts.
And that's clearly not a good measure of their ability to influence these duty bearers.
Yes, yes.
We're supposed to be checking for the effectiveness, their effectiveness in influencing the duty
bearers, not necessarily the frequency.
And the irony is they're getting there.
They've done what we've recommended in earlier indicators.
They've said, forget the women, we're going to go straight to the duty bearers because
that's where the impact is supposed to happen.
We're going to ask them how well they're doing.
But in this question, they just asked them how many times they knocked on the door.
So yeah, it's ironic because they're kind of in the right direction, but they're not
asking the obvious question, you know?
So they're getting there.
Next outcome.
I, sorry, going back to, sorry, David.
That's why it would have really been interesting to have a representative from Equitas here
because maybe they could have provided a bit more exposure to their rationale behind just
stating things like this, because it's possible in your brainstorming sessions, they might
have thought, oh, we can't do this, but we can do this.
Maybe they've thought about what we've just said, but canceled it out.
And then they just put this in.
Do you understand?
It would have been really exciting to hear from them to know why they stated this indicators
as they are.
Yeah, absolutely.
But go ahead.
Yes, I agree.
And I think I've alluded it to in my trailer.
Yeah, I think I have, which is one of the problems is Global Affairs Canada requires
this performance measurement framework before they get any funding, but they're not following
their own guidelines.
They could say, look, these outcome indicators are not valid measures of your outcomes.
What's going on here?
And that's one thing that is an issue here.
And they may have another document buried somewhere that explains why they went ahead
with this outcome indicator.
But based on the performance measurement framework, it's still an issue, but you're right.
It would be good to have a representative on to provide more detail about why they pick
these outcome indicators.
And sometimes it's an afterthought just to get the funding.
I don't know.
That may be an unfair statement, but that's the way it's happening these days.
So I think we're on enhanced collaboration again among women and human rights organizations
to advance human rights in their countries, and you have an outcome indicator for there.
Yes, and the indicator in this case is the degree of collaboration on a four-point scale
among women and human rights organizations to advance human rights in their countries.
Beautiful.
However, this still points towards a self-reporting bias because they need to measure externally
and not ask the organization to report the frequency of their collaborations.
We need to also measure more than just baseline and endline.
So I feel that this is a good indicator.
I don't think it's bad in and of itself, but however, the measurements for this indicator
should have a broader scope than what they presented.
Yes, and most important, well, I think importantly, again, they shouldn't be asking the women's
and human rights organizations to report on their own efforts on how well they are collaborating.
It's just a huge bias.
They should have an external third-party monitoring going in and going in and using their own
criteria figuring out if over time, here we've got just a baseline and an endline, are they
actually increasing their degree of collaboration between themselves because it is an important
activity, for lack of a better word, or outcome that you want to increase collaboration to
get this critical mass so that you can advance human rights in the country.
But don't ask them.
Go out and do it independently and show that the project is actually in some way through
networking or whatever, because there's a whole list.
If you read part one of this podcast, describes all the, there's 10 target groups, all the
activities that they're doing is in terms of outputs to achieve these outcomes.
So yes, it has to be an external measure because it just defeats the whole purpose if they
just ask themselves.
Yes.
Right.
Yes, I agree with you.
Okay.
So the next one is increased leadership of women and men, community mobilization activities
that advance gender equality in target countries.
And the outcome indicator for that outcome is, quote, percent of total women and men
trained demonstrating leadership in community mobilization activities that advance gender
equality in target countries.
This is just attendance.
Everybody goes home.
They say trained demonstrating leadership.
So that implies that during the training, they demonstrated leadership.
So how do they measure leadership skill?
It's not clear in the performance measurement framework.
So it looks like it's just attendance.
I don't see in the performance measurement framework, again, the following measurement
tool was measured, used to show that X percent actually showed leadership after the training
that they didn't have before in terms of leadership skill, for example.
Yeah.
I don't see that in the performance measurement framework.
Even if we did, and I missed it, they're not using a comparison group.
So we can't show that the training was directly responsible for an increase in leadership
skills.
So that's the problem there.
They only measure baseline and end line.
That suggests only at the beginning of the project and at the end of the project, which
means in the middle, they could easily, even if they have a proper measuring instrument,
they could easily teach themselves these leadership skills that have nothing to do with the training
on leadership.
See what I'm getting at?
Okay.
Yeah.
So even if they have a measurement tool, I can't find it in the PMF, but even if they
have a measurement tool that measures their level of demonstrating leadership and it's
gone up and the percents are going up, they only do it twice, once at the beginning of
the project and once at the end, which it's a four-year project.
So there's just no way they can claim that the training has increased the leadership
skill levels, even if they claim the percent of total women and men trained demonstrating
leadership has gone up from the baseline to the end line, right?
Just two measures is not good enough.
You have to measure frequently along the scale to show that the project training is responsible.
Like I say in my trailer, after the first training, it could go down the road and figure
out how to do this leadership skills on their own with somebody else.
And then they come back four years later, hello, percent goes up.
Yeah.
They have nothing to do with the project training.
Yeah, but David, in a little bit to their defense, in the PMF, they specified for specific
countries and now they're taking the example of Tanzania, they said they listed four specific
activities, awareness raising campaigns, capacity building activities, engagement with duty
bearers, support to service providers working with victims of gender-based violence.
So I guess maybe they're trying to check amongst this cohort of people, of men and women that
we trained at this specific point in time, how many of them from each cohort led awareness
raising campaign programs for Veterans warmth day leader and community.
That's a little off the top of my head.