Improving Development Evaluation Podcast.
I'm your host David Wand and welcome to episode seven,
part one, where we introduce you to another new
international development organization.
And that international development organization
for episode seven is Equitas, E-Q-U-I-T-A-S.
Their headquarters is in Montreal, Canada,
and you can learn more about Equitas at www.equitas.org.
The name of the project that they have been provided
with funding to the tune of $18,280,395
from the taxpayers of Canada is called
Achieving Equality Through Human Rights Education.
And they're delivering this project in Burkina Faso,
Haiti, Kenya, Senegal, and Tanzania.
But they're also spending 35% of that 18 million
in regions outside of those five countries,
including north of Sahara in Africa and south of Sahara,
as well as about 20% out of that 35%.
And then percent in Asia and 4% in Americas.
In this project, there were 10 outcomes.
And for each outcome in the performance
measurement framework, they had two outcome indicators
for a total of 20 outcome indicators,
measuring whether or not they achieved those 10 outcomes.
And as I've mentioned before, if any of you are interested
in receiving the performance measurement framework
for this project or any other project
that I've covered so far in the podcast,
as well as the Excel summary of the outcome indicators
for the project, feel free to email me
at evaluatecanadaaid at gmail.com.
I'll put that in the episode notes.
So before we begin a brief description of the project
in terms of the services they deliver,
I'll jump to the chase here, cut to the chase
and tell you that none of the 20 outcome indicators
adequately measure the outcomes.
So Equitas cannot make the claim that they're achieving
the outcomes in their performance measurement framework
simply because the way they're going about measuring them
is inadequate.
So with that summary, we're gonna move on
to a brief description of the services that this project
entitled Achieving Equality Through Human Rights Education,
what services they're delivering.
They have approximately nine target groups in the project.
So starting with the first target group,
women's organizations, they receive quote,
support to participate in events on advancing
gender equality, the problem there is we don't know
what they mean by support, there's no clear description
of what they mean by support.
Women's organizations also receive training
on engaging duty bearers to fulfill their human rights
obligations to advance gender equality.
Women's organizations also receive support
in their continued engagement with duty bearers
on the advancement of gender equality,
but again, it's not clear in their performance measurement
framework what exactly they mean by support.
Women's organizations also develop action plans
to advance gender equality.
They also participate in a number of joint
awareness activities and they also share globally
good practices in human rights based approach
in gender equality, they share those globally
and they also do context analysis related
to gender equality.
The second target group are called duty bearers
and they are interviewed when women's organizations
contact them on their advancement of gender equality.
They also develop action plans to advance gender equality.
They also receive training on integrating
a human rights based approach into their work
to advance gender equality.
Local structures are also supported to become
more inclusive of women and while that is going on,
the duty bearers are interviewed.
Also duty bearers attend community forums
that are promoting gender equality
and women's participation.
They're also involved in promoting gender equality
through community actions and they also are involved
in the context analysis on gender equality.
A third target group are human rights organizations
and they participate in a number of joint awareness
raising activities for advancing gender equality.
They also are sharing globally good practices
in human rights based approach in gender equality.
A fourth target group are civil society organizations
and they are supported by the project
to participate in alliances to advance gender equality
but again, we don't know exactly what that support entails
and in a good performance framework,
measurement framework for outputs which these are,
you would think they would quantify this
by figuring out what they mean by support
so that it could be better monitored.
A fifth target group are partners,
beneficiaries and intermediaries.
They're also involved in the context analysis
on gender equality.
They're also involved in sharing globally good practices
in human rights based approach gender equality
and they're also participating in the community forums.
A sixth target group are participants
and these participants participate in two training sessions,
one at the regional level held in East and West Africa
on human rights training but also taxpayers be aware,
there's one being held in Canada
on training on human rights
and according to the performance measurement framework,
461 people are gonna show up
and if you do this math, that's about 1500 bucks
for an airfare, I mean Haiti's probably a bit cheaper
but return airfare from Africa to Canada
is about 1500 Canadian, multiply that by 461
and that lovely little human rights training session
in Canada is costing you $691,500 Canadian dollars.
That's quite an expensive output.
Number seven target group, leaders of local structures.
They also are supported to become more inclusive of women.
Problem is again, they don't provide detail exactly
of what they mean by support
and if you look at the indicators,
it's not clear how many sessions
or what they're doing there.
The eighth target group are the partners in the countries,
the developing countries
and they receive organizational capacity building,
training from Equitas
and finally target group number nine are coaches
and they are trained on community mobilization
and human rights education to coach local organizations.
I think they mean the local structures
but again, it's not clear.
The funny thing about this is even though the coaches
are trained on this, after the training,
everybody goes home.
There's no indication that the coaches are measured
on whether or not they've learned
what they were supposed to learn in the training.
So that's a brief description of the services
that the project is delivering.
Now I thought I would just give you just two examples
because in part two, we're gonna go through
all 20 outcome indicators with another evaluation expert
and we're also going to invite Equitas to attend part two
to respond to our critique
because they may have some valid explanations
as to our critique.
So let's start with the outcome increased commitment
of select women's organizations to engage duty bearers
in dialogue on advancing gender equality in target countries.
One of the indicators for that outcome is
number of plans implemented by select women's organizations
to engage duty bearers in dialogue
on advancing gender equality in target countries.
Now the problem with that indicator is,
and they even mentioned this in the performance
measurement framework for another indicator,
that they actually are concerned about quality criteria
but they don't put it in this indicator.
So the number of plans is not adequate.
I mean the plans could be garbage, right?
You need to have some sort of minimum quality criteria
where you can say that these plans actually show
that the women actually indeed are committed
and they've made effort to produce high quality action plans
to advocate for gender equality, right?
Even better, a better indicator would be measure the percent
of plans that achieve quality criteria
to engage effectively.
The other thing that's interesting,
and you wouldn't know this unless you saw
the performance measurement framework,
is that in this outcome and in this indicator,
they also interview the other side,
that is the duty bearers that these women organizations
go to to engage in dialogue on advancing gender equality.
So I would say forget the women, don't interview them,
don't measure them, or if you're gonna measure them,
measure the quality of their plans,
go straight to the duty bearers and ask them,
not the women's organizations,
those women organizations that contacted you,
was their engagement with you of sufficient quality
to advance gender equality?
Do you think they knew what they were doing?
What they were talking about?
Did they do their research, et cetera, et cetera?
You could go to the duty bearers instead.
Now, the interesting thing is they interview
the duty bearers, but we don't know if that is of any use,
because the two indicators here for this outcome,
they both talk about the women's organizations.
They don't talk about the percentage of duty bearers,
for example, who report that the women who engage with them
in dialogue on promoting gender equality
were of sufficient quality in their engagement.
Neither of the outcome indicators even mentions that,
that they're gonna measure their duty bearers,
but it does say in the PMF
that they're gonna interview them.
So it makes you wonder, why are they interviewing them
if they're not included in either of the indicators
that they're using to measure the outcome?
So that's one example where the outcome indicator is flawed.
It doesn't adequately measure any increased commitment.
And even if it did, it would be better to go
to the duty bearers as a more objective measure,
because you may get some self-reporting bias
from the women organizations.
So let's go to the next one.
The next outcome statement I'm gonna look at
is increased capacity of select women's organizations
to engage duty bearers in better fulfilling
their human rights obligations to advance gender equality
in target countries.
And the indicator there is level of perceived capacity
on a four point scale of select women's organizations
to engage duty bearers in fulfilling
their human rights obligations to advance gender equality.
So we're talking about capacity to lobby these duty bearers.
But perceived capacity, as we've mentioned several times
on this podcast and other episodes,
it's self-reporting bias.
The women, of course, are gonna say,
sure, I'm much better in lobbying
than before this project started.
It's not a valid measure of technical capacity.
And even if it was, even if we keep it,
and their perceived levels of technical capacity
to engage with these duty bearers has gone up,
we still cannot say it was because of the project,
because there's no comparison group.
So what they should do is find other women's organizations
in those countries that are not part of the project
and ask them the same question over time.
Then, and only then can they show
that the perceived capacity has gone up more
in their project compared to other women's organizations
who answer the same question
but have nothing to do with the project.
So that's the major problem there.
So thank you for listening to part one featuring Equitas.
Stay tuned for part two,
where I'm going to send them part one here,
this episode, part one.
And also I'm going to send Equitas
the performance measurement framework,
which they already have,
as well as the Excel summary sheet
that shows for each of the 20 outcome indicators
why they are not properly designed to measure the outcomes.
And an evaluation expert will also be exchanging,
taking turns, going through the outcome indicators,
explaining why we have issues with their outcome indicators.
But we hope that Equitas will accept our invitation
and provide an opportunity to respond to our critique.
Thank you for listening and stay tuned
for episode seven, part two featuring Equitas.