I'm your host David Wand and welcome to episode six, part one.
In this episode six, part one, I'll be introducing you to a new
international development organization and in particular, one of their projects.
And giving you a summary of the issues with that performance measurement
framework, along with a description of the services that the project is delivering.
So that new international development organization is colleges and institutes Canada.
You can learn more about this international development organization at their website,
www.collegesinstitutes.ca.
The project is entitled, A Thousand Women.
I am woman, I exist, I participate.
The cost of this project to the Canadian taxpayer is $18 million.
And this project services are being delivered in Senegal, in West Africa.
I've acquired the performance measurement framework for this project through an
access to information request, because currently it is not available to the public.
Although I am advocating for these performance measurement frameworks to be
put on the government of Canada project browser website.
Before I describe the project in detail as to what the services are they deliver, I
will come to the conclusion that of the nine outcomes in this project, there are 17
outcome indicators and only one of those outcome indicators properly measures
whether or not the outcome has been achieved.
Therefore, I have led to conclude that this performance measurement framework in its
current design is unable to support any claim by colleges and institutes Canada
that this project is achieving its outcome.
So briefly, before I get into the description of the project, if you wish to receive the
performance measurement framework for this project, as well as the summary of the
outcome indicators, all 17 and how each one of them fails to properly measure the
outcome for the project, you can email me at evaluatecanadaaid at gmail.com.
And I'll make sure to put that information into the notes that are associated with
this podcast, part one, episode six.
So for this project of $18 million, there are 11 target groups that receive a variety
of services from this project.
So let me go through them with you.
The first target group are male leaders who receive training in leadership.
The second target group are vulnerable women and they are mentored by women leaders.
And they're also trained by women leaders on leadership.
And they also have partnerships established and support measures to support their
employment, integration and business development.
They also receive training on female entrepreneurship.
The third target group is male leaders and women leaders who are also trained on
leadership.
The fourth target group is communities.
They receive campaigns that promote, I quote, gender equality.
The fifth target group are associations of entrepreneurs and business people.
For this target group of associations and entrepreneurs and business people, they are
promoting to this target group, quote, the participation of women.
The sixth target group, they refer to as target entrepreneurs.
They refer to as target audiences.
To these audiences, awareness campaigns are delivered to, quote, raise awareness of the
importance of women's participation in all decision making bodies.
The seventh target group are regions.
And in these regions, advocacy actions, quote, to promote women as full partners in the
development of communities are reached.
Target group eight are the actual project staff and they receive training to support
the women target groups in accessing the project services.
That is access to training on leadership, entrepreneurship and integration into
employment.
The ninth target group are civil society organizations, women's groups and economic
interest groups.
They receive a consultation mechanism between these three groups that is put in place
and is functional.
The 10th target group are government partners where they also receive a consultation
mechanism between these government partners that is put in place and is functional.
And finally, the 11th target group are teachers and directors of training centers.
These teachers are trained on the mechanism for validating acquired experience.
They're also trained on the, and I quote, gender approach.
So that gives you an idea of the description of the services that are delivered under this
project, as well as the target groups that receive these services.
And as I mentioned earlier, when we reviewed the 17 outcome indicators, only one of those
17 outcome indicators was designed in such a way that we could claim that the outcome
indicator for the outcome was properly measuring that outcome.
So for part two, I'll be inviting an evaluation expert to attend to go through those
outcome indicators, at least nine of them.
So we make sure we cover all nine outcomes and critique each outcome indicator to explain
in detail how it fails to properly measure the outcome.
We will also be inviting a representative from colleges and institutes Canada to attend
part two of the podcast to respond to our critique.
And just to reiterate, there are nine outcomes for this project from the performance
measurement framework, and there are 17 outcome indicators that attempt to measure the
achievement of those nine outcomes.
Before I finish, I'm going to look just at three outcome indicators and explain why they
fail to properly measure the outcomes.
So one of the outcomes in the performance measurement framework is, and I quote,
strengthening the empowerment of women among the most vulnerable across Senegal.
And they use what's called the female empowerment framework.
What is fantastic about this measurement is they actually do use a comparison group
that is women who are not in this project receiving that spectrum of services that I
described to you earlier.
And they're also measuring their levels of female empowerment before the project starts,
as well as after the project starts.
However, there are two problems with the measurement of women.
Even though they're using a comparison group, which is a good sign.
First of all, they're only measuring it once per year.
So this means, as I mentioned in my trailer, it would be very easy for the project women
to achieve their outcomes by themselves in between the trainings that they receive in
the project.
Now, you could compare that with the comparison group.
But it would be better if they did it more frequently.
But the second reason there's a problem with this indicator, even though they're using
a comparison group, is the female empowerment index looks at six dimensions.
They are one, violence against women, two, employment, three, education, four,
reproductive health care, five, decision making, and six, access to contraceptives.
Now, for this project, there's clearly a focus on employment and decision making.
The problem is, when you look at the project services related to employment for these women
and the services related to improving their decision making, there are two indicators
in the performance measurement framework.
Related to these two areas, the first one is percent of women in employment or self-employment
after training.
The problem here is there's no comparison group there for that service being delivered,
the training.
Also, there's no measure of their incomes after the training, which would be a better
measure of whether or not, even if they do get employed, are they making any money?
Are they making any money from that employment?
And is it due to the project training that they receive, for example, in entrepreneurial
skills or how to set up a business?
So they didn't use a comparison group there.
So we need to know for the female empowerment index, did they also ask that comparison group
for the project women, by the way, you attended some training on entrepreneurship.
Did you earn any income after you received the training?
It's not clear if they actually asked them that because we don't know if the same two
comparison groups, that is the project women and the non-project women for the female
empowerment index, if they're also using that same group of women for those two groups for
all the other parts of the project, such as training them in entrepreneurship.
We hope they are, but we don't know.
It's not clear because in the performance measurement framework, it doesn't show a
comparison group on this indicator of percent of women in employment, self-employment after
the project training.
The other indicator that they use that's related to decision making is the percent of women
targeted by the project who participate in decision making bodies in their communities.
The problem here is it is only measured once per year.
So again, these women could easily increase their participation by themselves without
the project training.
There also is no comparison group at the time that they're receiving training in decision
making.
A better measure of whether or not they're participating in decision making is not to
ask the women themselves because they're self-reporting bias.
The project women, of course, are going to say, yes, social desirability bias.
Yes, I have increased my participation in the community in decision making bodies.
A better measure would be to see if these women have increased their membership on decision
making bodies, either municipal, elected, or state.
Either municipal, elected councils, or school boards, or whatever they consider to be a
decision making body.
Have these project women actually become members in positions of power where they can make
decisions.
That would be a better objective measure of whether or not they've increased their
participation on decision making bodies.
So that's the problem there.
Even though the female empowerment index, they're using comparison groups at the top,
at the end of the project.
We don't know if all the trainings they received during the project are making a significant
difference compared to the non-project women because they were not using comparison groups.
Now, hopefully, College and Institutes Canada will attend part two of the podcast and will
explain perhaps, maybe when we were using the female empowerment index, we were also
using these other questions on these other indicators at the same time.
And then we could show that even when the project was delivering these services, the
women were also reporting on their incomes, their employment levels, also to show that
their employment was going up and their ability to participate in decision making was also
going up compared to the non-project women.
So then now that you have a brief description of the project, as well as my conclusion that
the outcome indicators, at least three of them that I've talked to you about there,
are not properly designed to show that the outcomes were actually achieved for the project.
Now, we're going to send this part one to College and Institutes Canada to invite them
to attend part two and there will also be an evaluation expert from Sweden who will
be attending the podcast to take turns with me in explaining to College and Institutes Canada
why we believe the outcome indicators are flawed, but more importantly, we're also
going to suggest how these indicators could be improved so that the overall evaluation
of the project could be improved.
Thank you for listening and stay tuned for episode six, part two, featuring College and
Institutes Canada and the flaws in their performance measurement framework and how we can improve
those indicators leading to a better evaluation of the project.