Welcome to the Improving Development Evaluation Podcast.
I'm your host David Wand and welcome to season 1, episode 1, part 1.
In this episode we're going to introduce you to an international development organization.
That international development organization is Care Canada.
You can learn more about Care Canada if you go to their website at www.care.ca.
Their headquarters is located in Ottawa, Canada.
The project we're going to be talking about that Care Canada has been funded by the government
of Canada to deliver is called Advancing Women's Economic Empowerment.
This project is being delivered in Vietnam and the cost of this project to the Canadian
taxpayer is $3,400,000.
And in this part 1 I'm going to give you a brief description of the project services
that are being delivered and a critique of Care Canada's performance measurement framework
that they have developed to measure the project's performance.
That is, the achievement of the project's outcomes using the outcome indicators in that
performance measurement framework.
But before I describe the project services that are being delivered for this project
by Care Canada and their partners in Vietnam, I'm going to go to my summary and conclude
that in this performance measurement framework for this project there are 19 outcome indicators
that are measuring 9 outcomes.
My review of all 19 outcome indicators has led me to conclude that all of these indicators
are not measuring the outcomes.
So the organization, Care Canada, cannot make the claim that it's achieving its project
outcomes with the services they're delivering in the project simply because their outcome
indicators are not properly measuring the outcomes for this project.
As you probably know from the trailer for this podcast, those 19 outcome indicators
fall into one of five problem areas.
In this project, 11 of them, those outcome indicators, fall into the how do you feel
self-reporting problem where people are asked how they feel about achieving the outcome
rather than being objectively measured on whether they achieved that outcome.
Another seven of the outcome indicators fall into problem area number five, which is there's
no comparison group of people outside of the project that are also measured on the same
outcome, which means even if the people in the project are achieving the outcomes, we
don't know if they're achieving them any better than another group of people that could
also achieve the outcome outside of the project.
And finally, we have one indicator that is falling into problem number four, where it's
simply not being measured frequently enough.
If you wish to receive Care Canada's performance measurement framework for this project and
my summary of the outcome indicators for this project, send me an email to evaluatecanadaaid
at gmail.com and I would be happy to send those documents to you.
I plan to post all of these documents for each episode on a website that is currently
being developed.
So now that I've concluded that the PMF is flawed, let's get into more detail about the
project services.
So there are four target groups that this project delivers services to.
The first target group are poor, rural, ethnic minority women, and the services being delivered
to them are training on negotiation and household decision making, action plans being developed
for them to promote their economic rights and are promoted by male champions.
Dialogues are held between these women and men on household decision making and gender
based violence and unequal caregiving burden.
These dialogues are also held on the same topics between these women and communal officers.
Community events are also held, attended by women and men to promote the economic rights
of these women.
And gender based violence services that are available for these women are promoted to
these women.
And finally, a feasibility study is completed on child care services and household time
saving appliances.
And then there's a variety of trainings that are also delivered to these women, such as
training on financial literacy, savings group operations, digital record keeping, and business
plan development.
These women also receive unpaid care time saving options.
And also studies and policy briefs are produced on the benefits of reducing unpaid work for
these women.
Training is also delivered to these women on climate appropriate agricultural practices
and on high quality tea production and safe migration.
A second target group in this project delivers services.
And that target group are civil society organizations and media organizations.
The services they provide are they promote the economic rights for these poor rural ethnic
minority women, such as including success stories of women entrepreneurs, which are
promoted in the media.
These civil society organizations and media organizations also attend learning events
paid for by this project, where these civil society organizations and media organizations
are expected to learn on how to work with these women whose economic rights they're
promoting.
These civil society organizations and media organizations are also expected to produce
documentation.
During the promotion of these women's economic rights, this documentation is sent to policymakers.
Also round tables are organized by this project between these civil society organizations
and policymakers on how to promote the economic rights of these women.
The third target group are private sector individuals.
And they receive from this project training on women's economic rights and gender responsive
value chain approaches.
And finally, a fourth target group are what we call duty bearers or what they refer to
as village leaders and communal officers.
The services the project delivers to them is training on gender responsive programs
for women.
So that gives you an idea of the services the project delivers.
Now, there's nine outcomes in this project.
And they are outcome number one, increased entrepreneurship capacity of women led grassroots
social enterprises to pursue economic opportunities in Vietnam.
Outcome number two, increased gender responsive capacity of poor rural and economic minorities,
especially women to save and borrow money to support productive and sustainable economic
activities in Vietnam.
Outcome number three, increased gender responsive capacity of poor rural and ethnic minorities,
especially women to earn income from climate smart and sustainable on farm and off farm
livelihoods in Vietnam.
Outcome number four, increased willingness of poor rural and ethnic minority women and
men duty bearers, in particular, commune officers to address the issue of women's unequal caregiving
burden and economic rights in Vietnam.
Outcome number five, increased ability of ethnic minority women to meaningfully participate
and make decisions related to economic activities at the household level and in their livelihoods.
Outcome number six, increased capacity of civil society organizations, including women's
rights organizations, private sector companies and media organizations to promote poor rural
and ethnic minority women economic rights in Vietnam.
Outcome number seven, increased participation by poor and ethnic minority women in paid
economic activities.
Outcome number eight, enhanced promotion, economic rights for poor rural women, especially
ethnic minority women in Vietnam.
And finally, outcome number nine, improved economic well-being for poor rural women,
especially ethnic minority women in Vietnam.
So we can't go into all the outcome indicators, but I'm going to give you just three of those
outcome indicators as to why they fall into one of those five problem areas that we outlined
in the trailer.
So in part two of this episode, where Care Canada will be invited to respond to this
critique, we will cover all nine outcomes with at least one outcome indicator to be
discussed for each outcome.
If we have more time, great, we'll go into further outcome indicators, but we're going
to focus on making sure we cover at least one outcome indicator for each of the nine
outcomes so you get an idea of right across the entire spectrum of outcomes, the nine
of them for this project.
At the end of part two, the evaluation panel, that's me and two evaluation experts, will
be concluding as to whether Care Canada can make the claim that its project services are
achieving its project outcomes.
As host, I conclude Care Canada cannot make this claim, but I want our listeners to hear
what Care Canada has to say as well as the other evaluation experts.
So let's look at the outcome increased capacity of civil society organizations, including
women's rights organizations, private sector companies, and media organizations to promote
poor, rural, and ethnic minority women's economic rights in Vietnam.
So that's the outcome.
One of the indicators to measure this increased capacity to promote is number of community,
civil society organizations, media organizations, and network members, private sector companies,
who report an improvement in their perception, opinion related to ethnic minority women's
economic rights.
You probably can tell already from this indicator that it's flawed, but it's interestingly
flawed in two ways.
First, reporting on the number of civil society organizations who report an improvement in
their perception or opinion related to ethnic minority women's economic rights, it's got
nothing to do with increased capacity to promote their rights.
They're just asking them for their opinion, but the outcome is increased capacity to promote.
And if you look at the project services, they are expected to improve their capacity to
promote because if you remember from the description of the project, they are showing up at a cost
to you, the Canadian taxpayer to learning events where they're expected to learn how
to work with the women whose economic rights they're promoting.
They're also being paid by you, the Canadian taxpayer to promote the economic rights of
these women.
And of course, they are going to say they have improved their opinion on promoting these
women's economic rights.
They're also expected to produce documentation on how they're promoting these women's economics
rights with that documentation being sent to policymakers also at a cost to the Canadian
taxpayer and are showing up to round tables where they sit with policymakers on how they're
going to promote women's economic rights.
So it's clear the project funds are being spent to increase their capacity, technical
capacity on how best to promote.
But the outcome indicator is not measuring that at all.
It's just measuring how they feel, their perception, their opinion.
We don't want their opinion.
We don't want their perception.
We want to know if they're technically capable, thanks to all that money spent showing up
at these learning events, showing up at those round tables, producing documentation that
shows they know how to promote the economic rights of these women.
And more importantly, that they've significantly increased their capacity.
So the indicators got nothing to do with the outcome.
That's the first problem.
Even if capacity is defined not as technical capacity to promote, but rather an improved
willingness to promote, they are going to report yes, because they are paid by the project
to promote.
And even you can go a step further.
Let's say we've got an objective measure that they've improved their willingness.
There could be another group of civil society organizations out there in another part of
the country that have got nothing to do with the project and their willingness to promote
has also gotten.
So we'll never know on several levels whether these organizations have increased their technical
capacity or even their willingness to promote.
Another indicator we're going to look at is for the outcome enhanced promotion of economic
rights for poor rural women, especially ethnic minority women in Vietnam.
One of the indicators for that is the average total number and proportion of daily hours
spent on unpaid domestic care work.
Now if you remember from the project services, one of the things they deliver to these women
is unpaid care time saving options.
We don't know what those options are, but we suspect and we logically expect them to
reduce the number of hours spent on unpaid domestic work for these women.
If they're given these unpaid care time saving options.
Unpaid care time saving options.
It means they don't have to spend as much time on unpaid care themselves.
Sounds good to me.
Problem is we don't have any comparison group.
What about another group of women who are not in the project in some other part of the
country in Vietnam who could say, ah, let's get together and set up a communal child care
center.
We'll all send our children there and just one of us will watch all the children together
collectively while the rest of us reduce our hours of unpaid child care.
So we also as a group perform well on the outcome indicator with our hours also going
down like the project group of women.
So there's nothing wrong with the indicator, but if Care Canada wants to claim that their
project service of delivering these time saving options is making a greater reduction in hours
spent on unpaid child care by this service, they have to compare it with another group
outside of the project.
And finally, one more indicator we're going to look at is related to the outcome improve
economic well-being for poor rural women, especially ethnic minority women in Vietnam.
One of the indicators there in the performance measurement framework is percent and number
of poor rural and ethnic minority women and men who report an improvement in their economic
well-being.
Now, we don't know exactly what Care Canada asks them in terms of economic well-being.
So to be fair, maybe at the podcast part two, they'll bring up something like a world bank
survey or something, but we just don't know.
So what I'm suggesting here is maybe they should measure their incomes or measure their
assets rather than asking them to self-report whether they feel their economic well-being
has improved.
Maybe it's too intrusive, but I would think measuring their incomes or assets over time
before, during, and after the project would be a better measure of showing that their
actual economic well-being has improved.
That world bank survey that you often hear about with the percentages of those living
on $2 a day or less comes to mind.
And again, even if their incomes or assets increase for the project women, what about
a comparison group of women not in the project and measuring their incomes or assets as well
over the same time period so you can compare the two groups.
So those are just three examples from three outcome indicators that fall into these problem
areas.
If you have any comments or suggestions for this episode, feel free to email me at evaluatecanadaaid.gmail.com.
And if there are any evaluation experts out there who'd like to be on the podcast to look
at other performance measurement frameworks and in particular the outcome indicators,
just like I have here, email me and I'd be happy to send you the remaining performance
measurement frameworks that we have lined up for season one where we don't have evaluation
experts so far.
Right now that's eight performance measurement frameworks that we're looking for experts
to comment on.
What we're going to do now is invite a Care Canada representative to attend part two of
this episode to respond to this critique here in part one, along with two evaluation experts
that will attend part two of this podcast.
Thank you for listening and stay tuned for Care Canada, part two, episode one, season one.