Welcome to the Improving Development Evaluation Podcast.
I'm your host David Wand and welcome to episode one,
part two, where we feature Care Canada
as our international development organization.
Before I introduce you to our two evaluation experts,
I'm gonna give you a brief synopsis
of what I've done with part one.
On October 19th, I sent to Care Canada by email
part one of this episode,
explaining my critique of their
performance measurement framework.
And then on October 20th,
I followed up with a phone call
to make sure they had received the email.
And then I followed up again on October 30th
with a repeat email reminding them
of the invitation to attend the podcast.
It's now November 6th.
We have not received a reply from Care Canada
regarding whether to attend
and we have not received a written response,
which was another option for them.
If they were unable to attend,
they could send a written response
to the critique of their performance measurement framework.
To be fair to Care Canada,
today it's 7 a.m. Ottawa time,
where Care Canada is located.
That's a bit early in the morning,
but they could have sent a written response,
but we haven't heard anything from them.
So now we're gonna proceed and go into more detail
about the outcome indicators and why they're flawed
in their performance measurement framework.
From Georgetown, Guyana, I have Donald Cole.
Donald Cole holds a master's of science degree
in data engineering from Edinburgh Napier University
and is a monitoring and evaluation specialist.
And we also have from Kampala, Uganda, Dr. Mark Degner,
who holds PhD in social and economic science
from Karl Franzens University in Austria.
And he is currently a technical advisor
in advocacy, communication, and documentation.
So what we're gonna do now
is we're gonna go through the outcomes.
So let's start, I'll read out the outcome,
and then Donald, you can talk about
the outcome indicator and why it's flawed,
and even if you wish to provide an alternative indicator.
So beginning with the first outcome in the project,
increased entrepreneurship capacity
of women-led grassroots social enterprises
to pursue economic opportunities in Vietnam.
That's the outcome and the outcome indicator?
Yes, the outcome indicator for that particular outcome
is number of women-led grassroots social enterprises
reported increased capacity to engage
in entrepreneurship opportunities in Vietnam.
So this particular indicator,
there's like three main points I would like to highlight
for this indicator.
The first one is the vagueness in increased capacity.
The term increased capacity is vague
and it does not really specify what aspects of capacity
are being measured.
The second point is its reliance on self-reporting.
And that data can be very biased
because self-reporting data persons tend to be very biased
in terms of how they report and provide feedback.
And the third key point on this indicator
is the lack of objective measures.
There are no objective measures for entrepreneurship capacity
or success.
In terms of improving this particular indicator,
the first thing would need to be done
is to have a clear definition, obviously,
of what capacity entails,
whether it entails business skills, access to capital,
market access, et cetera.
The second thing that would be needed to address
is the objective matrix of the indicator.
What are the objective matrix are we going to measure?
Are we measuring revenue growth, employment rates,
number of new businesses started, and so forth.
The third thing that would need to be addressed
is periodic and independent assessments.
Conduct periodic assessments.
And these assessments should be done
by independent evaluators to ensure
that they measure the growth and the sustainability
of the enterprises.
And then there's an issue with confidence versus competence.
While confidence is important,
it should be measured alongside competence,
especially when it comes to entrepreneurship skills.
That would be my brief critique of this particular indicator
and ways in which this indicator can be improved.
I didn't come up with a rewarded indicator
because I think I would probably need
a bit more information,
but those key points would definitely address the issues
with this particular indicator.
That's excellent, Donald.
Thank you.
So we'll move on to the next outcome statement
in the performance measurement framework
for this project by Care Canada delivered in Vietnam.
That's increased gender responsive capacity
of poor, rural, and ethnic minorities,
especially women, to save, borrow money,
to support productive and sustainable economic activities
in Vietnam.
The indicator I'll be looking at there is
percent and number
targeted poor, rural, and ethnic minority women and men
who report an improvement in capacity to save money.
And I have the same issue that Donald's explained
is increased gender responsive capacity
to save and borrow money.
They're depending here again on self-reporting.
So you have this self-reporting bias
or what some people call social desirability bias
where they will have a tendency to want to say,
yes, we've increased our capacity.
And another point is,
if they have the capacity to save money and borrow money,
why not just check out their bank accounts?
Why not just go right to the chase?
Some people may say that's intrusive,
but if they really wanna show
that they've been able to save and borrow money,
they should look at their savings accounts.
Their ability to borrow implies the ability
to pay back a loan on time.
So they could also use that as an alternate way
of measuring their capacity.
And also as we allude to in the trailer of this podcast,
even if their capacity is properly measured,
and it is not here, but let's assume it is,
how do we know it's the project training
that was responsible for this increase
in gender responsive capacity to save and borrow money?
There could be another group of women
in another part of Vietnam
who are also doing this just fine,
and there's no project training going on.
They figured it out all by themselves.
So that's what my points would be for that.
Now, the next indicator is 1210.1.
So that would be Donald again.
This is where we have the outcome,
increased gender responsive capacity
of poor rural and ethnic minorities,
especially women, to earn income
from climate smart and sustainable on-farm
and off-farm livelihoods in Vietnam.
Donald?
Sure, thanks, David.
This particular indicator that supports that outcome
is percent and normal.
Targeted poor rural and ethnic minority women
and men who report an improvement
in their capacity to earn income.
And this indicator is disaggregated by sex,
ethnicity, age, disability status.
Three main issues again with this indicator.
The first one, again, same issue
like the last two indicators.
It's based on self-reported improvements,
which may not reflect actual income changes.
The second issue I have with this indicator
is the frequency of measurement.
It's unclear how often this measurement is taken,
which can affect the reliability of the data being collected.
And then there's no comparison group.
It's difficult to attribute any improvements
to the project, specifically if you don't have
a comparison group.
How this indicator would need to be improved
to measure the impact project is,
the first thing that would be needed to address is,
you know, having that control group being selected.
And that control group should be matched
on key demographic and soco-economic variables
to ensure compatibility and implements physical techniques
such as propensity score matching can be used, for example,
to select a control group that mirrors the intervention group
as closely as possible.
The second thing is the frequency of the data.
Determine an optimal data collection frequency
that balances the need for the timely data
with the logistical constraints of data collection.
This may involve seasonal assessment
to account for agricultural or market cycles
that affect income.
And then there's income measurement.
A comprehensive income measurement tool
would need to capture not only the amount of income,
but also the sources and stability of that income.
And this tool should be sensitive enough
to detect changes over time
and attributable to specific project interventions.
And the fourth thing I would suggest
that would need to be considered for this indicator
for improvement is link the specific activities
to income improvement, to specify project activities,
to identify what exactly is really working.
So the next outcome is increased willingness
of poor rural and ethnic minority women and men,
duty bearers in particular, commune officers,
to address the issue of women's unequal caregiving burden
and economic rights in Vietnam.
And I think Donald, you're looking at the indicator there.
Yes, that particular indicator states
average score of ethnic minority men and women,
duty bearers in particular,
commune officers who report gender equitably attitudes.
And this is disaggregated by sex, ethnicity, age,
disability status, type of participant.
I have three main issues with this particular indicator.
The first one is the subjectivity of it.
Discard indicators relies again
on self-reported measures of attitudes.
And this can be highly, and in more cases
are very highly subjective and influenced
by social desirability biases.
The second issue I have with this indicator
is the lack of baseline and follow-up.
Without the baseline measure or regular follow-up,
it's very difficult to assess whether any changes
in attitudes are due to the project
or the activities coming out of the project.
And there's no actionable outcome.
Porting an attitude does not necessarily translate
into actionable change or improvement in gender equality.
Let me just ask you, Donald, that's correct.
But in this outcome, to be fair to Care Canada,
they just wanna make sure that the attitude
has changed in the right direction, increased willingness.
So you mentioned lack of frequency
in measuring this attitude change.
If we have a chance, we can look at the performance
measurement framework.
What exactly is the frequency they measure
this attitude change?
Does it say in the performance measurement framework?
I think they mention outcome harvesting
and it might be annual reporting.
Let's check to make sure.
I'm quite sure the frequency is not enough,
but here it is.
Yeah, you're right.
It's outcome harvesting and it's only done annually.
So that means you get an attitude change,
you get it in the right direction,
but they only measure it once a year.
So there's all sorts of different things
that could be going on that could change their attitude
that's got nothing to do with the project.
Yeah, so, yeah, that's a good point.
Yeah.
Yeah, so suggestions on how this could be improved
is definitely, first of all,
addressing the regular assessments, right?
Implement more pre-project surveys
to establish a baseline,
conduct regular follow-up surveys
to track changes over time.
They could also use a comparison group
where people that are not in the project,
that are not doing anything to any of these women
or this target group to change their attitudes,
they could also measure their attitudes
on the same variable, which is increased willingness
to particularly these commune officers
to address the issue of women's unequal caregiving burden.
And they could go to other districts in Vietnam
and ask commune officers there
where Care Canada is not working
and trying to change their attitudes.
Their attitudes could also be going in the right direction.
Right, right.
Yeah, that's a key point.
And one of the methodologies I have here to note
that would expand on the only tactical assessments
of this particular indicator is,
one, they can also implement
longitudinal study design methodology
that measures attitude over multiple time points.
And this would allow for assessments of changes
in attitudes and the sustainability
of these changes over time.
Another thing that they can do
is implement a mixed method approach
where they incorporate qualitative methods
such as focus group discussions
or in-depth interviews.
And that is complemented
with the quantitative self-reporting data.
And that would provide a bit more context
to the attitudes expressed
and the reason behind any changes
or lack of any changes thereof.
Yeah, that's right.
There's a common problem.
Our focus groups are used just to measure
the outcome indicator
rather than measuring the outcome indicator first
and then seeing that the outcome indicator
is not moving in the right direction.
And then, and only then,
should they have a focus group
where they ask the question,
hello, why did your increased willingness not go up?
And then you hold a focus group to find out.
But often focus groups are abused, misused,
and they only use them like outcome harvesting
to try to measure the outcome indicator in the first place.
Right, correct.
Okay, so let's move on to the next outcome,
which is increased ability of ethnic minority women
to meaningfully participate
and make decisions related to economic activities
at the household level and in their livelihoods.
And Mark, I think you're gonna be dealing
with 1120.1 indicator.
Well, yes, I'm reading.
Percentage and number of ethnic minority women
who have meaningfully participated
in economic decision-making at the household.
Yes, I think general increased ability participation
are difficult terms to evaluate and also to monitor
because meaningfully participation,
what does it mean in detail?
Does it mean,
that they have not the chance to participate
in the household before?
Is there, because of what?
Because of cultural aspects or social aspects.
And now if they are able to do this,
it's, we can maybe, this is for me also a question
of a qualitative approach necessary,
a question of qualitative measurement.
And general decision-making,
you have to go back and ask them,
maybe in interview situations,
what has improved your participation at the household?
Is the husband now allowing you,
or is your role in the family has been stronger than before
to explain this indicator?
And the other way is,
sometimes the stories of change can also help
to get to the point where
you get to verify
and to find out what is now the improvement of the situation.
And the second one is similar,
it's the percentage of numbers of ethnic minority women
who report participating and voicing out their concerns
about pay economic activities to local authorities
and DTPers here.
Here we have a similar situation.
There's one aspect,
it was written women who report,
they're reporting their participation
and voicing out their concern.
So it's a self-reporting or it's a reporting that happens
via a survey, I don't know.
But for me, when I look at this thing,
what local authorities are they,
what type of DTPers are these who are concerning about this?
Are they from the district, from the communities?
And how can they see improvement to the situation before?
It's a little bit tricky to use such indicators
who are, you can use your qualitative approaches
to by the end of the day,
it's to see the change from the baseline to the target.
It's sometimes not so, we will not know
it's not so visible and also hard to measure it, yeah.
Right, and if you look at the outcome,
it says we've bolded it for,
if anybody's interested in getting
the performance measurement framework,
you just email me,
it says increased ability to meaningfully participate.
Now you could argue, you can measure that quantitatively.
You could measure their ability to negotiate or whatever,
you could test them, you could do some sort of assessment
before, during, and after the training
instead of self-reporting.
So that's a really quick way of arguing
you could actually objectively measure increased ability
based on some sort of training
where they're training them how to negotiate
for better participation in the household.
And that would be good enough because the outcome is clear.
It just says increased ability.
After that, good luck.
We don't know if that's gonna lead to greater levels
of satisfaction for these women in the household,
but at least that piece, that short-term piece
in the immediate outcome of increased ability,
they could measure it.
But they're going straight to stories of change.
And to be fair to Care Canada,
of course they're doing what Mark has explained,
which is trying to find out how to measure this level
of participation in a household, which is very difficult.
It's intrusive.
So we understand stories of change may be a way of doing it,
but you've forgotten your outcome in the first place,
which is increased ability.
So they've kind of missed, in my opinion,
that first step where they could actually measure
their ability using some sort of psychometric test
of their skills to be able to negotiate
based on training they have received in the project.
And then you could also do this with another group of women
who are not even in the project
or in some other part of the country,
and that would be good enough.
They could show that ability has actually increased.
The other question is also when you talk about household,
then you have also to ask other members of the household,
the husband or children or people who live in this household
if the economic decision-making has been improved
because of the project.
So you need more voices to understand
what is really the improvement.
Even though they've made it clear in that second indicator,
they only are interested in the women,
but it's a good point is that it's a more complex dynamic
in the decision-making within the household.
But they say right in the,
at least I give them credit for specifying,
it's the ethnic minority women.
They want to increase their participation.
But you're right, it may depend on the other members
in the household, absolutely, yeah.
Okay, so we're gonna move on to the next outcome,
which is in the performance measure framework,
increased capacity of civil society organizations,
including women's rights organizations,
private sector companies and media organizations
to promote poor rural and ethnic minority women's
economic rights in Vietnam.
And the indicator, number of civil society organizations,
media organizations and network members,
private sector companies, who report an improvement
in their perception, opinion related to
ethnic minority women's economic rights.
So again, outcome is clear, increased capacity.
So that means we should be looking at their ability,
increased ability to actually advocate for these women
regarding their economic rights.
So there's a huge bias in here because these organizations
are paid for by Care Canada, by the project
to advocate for these women.
So they're expected to advocate on their behalf.
So if you ask them, have you improved your ability
or your perception or opinion related to
women's economic rights that you happen to be paid for
by the project to advocate?
Of course, they're gonna say yes.
So this is a huge bias.
And again, they missed the point of the outcome,
which is they should be just measuring
their increased capacity to advocate.
There's all sorts of projects where they train people
on how to advocate for another group,
which is what's going on here in this outcome.
So that's the major problem I see with that.
And I pointed it out, why not objectively measure
and also the other point is it's not the institutions,
it's the human beings inside of them
that we need to look at.
We need to measure their technical capacity
to advocate for others, right?
So it's not just the number, that's the real problem here.
The indicator says the number of organizations.
So you just tick a box off.
How do you tick that box off that says
this civil society organization in Vietnam
has met all our criteria of equity
of adequately and increasingly in better ways
advocating for these women?
How do we know?
So what we need to do instead is go inside that organization
and test a group of the individuals
who are expected to advocate for those women.
I think it could be duty bearers, I'm not sure.
And make sure that their technical ability
to advocate has gone up
and maybe even their attitudes have gone up.
And then the other problem of course is
even if they have gone up, there may be another group
of civil society organizations that are doing
just as good a job or even better job
at advocating for ethnic minority women's economic rights
in some other part of the country
where Care Canada is not involved at all.
So that's the problem with that indicator there.
Now we'll move on to the next outcome
which is increased participation by poor
and ethnic minority women in paid economic activities.
I think Mark you're doing this indicator here.
Yes, percentage and number targeted poor, rural
and ethnic minority women and men
who report access to and control over economic resources
desegregated by sex, ethnicity, age, disability,
status and commune.
Well, here we have again the same question
like we had before, it's about self-reporting.
Because you want to know this ethnic minority women and men
who are reporting their access and control
of our economic resources.
And it's somehow difficult to value it.
The measurement is somehow you can maybe check them
in a questionnaire to see what economic resources
they had before the project, what do they have now?
So you can see the change in the impact
but there's always some bias about self-reporting
but it's sometimes not bad.
It can also be if you do it in a way with interviews
and focus group discussions to try to get the view
of the people and to put it in numbers
and figures it's time to see the track changes
that it's a little bit not appropriate I think.
You can maybe track their incomes.
You can track their incomes against a non-project group
of women and men to see if they have really increased
their incomes before.
Exactly, yeah.
Now to be fair to Care Canada,
the outcome is increased participation
but that's difficult to measure but they've said
they're gonna claim that their project is going
to increase the levels of participation for these women
and they're gonna train them in some way
that this is going to happen through a variety
of project services that are described in part one.
But yes, it's difficult but yeah,
you could just cut to the chase and figure out
because you want to achieve ultimately increased incomes,
economic resources for the family although we can't assume
that all that money, increased income goes
into the women's pocket may still be held by the husband
and that is understandable.
That's why they talk about empowerment
of the women specifically.
So it's difficult but I think we've noted it's a challenge
for sure in measuring increased participation.
Absolutely, yeah.
Yeah, it's a general hard aspect.
Participation is generally very difficult in developing
projects so very often you use quantitative approaches
because quantitative, you can work with a questionnaire
and make a survey if you have a reliable sample
but it's not always giving you the right view.
Right, okay good.
So the next outcome is enhanced promotion
of economic rights for poor rural women,
especially ethnic minority women in Vietnam
and the indicator I'm looking at here is average total number
and proportion of daily hours spent
on unpaid domestic and care work.
Now this is an interesting one because they are trying
to have these women, part of the project actually delivers
childcare saving options and they don't describe the options
but we can assume that just by delivering that service
in the project which is very important,
it will reduce the hours that these women spend
on unpaid domestic care work just
by providing these alternate options for them to choose,
which is fine.
Two problems again, the one is even if that happens,
we need to see a comparison group where there's another group
of women who don't receive these alternate options
of other people taking care of their children for example
in this project.
In another part of the country where another group
of women get together for example and set
up their own community childcare center
where one woman says all your children can come
to my house this day and the rest
of you can do what you have to do during the day and I'll take
and they do that on a rotational basis.
So that's the first problem.
The other problem is when you look at the outcome, again,
they're looking at enhanced promotion of economic rights
for poor rural women.
So this particular indicator isn't promoting their rights
specifically so enhanced promotion is not really being
measured here directly.
So that's the only way I could look at this is saying
that is there another group out there
that are also promoting the economic rights of these women
in a way that's having the same benefits.
So there's no comparison group.
So that's the problem there with that indicator.
Now to be fair to them, they have other indicators
where they are trying to measure enhanced promotion
but we're not going to discuss that right now
because we have limited time to cover all the indicators
but the summary is there for people to look at as to why all
of the outcome indicators do fall into one of those five
problem areas that we're discussing.
So the next one, the next outcome is improve economic
well-being for poor rural women, especially ethnic minority
women in Vietnam and the indicator there is percent
and number of poor rural and ethnic minority women and men
who report an improvement in their economic well-being.
And it's the same issue we've talked about before,
which is instead of having them self-report
on their economic well-being,
why not just measure their incomes?
And more importantly, do it compared to another group
of women in another part of Vietnam
that aren't receiving all these project services
and track their incomes before, during and after the project
and see if there is a significant increase
or greater increase in incomes for the project group.
So now that we've done that,
we've covered all nine outcomes of the project
and picked at least nine outcome indicators.
Let's, I think Mark, thank you.
You've already covered the other indicator for that outcome,
but we'll just do one more.
And that is, we'll go back to increased entrepreneurship
capacity of women-led grassroots social enterprises
to pursue economic opportunities in Vietnam.
And we'll look at another indicator.
Donald's gonna look at 1230.2.
Yes, thanks David.
1230.2, this indicator states,
percent a number of ethnic minority women-led
grassroots enterprise members who report a change.
And here they're using a five-point scale
in their confidence level of their entrepreneurship capacity.
This has the same issues as the 1230.1
that we discussed,
or issues that relate to increased capacity,
how that's being defined.
Issues with self-reporting,
the reliance of it being self-reporting.
And the major issue I think that I have with this indicator
is confidence assessment.
To measure confidence and to improve on this indicator,
they should probably use a psychrometrically
sound instrument that would differentiate
between perceived self-efficacy and overconfidence.
And these often do not correlate with actual competence.
So there are some clear technical issues
with this particular indicator.
And to improve it, those issues will be need to improve.
Define what is capacity.
Is it financial, market assessment,
operational, and so forth.
Address the issue with confidence, competence,
and that would clear up the major issues
with this particular indicator.
Before we make a decision on this performance measurement
framework and its ability to support Care Canada
in its claim that its project outcomes have been achieved
by its project services,
I'm gonna go again to the last indicator
on the last outcome,
which is improve economic well-being
for poor rural women,
especially ethnic minority women in Vietnam.
They have one other indicator,
which was level of improvement of economic rights
as perceived by ethnic minority women.
And that's fine.
It's a good way to look at,
measure their perception levels
to see if their perception levels
of how they're doing economically,
it's obviously subjective,
but there's nothing wrong with that.
It's not a bad indicator.
But again, the problem is there's no comparison group.
So we need to get another group of women
outside of the project,
some other part of the country in Vietnam,
and measure also at the same time
their perceived levels of economic well-being
where the project's not operating.
And if this project wants to claim
that they're more successful, then they have to show
that their levels of perceived economic well-being
have increased at a greater rate
or are significantly greater than the non-project group.
So now that that's done,
what do you think guys,
after we've reviewed this performance measurement framework,
do you think that Care Canada can make the claim
that its project services are achieving its project outcomes
at an expense of $3.4 million to the Canadian taxpayer?
Donald?
Based on the analysis that we just provided
and the limitations that we identified with the indicators,
I think it would be extremely difficult for us
to conclusively claim, or Care Canada
to conclusively claim that the program
has achieved its objectives.
And those are based on key issues we discussed,
self-reporting bias, lack of objective measures,
no baseline or control groups,
measurement frequency issues, vague definitions,
and no link to specific activities at all.
With all these issues, it's very difficult
to conclusively claim that the program
has achieved its objectives.
Good, Mark?
Well, I think in general, I have no...
The issue is that for me,
it is not stopping at economic rights.
You also consider social and cultural rights,
because in the end, you want to include them,
the rights to have adequate food, adequate housing,
adequate education for minority groups.
And to measure that in this template,
to measure that in this type of project,
it seems for me complicated,
and they should be interlinked also,
like I said before, the economic, social,
and cultural aspects should be interlinked.
And one point is that if you want to see improvement,
you have also to talk with the authorities
without duty bearers, if they have measures,
if they have laws or policies that can improve
the level of ethnic minorities in this district
or in these places, yeah.
Yeah, that's great.
I think I mentioned in part one,
that issue where the duty bearers
are supposed to advocate for the women,
but the bias is they're asked how they believe themselves
are doing in that role,
and it's not objectively looked at by a third party.
And you need to look at that, I think.
So that's good, okay.
So we basically summarize that CARE Canada
needs to do some work on its outcome indicators
for starters, along with a lot of other issues.
So what we're gonna do now is send this part two
to the Minister for International Development in Canada,
as well as the shadow critics for the Conservative Party
and the New Democratic Party,
and we'll also copy CARE Canada.
And for you listeners out there,
send me an email at evaluatecanadaaid at gmail.com
if you want a copy of the performance measurement framework
and a copy of the Excel summary
of all the outcome indicators and why they fall
into one of these five problem areas.
We're also looking for evaluation experts
who may wanna be interested in doing similar work
to what Donald and Mark have done today.
Send me an email, and I'd be happy to send you
other performance measurement frameworks
that I'm looking at for future podcast episodes.
And if you have any comments or suggestions
on this particular episode regarding CARE Canada
and our review of their performance measurement framework,
send me an email and I'd be happy to respond.
And at some point, we'll be putting
the performance measurement frameworks
and the summaries of their outcome indicators on a website.
And finally, for you listeners, Canadian taxpayers,
you may wanna email CARE Canada at info at care.ca
and simply ask them are they gonna be improving
their outcome indicators, or you could also ask them
if they're gonna put all their other project
performance measurement frameworks on their website
so that we can all have a look,
because right now they're not available to the public.
So thank you for listening and stay tuned
for an upcoming new international development organization,
$37 million project, so it's getting a little heavier.
Thank you, Donald, and thank you, Mark.
Thanks, Evan.
You're welcome, David.
All right, thanks and stay tuned.
Bye for now.