Welcome to the Improving Development Evaluation Podcast.
I'm your host David Wand and welcome to episode six,
part two, where we continue our discussion
on colleges and institutes Canada
and their $18 million project funded
by the Canadian taxpayer that is operating in Senegal
and is entitled, A Thousand Women, I Am Woman,
I Exist, I Participate.
And just to give you a brief recap,
I contacted the colleges and institutes Canada
on March 9th to invite them to this podcast
to respond to our critique
of their performance measurement framework.
They did not reply.
I then followed up on March 18th,
along with a phone call and left a message.
We still have not received a reply.
Today is now March 28th
and they have decided obviously
not to attend this part two of the podcast.
But fortunately, I do have with me an evaluation expert
that I will introduce to you shortly.
This $18 million project,
looking at the performance measurement framework,
had nine outcomes and used 17 outcome indicators
to measure whether or not those outcomes
had been achieved for the project.
And I led to the conclusion that colleges
and institutes Canada could not make the claim
that its project was achieving its outcomes
simply because the outcome indicators
were not properly measuring the achievement of the outcomes
with only one out of the 17 outcome indicators
properly measuring whether the outcome was achieved.
So with that being said,
I'd like to introduce to you Dr. Jenny Yau Jorgensen,
holds a PhD in system safety from Lund University
and she is in Sweden and has been kind enough
to participate in part two of this podcast.
And she is also an evaluation practitioner.
Welcome to the podcast, Jenny.
Thank you for having me, David.
Good, great.
So what we're going to do
is we're gonna go through each of the nine outcomes
and select some indicators.
And we're going to in detail,
describe why they are flawed
in properly measuring the outcome.
And we will start with a few outcomes
that are actually in the performance measurement
framework as outputs,
but we put them as outcomes
because they expect outcomes to be achieved.
So starting with the first one,
the outcome was quote leadership training
led by male leaders in the communities.
And you would expect something to be learned
from that training.
And the outcome indicator that they stated was
and I quote,
number of people benefiting from leadership training
by male leaders.
And the problem with that indicator
is that it's just assuming that people show up,
they take attendance, number of people,
but we don't know if they actually benefited
from the leadership training.
It's just taking attendance.
And we would prefer obviously,
that they would go a step further
and actually measure whether or not
they learned the content of the training
with respect to leadership.
So moving on, I will now state the second outcome
and then Jenny, you can state the outcome indicator
and make your comments on why you believe
it may or may not be properly measuring the outcome.
The outcome is and I quote,
strengthening workshops on leadership
organized with women and men leaders.
And what would be the outcome indicator
that you're looking at?
Right, then the indicator that we analyze
is the number of men and women leaders
strengthened in leadership.
So the issue that's sort of associated with this indicator
can be looked at from different perspective.
One is that strengthening,
strengthening leadership is such a broad term.
So that means the indicator itself would require
different kinds of qualitative indicators.
At the moment, the focus is mainly on the quantitative,
the number of leaders being strengthened in leadership.
So that's one aspect we thought
the indicator could be enhanced.
That's great.
Okay, thank you, Jenny.
And we'll move on to the next outcome.
Consultation mechanism with women's groups,
economic interest groups and organizations.
And the outcome indicator that you're looking at?
Is the number of consultation mechanisms
established and functional.
So again, the issues with the functionality,
which also require indicators
that goes beyond the quantitative.
What kind of mechanism?
Is it about addressing the effectiveness
and different qualitative sort of matrix, for example?
Is it really like regularity of meetings?
Are we talking about that?
Or is it talking about decision-making processes?
And what are the follow-up actions?
So these are sort of missing, missing matrix
in this over quantitative indicator.
Right, and I'm doing a similar one,
the mechanism for consultation between government partners
put in place integrating gender.
And the outcome indicator there
is number of consultation mechanisms
established and functional.
And the key thing I would add to what you've said is,
they've written right in the outcome indicator functional.
So you would think they would say,
okay, let's get a tool, measurement tool,
as how do we define and measure
that that consultation mechanism is actually functional?
And they don't.
All they do is count the number of mechanisms
they've produced on a piece of paper and maybe used,
but they haven't gone the next step,
which is what's the expected outcome of that mechanism?
It is, it's functional.
So what do you mean by it's working?
It's functional.
How are you gonna measure that?
And that's missing in this performance measurement framework.
So they can't claim it's functional
because they're not even measuring it.
Exactly.
Maybe just to add a little bit,
going back to my sort of similar indicator,
because there's actually, in the outcome statement,
is looking into the economic interest groups
and organizations.
So in terms of measuring the functionality,
I think another aspect kind of missing
is about the inclusivity and accessibility
by underrepresented groups.
So that is quite missing out in the indicators,
the way it's being formulated at the moment.
Right.
And as we get further down,
we'll return to your point about decision-making,
because in particular with women in the project,
and I should elaborate and recap that in part one,
major part of these services of the project
is for the vulnerable women to receive training
in entrepreneurship and leadership,
to the point where they have more power
in decision-making bodies and they have more money
from learning how to run a business.
And that's basically what they call the empowerment.
So we'll get to that.
So that's a good point about the decision-making
that you raised.
So the next outcome is increased community awareness
of the importance of women's participation
in decision-making,
and the outcome indicator you'll be looking at
to measure that outcome?
Yes, it's the number of people in communities
directly sensitized to the importance
of women's participation in decision-making.
Desegregated by sex and occupation
in practice leaders or not.
So the issue that we find about the indicators
that while the indicators track the reach,
the outreach of the sensitization efforts,
what is missing in terms is kind of related to the quality
or the depth of the impact, for example,
the attitudes, whether attitudes change
after the sensitization.
So that sort of aspect were not really captured
in the indicator.
Another bit is about, you know, in terms of who actually came
forward to this, who are being reached out
in terms of these activities,
those were also not quite captured.
So some other qualitative measures might be needed.
Right, and I would add to that also the fact that they do this
from the performance measurement framework.
You can see it's an annual survey with targets
to reach levels of awareness,
but how do we know these targets were reached
due to the project services of raising awareness?
We don't because they only measure it once
and they didn't use a comparison group,
even though interestingly enough later on in this podcast,
they actually do use a comparison group.
So it begs the question, why did they not use
a comparison group here to see if communities outside
of where the project is operating also got sensitized,
but at a lower level?
And so that's not being done here.
And also, did they ask the people surveyed
if they knew about the project and if no,
use them as a comparison group?
So that was a key thing I would add to what you've said, yeah.
Yeah, and then we have quite a number we will get back
when it comes to other indicators,
it's about the desegregation.
I think there's more intersectionality aspects
we can look at, but we will come back
to the desegregation issue.
Sure, yeah.
The next outcome is increased capacity of men and women
leaders to intervene in communities.
That's the outcome.
The outcome indicator that I'll be looking at is, quote,
perception of community members on the effectiveness
of the promotion of gender equality
by the project's gender equality chance
and gender equality champions.
Now, the issue here is it's measured only once per year
through focus groups.
What about other communities that did not
receive the leadership training?
How are those women doing on leadership there?
So we have this issue of capacity of men and women
leaders to intervene, but they could
be intervening in areas where the project is not operating.
And so again, it's only measured once a year,
and it could be very easy that they could have figured this out
all on their own without the project services teaching them
how to intervene in these communities.
And even the perception of community members
on the effectiveness of promotion of gender equality
could be going up in these other communities
where the project's not operating.
So it's the same problem of where they're not measuring.
If they don't want to use a comparison group, that's fine.
But the other issue is if they're only measuring once
a year, that's not enough.
They need to measure frequently before they start the project,
during, during, during the project,
and after to show that there's a trend where the sensitization
is going up.
Yeah, otherwise there would be causality issue, right?
How do we claim that is the contribution of the project
intervention?
Right, and you don't necessarily need a comparison group.
You can just do proper quasi-experimental
and measure several times rather than just once.
You're exactly right.
You can't make these claims.
You can at least try to improve on it,
that the perception levels are moving in the right direction.
But if you want to claim that, then you've
got to measure more frequently, right?
Yeah, over time, tracking the over time, yeah.
Exactly.
Thank you.
The next outcome is increased influence
of women as active citizens in their communities.
And you're looking at an outcome indicator for that outcome.
Yeah, we have a couple of indicators here.
So the first one is the percentage
of women targeted by the project who
participate in decision-making bodies in their community.
So this indicator measures the participation rate of women,
again, quantitative in decision-making bodies,
which is a good start.
But it really doesn't reflect or capture
the level of influence on the effectiveness
of their participation.
So again, this participation does not always
equate to having an active role or voices.
We have seen even from our experiences,
sometimes the typical gender role, taking notes,
are happening in many of these decision-making bodies.
So it's important to look at what kind of roles, what kind
of influence do they have.
Can they set the agenda, for example?
And does it really make a difference
if those decisions are being initiated by women?
So these are some missing aspects.
Yes, and I would add to that, again, it's
only measured once a year.
But it may not be a problem if it's only once a year,
as long as they look at the decision-making bodies,
like you said, and see what exactly do they
have any power in making decisions.
And has their representation as women gone up over time?
And we just don't know.
We just know the percent of women targeted,
but they don't go into detail about.
It's another issue of defining and measuring actual influence.
And just being a member of the decision-making body
is not enough, is what you point to.
Absolutely, yeah.
So we'll come back to some of the other two indicators
later on.
Right.
And I think we're going to the next one, which
is increased inclusion of women as actors
in social and economic development.
And you have an indicator there that you want to look at.
Yes.
So that's the percentage of women
targeted by the project who have improved
the management of their income-generating activity.
So this, again, is what do we mean by improved management?
That's a broad term that requires a specific matrix.
But again, it's also about why not measure the actual improved
income generated because of their participation,
their empowerment through the project.
So these are kind of the missing aspects in the indicator.
Absolutely, yeah.
And I've noticed this in another episode I did,
where they just seem to get scared of using
the I-word, income.
They will talk about savings plans,
but they won't say, show me the money.
And this is a good example where if you just show me
the money, please, and this will show that women are actually
inclusive, being included in economic development
because their incomes are rising.
And you would think it would just look at that.
Exactly.
Again, you can categorize it almost
as an output rather than an outcome indicator.
Yes, yes, exactly.
Because again, they're not measuring increased.
And the key word is increased inclusion.
It just says percent of women targeted
who have improved the management.
So maybe their management has improved,
but that's evading the actual inclusion of economics.
Exactly, exactly.
And then the last outcome of the nine
is strengthening the empowerment of women
among the most vulnerable across Senegal.
And this is where the irony comes
because this is the one indicator
at the very top of the food chain, results chain,
where they are actually using what's
called the Female Empowerment Index, FEMI.
And they probably named it after the well-known musician
from Nigeria, which is kind of cool.
Give them credit for that, even though it's Senegal.
And it's good.
They've used a comparison group.
So they've actually set in the performance measurement
framework, we're going to use a group of women that are not
participating in the project.
And we're going to measure their empowerment levels
on this index.
But the problem is it's a bit late.
So even if they can show, the first thing they need to do
is even before the project starts,
hopefully they've done this, they've
shown that there's no statistical difference
between the two groups to begin with.
That's how you start before the project.
And then you can show over time that the group that they
are engaging with in the project is actually
their index is going up in a much greater rate
than the control group.
And that could be happening.
We don't know.
Because the other issue is we don't have the data.
They only provide a blank PMF.
And if you go to the project browser, Global Affairs Canada,
they don't give the data for this indicator, which
is ironically one of the few good ones.
And there's no data available.
So we don't know.
Yeah.
Sorry, go ahead.
I quite like what they use in the indicator,
like in brackets, adaptive version.
That means they are quite sensitive to adapt
to the local context.
But in terms of improving the indicator,
they could also measure a bit of what we call it
the process indicator.
It's like to what extent that the adaptation process
to the local context is addressing
the cultural diversity sensitivity, for example,
to make those indicators relevant.
So I feel that that could be an area for the organization
to rethink about these kind of composite indicators.
Yes.
And to add to that, there are six components
of the FEMI index, the Female Empowerment Index.
And two of them are relevant to this project.
One is employment, achievements, and achievements
on decision making.
And those two, they obviously I think
have adapted so that they're going
to be asking the women specifically on how are you
doing on employment and how are you doing on decision making.
But the irony is, as we've just covered,
is that for employment, they're not measuring
incomes in the project.
So even if they say yes, we have no data
to support their claim because it's biased,
because they're self-reporting it.
To what extent the income has actually increased.
Right.
They could say yes, but we just don't know.
Because the indicators underneath the FEMI empowerment,
like employment, they're not measuring income.
And on decision making, as we just
discussed, which is the other dimension of the six
they could use in the index, they're not looking at,
did they increase their power on these decision making bodies?
We just don't know.
So that's the irony in all of this
is they've got this wonderful indicator at the end
where they can wave the flag and claim that their project women
compared to the comparison group have become more empowered.
But then when you ask them, what do you mean by more empowered,
you go to the performance measurement framework
and there's no support for whether or not
their incomes went up or whether or not
their decision making power went up
because they're not asking them in the performance measurement
framework for those indicators.
So that's what's really interesting about it
is it's kind of like a smoke and mirrors, as they say.
But that's why we're here is to bring that out.
And if they were to improve the indicators the way
we have discussed them, then they
could make the claim that the reason their empowerment has
gone up, or assuming it is, we don't have the data,
but if we get the data, it would show that.
And it's because we can show their incomes went up
compared to the comparison group,
their decision making power went up
compared to the comparison group, et cetera.
So we just don't know, but at least they're
trying to move in the right direction.
So what we can do now is go back to the outcomes
and look at the other indicators where we could cover all 17
because I think we still have a bit of time.
So we'll go back to the outcome strengthening workshops
on leadership organized with women and men leaders.
The other outcome indicator for that
was number of leadership workshops held.
And it clearly falls into that problem category
of just attendance and everybody goes home,
as I say in my trailer.
They don't even measure whether or not
they've strengthened their leadership skills or knowledge
with respect to leadership after the trading.
So that's a problem there.
The next indicator was for, sorry,
you wanted to make a comment?
Go ahead.
Yeah, I would just like to add to in terms of,
they could also track in terms of the active engagement
of both women and men participants,
but maybe even organizers because those are the women
and men leaders who will organize this training.
So in terms of the engagement, what kind of roles
they are also playing out in running these workshops,
that could also be an interesting aspects
in setting the indicator.
Oh, yes, because one of the target groups
is male champions.
And that's a key, very important point is disaggregation
and seeing if the men are actually championing
and moving in the right direction in that area.
Because right now all we have is we're assuming
that their leadership has been strengthened
from the training, but again, there's really no measurement.
The pre-post, right?
The pre-post workshop assessment is missing out.
Yeah, even a pre-post is limited,
but at least it's better than nothing.
You could do in the same day,
figure out if they've actually learned
and their levels have gone up,
even for that just workshop on training
with respect to leadership, yeah.
So the next one is increased community awareness
of the importance of women's participation.
And what I was looking at there is the other indicator was,
quote, direct beneficiaries in the project
through focus groups also measured on importance
of women's participation in decision-making.
Well, of course there's self-reporting bias there.
They're holding a focus group.
It's of the people that are participating in the project,
the women.
So they're obviously a bias for them to say,
yes, I've become increased aware of the importance
of women's participation in decision-making.
So this is a case where I believe focus groups
are improperly being used, and this is a common problem,
where instead of using the focus group
to compliment a problem on an indicator,
to say, why are you not achieving your target
on this indicator, going in and having a focus group
and asking them, which is why you use focus groups,
to compliment a quantitative indicator.
They're instead using it to replace
the quantitative indicator measure on community awareness,
for example.
So this is a problem that I often see.
Yeah.
Totally agree, yeah.
The next one's for you.
I'll read the outcome.
Next one, yes.
Increased capacity of men and women leaders
to intervene in communities,
and you have an outcome indicator for that.
Yes, the number of men and women trained in leadership
who carry out gender equality sensitization activities
in communities desegregated by gender.
This indicator measures the quantity of activities
and people trained, yeah, but really does not necessarily
reflect the quality of public awareness activities
and what came out of it.
So what they could actually improve is to really
maybe have a follow-up study to actually see
does these kind of activities generate
some sort of behavioral change or attitude change
in the communities.
But again, I think that desegregation by gender
is good to have, but depending on the communities,
you might need also other sort of different matrix
to understand the social economic background
of the community members who actually come forward
to participate in these activities.
Yes, it seems quite complicated,
like when you look at it, how are we gonna say
that these men and women who showed up
for these project trainings actually increase their capacity
to quote intervene in communities effectively?
I mean, the first part is do they have the technical capacity
to go into these communities?
And that's the immediate outcome that they put here.
But you've made a good point about even if they get
to that level, what's gonna happen next is actually
gonna lead to the awareness and changes at the,
downstream as they say.
So yeah, attendance simply is just not good enough.
Yeah, absolutely.
And the next one you're gonna be doing is for the outcome,
increased influence of women as active citizens
in their communities.
Yeah, and the indicator that I was looking at
is the perception of members of community organizations,
men and women on women's participation in decision making
within communities desegregated by sex.
So this is again reliance on the subjective perceptions
which have already we discussed earlier on,
which can be influenced by personal biases, cultural norms.
So we might need other sort of qualitative approaches
or other sources of data to substantiate
or at least to unpack some of these perceptions,
subjective perceptions.
Yeah, and then another aspect just before going back to you
is again, the desegregation just by sex.
I think it also kind of overlooking
on other social economic dimensions, for example,
age, ethnicity, disability, social economic status.
So I think that desegregation is good,
is a good start by gender, but a lot of indicators,
you know, kind of miss out on collecting data
that desegregate on these other important dimensions.
Right, and it's also only measured once a year.
So it's very difficult for the project to make the claim
that their trainings with these women
have actually led to the achievement of the outcome,
which remember is quote, increased influence of women
as active citizens in their communities.
And the way they're gonna try to show that
is by asking, I think it makes sense.
They wanna go to the people on the other end,
the communities that have received this engagement
from these women and say, hey, what do you think?
Do you think these women actually, you know,
influenced you in the desired direction?
Women's participation in decision-making in communities.
That's basically what they're looking at
in terms of the increased influence.
But it seems to be quite a challenge to actually show that.
But only measuring once a year is not good enough.
They have to do it more frequently or also,
or pick a comparison group and ask another group of women
outside of the project, sorry, other communities,
what they think of women's participation in decision-making
and show that there's a major difference
or measure more than just once a year.
Exactly.
The next one.
Next one is also mine, right?
Yeah, I'll read the outcome.
Increased influence of women as active citizens
in their communities.
That's the same outcome.
But I think the outcome indicator is slightly different.
Yes.
Yes, this is about the perception of women
targeted by the project on the participation
in decision-making within communities and in the family.
So, but again, it's kind of similar
to what we talked about previously
is about the perception, again, that can be sort of affected
by other sort of internalized gender roles,
self-esteem and societal expectations.
So this may not align with the actual.
So instead of measuring the actual,
which we highly recommend in terms of the influence
of their participation, purely relying on perception
may not be sufficient to really be an indicator
for the outcomes, the increased influence of women
in the, as active citizens in their communities.
And in this case, it's in the family.
So, you know, what we could sort of an improved indicator
would could, for example, is maybe, you know,
getting some information from the family members,
like community members, for example.
So this again, self-reporting bias,
I think David, you already talk about
on some other indicators that also comes into this indicator.
Yeah, exactly.
We preferred that they use the other approach
that they used in the previous indicator,
go to community members, not the women themselves
in the project, or like you said, quite rightly,
other family members, because I think they're claiming here
that their decision-making power is going to go up
even within the family for these women,
not just in the community at large.
And this is in other episodes, sorry to bore you, Jenny,
I'll bring it in, in Vietnam with Care Canada,
we raised this same problem where,
how do you get at measuring increased women's power
in the household?
It's quite tricky, but you could go and ask
other members in the household, but that's not even done.
Self-efficacy is important, yeah,
but at the same time, I think for,
in order to generate robust evidence base,
other type of data collection methods would also be needed
to strengthen the evidence base.
Right, exactly.
The same outcome, increased inclusion of women as actors
in social and economic development,
sorry, that's not the same outcome, it's the next one.
There was another indicator they used called,
and I quote, percent of women targeted by the project
who are business leaders or entrepreneurs,
that it just doesn't go far enough.
They could tick off the box, but are they earning any money?
And are they earning more money than those in the,
that are not participating in the project
and not receiving business training,
leadership training, et cetera?
So you definitely need to have a comparison group,
or even without a comparison group, it's the same problem.
They can't just tick off the box and say,
I'm a business leader and raise their hand.
They've got to show that they've actually earned income
as a better measure of increased inclusion
as part of economic development.
And the next one, indicator for that same outcome,
increased inclusion of women as actors
in social and economic development.
Another indicator they used
in the performance measurement framework
was percent of women in employment
or self-employment after training.
So again, the problem there is the same,
is it's a good idea that they've been employed,
but they need to show that the training from the project
has made the higher percentage of these women
become employed or self-employed, specifically with income,
compared to women who didn't benefit from the project,
didn't receive the services such as leadership training
and entrepreneurship.
Maybe another aspect is also the nature of the jobs,
the employment, is it related to the training or not?
Exactly.
Disaggregation, please.
Absolutely, yeah.
And I think we've got one more here,
increased inclusion of women as actors
in social and economic development.
And again, one of the indicators is perception of women
targeted by the project and men
guarding their socioeconomic situation.
So your point's a good one, Jenny,
that they're disaggregating the men and the women,
which is good, to see if both groups
are moving in the right direction.
The problem is, is it's the targeted by the project.
So again, you're gonna have some self-reporting bias
where they're gonna say, oh yeah, my measure,
or however they define it, of being included
in the economy has increased.
But they need to use other measures
that we've suggested before to get at this increased
inclusion rather than asking the project participants
themselves because of this self-reporting bias.
Yeah, you also have an earlier point
about the targeted group.
Are they reaching out to the targeted group?
Because it's all about the underrepresented
or marginalized women and men.
But again, it's that, are they, are the target group,
have improved their socioeconomic status or situation?
That is also of interest to measure the outcome.
Oh, absolutely, that's one of the issues is,
I've found with many projects,
the vulnerable women as they identify them here,
sometimes they're not even reached.
Exactly.
And it's the women who run the NGOs
who benefit the most from the project
in terms of salary, et cetera,
and the actual delivery to the vulnerable women is minimal.
And in this case, the measurement is certainly minimal.
So it's very difficult to reach.
So I think we can conclude that,
at least from my point of view,
that this project in its current design
from performance measurement is not adequate
to support the project in colleges
and institutes Canada claiming
that they're achieving their outcomes.
That is, that women are becoming more empowered economically
and politically.
But that's my opinion.
What do you think, Jenny?
Now, I agree with you with your conclusion,
but I also want to applaud the organization
for setting actually very good outcome statements.
But I think there's really a space as we,
the objective of our podcast is actually
try to improve practice when it comes
to formulating the indicators.
So if there's already a lot of investment
in setting out very good outcomes,
with suggestion then why not just also improve the indicators?
So there is actually a very clear,
more clarity and specific matrix
that actually generate evidence base
to support the argument,
whether the outcomes have been achieved or not.
Well said.
Of course, I have a bias myself,
setting up this podcast for that whole intent and purpose.
So thank you, Jenny.
Thank you so much, David.
Yeah.
Thank you for having me.
You're welcome.
And so what we'll do now is the last sentence in this podcast
or a few summary remarks is that,
we will send these part one and part two,
two colleges and institutes Canada,
but more importantly,
we'll send it to the honorable minister
for international development,
as well as the shadow critics for international development
in the conservative party,
the new democratic party,
the Bloc Québécois party,
as well as I think maybe even the leader of the Green Party.
And we'll leave it at that and continue to recommend
that one, they make the performance measurement frameworks
available to the public
on the Global Affairs Canada website.
Number two, they actually provide the data
from the indicators.
Here, we don't even have the data.
It's just a blank performance measurement framework.
And also finally, as we've discussed,
the Global Affairs Canada should follow their own guidelines.
And that is when they approve
a performance measurement framework,
they should be saying,
why are your outcome indicators
not properly measuring your outcomes?
Please revise before we disperse $18 million.
Finally, if you're gonna be a charity,
become a charity and don't make these claims
of these outcomes that you're achieving.
There's nothing wrong with that,
nothing to be ashamed of.
And that's the other option they can have,
these organizations,
is don't have any outcomes at all.
Just deliver the services because everybody needs them
and everybody goes home.
There's nothing wrong with that as another option.
What do you think, Jenny?
No, that's a very good remarks to sum up.
But as a practitioner,
I also suspect that there's actually maybe
updated performance measurement framework
that we're not really shared.
So maybe there are some ongoing substantive work
to improve the monitoring evaluation,
but maybe these kind of documents
are not always shared externally.
So I also encourage implementing agency,
particularly with such a huge budget,
to really make this kind of information
also available.
That's a good point.
And also some of these NGOs have claimed
they don't have the evaluation expertise
to design proper indicators
or they don't have the resources or money
to have proper comparison groups
because it's expensive,
you have to measure another control group.
Or as we've advocated,
you have to measure more frequently,
which costs more money.
But my position is the government of Canada
should say, here's your 18 million,
but by the way, 10 to 15% of it
has to go to monitoring and evaluation.
Right there.
I think what, yeah, I totally agree with you, David.
There's always the budget concern,
the expertise concern,
but even though there's a wide range of resources
out there to improve monitoring evaluation practices,
the issue is I think what we're advocating
is that maybe steer a little bit away from the desire
to just quantitative indicators
could maybe perhaps use a mixture
of quantitative and qualitative indicators.
So at least you can have more substantive,
more robust sort of measurement to measure results.
Yeah, I agree.
As I mentioned earlier,
some people may suspect I have a quantitative bias,
even if that's the case,
they have to be complimentary.
You can have increased inclusion.
You can have a quantitative indicator for that,
but you can also have a focus group qualitative
to compliment that by saying,
okay, we have statistical significance here.
It's clearly shown that your incomes have gone up
greater than the control group,
but we'd also like to know what worked, what didn't
in terms of increasing your incomes.
And that could compliment the actual achievement
in the right direction of the quantitative indicator.
And you certainly can use both.
The problem is they often don't,
and they often miss the mark on the quantitative
as we've I think highlighted here,
or they abuse the qualitative by using it just by itself.
So I think we're moving in the right direction, yes.
Yeah, maybe just one last comment, if I may, David.
Of course, we don't know the actual implementation
of the monitoring evaluation
or performance management framework, as you call it,
that we also encourage, at least from my point of view,
to encourage, if that wasn't already done,
collaboration with the local research institutes
to collect this kind of qualitative data
or quantitative data, so to improve actual measurement
of outcomes.
Absolutely, and in fact,
I think a lot of the organizations do that,
use local research institutes.
I've been involved in that when I was in Nigeria
doing third-party oversight on local Nigerian survey firms
that were collecting the data for the indicators.
Absolutely, yeah, that's a very good point.
In my trailer, I talk about that's important,
but what's really important is the initial design,
where if you don't design the indicators properly,
you have garbage in, garbage out.
And that's the focal point of my podcast,
but yes, absolutely, you have to use local firms,
and it's quite fascinating how many local firms
in this global economy have expanded
by relocating in developing countries.
There's quite a few very impressive
local data collection firms that have developed
to address exactly what you've raised, yes.
Yeah, maybe on that note,
because I remember in your part one podcast,
you talk about one of the target group of the project
is the project staff itself.
So maybe that's also an avenue,
if that is not already done,
is to include performance management
or measurement capacity as part of the key competencies
to develop the local project team members,
sort of part of their competence development framework.
Oh, absolutely, and that's a big issue,
is some of these NGOs will respond
that they don't have the technical capacity
to do proper measurement and evaluation,
and it's always a challenge.
Absolutely, yes, very good point.
So I think we'll end it at that,
and we'll send it off to the minister,
and hopefully we'll build this slow body of evidence
where maybe someday performance measurement frameworks
will be on Government of Canada websites
for the four to five billion dollars worth of projects
that they have every year,
and the indicators will properly measure their outcomes.
Okay. Thank you so much, David.
Okay, we'll keep in touch,
and maybe you'll be on another podcast if you want.
Thank you.