I'm your host David Wand and welcome to episode five,
part one, where we introduce a new
international development organization.
That international development organization is
the Mennonite Development Associates,
located headquarters in Waterloo, Canada.
You can learn more about the Mennonite Development
Associates International Development Organization
by going to their website at www.meda.org.
We're gonna be talking about one of the projects
that they have been provided money from the government
of Canada, courtesy of the Canadian taxpayer.
This project cost the Canadian taxpayer $19,040,307.
The title of this project is,
A New Partnership for Sustainable Impact Investing
in Frontier Markets.
This project, unfortunately, doesn't specify
the specific developing countries
where these monies are spent.
Instead, it only indicates that 30% of the $19 million
will be spent in Africa, 30% spent in the Americas,
and the remaining 40% being spent in Asia.
I have acquired, through an access to information request,
the performance measurement framework, or PMF,
for this project.
Reviewing the PMF, I have discovered there are nine outcomes
that are expected to be achieved from this $19 million,
and 23 outcome indicators are in the PMF
that are being used to determine whether or not
these nine outcomes have been achieved.
Unfortunately, based on my review of all 23 outcome
indicators, none of those indicators are able
to adequately measure the achievement of the outcomes.
Therefore, the Mennonite Development Associates
cannot make the claim that they are achieving
these nine outcomes based on the performance
measurement framework that they have designed.
So in this part one, I will just give you
a brief description of what the services are
that this $19 million is being paid for,
and it falls into three areas.
The first area is funds that are provided
to investment firms, and funds or grants
that are provided to small, medium enterprises.
And a third component of this project is paying
for Canadian firm managers, investment firm managers,
to mentor investment firms in developing countries.
That is, mentoring the managers of investment firms
in developing countries.
In those three areas, particularly investment funds
and small, medium enterprises in these developing countries
receive money.
They also receive, these investment firms
and small, medium enterprises, technical support
on social and environmental principles
so that they can access these funds from this project.
And as I mentioned, the investment firms
in these development countries, their managers
also receive mentoring from investment fund managers
from Canada.
The small, medium enterprises receive grants
for social innovation.
And they also receive technical support
on social innovation.
So basically, those are what the funds,
the $19 million are used for.
And what I'm now going to do is give you two examples
of outcomes from the performance measurement framework
where the indicator, the outcome indicator
that's been designed to determine whether or not
the outcome has been achieved is flawed.
So we'll start with this outcome from the performance
measurement framework.
The outcome is strengthened management capacity
amongst investment fund managers in selected countries.
The indicator for measuring this strengthened
management capacity is, and I quote,
level of business skills and knowledge
among mentored fund managers.
Sounds pretty good.
It sounds like they're ready to measure some skills
and knowledge levels.
But unfortunately, when you look at the performance
measurement framework, once again, it falls into
the problem of self-reporting bias,
where these mentored fund managers are themselves
expected to self-assess their quote,
acquisition of knowledge and skill,
rather than being tested objectively on their knowledge
and skill levels regarding how to work as a fund manager
in terms of their management capacity.
They need to be objectively tested on their knowledge
and skill levels.
So that's the problem with that indicator.
Even though it sounds good, it's only by getting a hold
of the performance measurement framework that we see
that once again, there's a bias and they're being asked
to self-assess themselves rather than being tested
objectively from somebody externally.
The second outcome that I'm going to look at is called
strengthened business practices for economic growth,
job creation, and social impact amongst investment firms
and small, medium enterprises.
The indicator to measure whether or not these
strengthened business practices have been achieved is
qualitative evidence of strengthened capacity
of investment firms to promote improved social
and environmental practices.
The problem here again is self-reporting.
These investment firms self-report their quote,
strengthened capacity to promote through case studies
and most significant change stories.
Again, it would be better to test people in these
investment firms on their knowledge of social
and environmental principles, plus their skill levels
on how to promote these principles.
Plus, a comparison group of people in investment firms
outside of this $19 million project would be also helpful
so that you can show that the $19 million project
had people at these investment firms that you were
providing them with training and technical support
on these principles.
You would hope that they would have higher knowledge
and skill levels on how to promote these social
and environmental business practices compared to another
group of people in investment firms that don't receive
this project training.
So in part two, we're going to invite the Mennonite
Development Associates to respond to our critique
of all 23 outcome indicators and why they're flawed.
And we'll also have another evaluation expert on part two
of the podcast who will work with me in going through
at least nine of those outcome indicators to make sure
that we cover all nine outcomes of the project.
Hopefully we'll go through more than just nine outcome
indicators and hopefully cover all 23 outcome indicators
to explain how each one fails to properly measure
the outcome.
So stay tuned for part two where we will go through
in more detail all 23 outcome indicators as to why they
have a problem with not properly measuring the outcomes.
So if you're interested in receiving the performance
measurement framework for this project and also the outcome
indicator summary for all 23 outcome indicators,
explaining which of the five problem areas each outcome
indicator falls into, feel free to send me an email
at evaluatecanadaaid at gmail.com.
And if there's any evaluation experts out there who would
be interested in being on a part two of the podcast
where we go into depth on each outcome indicator,
feel free to email me and I will send you the remaining
performance measurement frameworks that I have
for season one and you can choose one and then I can also
send you the notes that I have from my perspective
of why the outcome indicators are flawed or not.
I look forward to doing part two and stay tuned for part
two which will be coming up very soon in a couple of weeks.
Bye for now.