Welcome to the Improving Development Evaluation Podcast.
I'm your host David Wand and in this trailer
we're going to give you some more detail about what the podcast is actually
trying to do. We're going to talk about three things.
One of the content of each episode including who the guests will be on the
podcast. Secondly we're going to talk about
season one about what we plan to cover in season one
which includes 12 episodes and then finally we're going to talk about some
ideas about how you the listener may want to
get more involved with the podcast in addition to just
hopefully listening to it. So let's get right into it.
So the content of each episode we're going to have two parts.
In part one the host is going to introduce
a new international development organization for each episode
and in this part one I'm going to talk about the project
that this new international development organization
has been funded to deliver in a developing country by the government of
Canada. So I'm going to briefly describe the
services the project delivers, the cost of the project to the Canadian
taxpayer and most provocatively and most importantly
I'm going to conclude at the end of part one that we've got a problem.
That is the project has failed to achieve its outcomes and you may be
wondering how I'm going to get there. First of all what do we mean by
outcomes? So for example a project trains people on how to
grow peanuts and how to sell peanuts. The outcomes you would expect from that
training would be that those people would learn how to
grow peanuts by having increased skill and knowledge
levels as well as increased skill and knowledge levels on how
to sell those peanuts.
And I've come to the conclusion that they have
not been able to in the project claim that those outcomes have been achieved.
Why? Well it's because I've acquired through a access to information request
a PMF, a performance measurement framework.
These are not available to the public. You have to first make
an access to information request and I have acquired 12 of them.
So in part one what I'm going to do is briefly describe
how the PMF or performance measurement framework fails
to adequately measure these outcomes. So they fall into five problem areas
and the first problem is we have to look at the outcome indicators that are
trying to measure the outcomes like I've just given an example.
So in the example and the first problem would be
people show up for the training on how to grow peanuts and how to sell those
peanuts but they're never measured on their
skill and knowledge levels. Instead they just take attendance and
everybody goes home. So that's the first problem. The second
problem they could have is they show up they measure their skill
levels but they don't measure them directly.
Instead they ask them how do they feel about their ability to grow peanuts
and their ability to sell those peanuts after getting the training on how to
grow peanuts and how to sell peanuts. They're asking them how they feel
about their ability. They're not being measured objectively like through a test
or a certification process or whatever. So that's the second problem. The third
problem is sometimes the outcome indicator doesn't measure the outcome at
all. So let's say expected outcome from the
training on how to grow and sell peanuts is
an increase in income from selling those peanuts.
Sounds logical. Problem is they don't say show me the money. Let's see your bank
account. Let's see your bank balance from selling
those peanuts. Instead what they do is they say show me a savings plan
that you have. Well that's not really related.
They've got to show direct measurement of the income. So that's the third problem.
The fourth problem is maybe they show up they properly measure them objectively
on their skill and knowledge levels on how to sell peanuts and how to
grow peanuts. But they only do it once before the training and they do it once
after the training. So this is a problem also
because let's say they measure them as a group
on their skill and knowledge levels on how to grow peanuts and how to sell
peanuts before the training starts. And then they get their test scores back
and some of them say oh jeez I'm really bad. My knowledge levels
are really low on how to grow peanuts and how to sell
peanuts. I think I'm going to go down the road
and have a chat with a very successful peanut farmer and ask him or her
how do you do it. Or maybe they'll go online
and figure it out if they have access to the internet through a YouTube
video on how to grow and sell peanuts. So then they after the training
they measure them as a group once and guess what?
Their test scores go up. The group has improved
in their knowledge and skill levels which you would expect.
Problem is we don't know why. Could be because of the peanut farmers they were
talking to down the road before they got the last
test. Or they could be figuring it out on the
internet all by themselves. So that's problem number four. We just
don't know if the training was responsible for the increases
in knowledge and skill levels that they actually observed and measured.
And then finally problem number five which is let's say they do it
perfectly. They measure them before during and
after the training. So you can see this nice
trend of increases in skill levels and
knowledge level. They also watch their incomes. They
look at their bank accounts and they notice oh this is great. Their
incomes from selling peanuts has gone up due to the training
before during and after. It's gradually gone up which is pretty good.
But problem is there's no comparison group of people
who also grow peanuts but did not attend the training.
So there could be another group of people in some other part of the country
that are doing just fine teaching themselves how to grow peanuts
and how to sell those peanuts by talking to other peanut farmers
or whatever. So the problem there is we don't know
if the project training that you paid for as a Canadian taxpayer
was actually responsible for the increase in skill and knowledge
levels of how to grow and how to sell peanuts.
And also the increase in incomes from selling those peanuts. We just don't know.
It could be equal or it could be even lower even though we're seeing increases
in incomes and knowledge and skill levels. It could be lower than the group
that never attended the project training. So that's problem number five.
So the podcast is going to focus in part one
on a summary of those problems for the indicators for the performance
measurement framework and where they fall.
Now there are so many outcome indicators in these projects.
We will not have time to go through all of them but I will provide a summary
of all of those indicators as to which ones
fall into one of those five problem areas.
So at the end of part one I will summarize
the PMF or performance measurement framework and I will just cut to the
chase here. I will be concluding that the project
in its current form with its current performance measurement framework
is flawed and cannot support the international development organization
making the claim that their project services are achieving their project
outcomes. That's why we have this podcast to
bring this out and provide solutions on how to improve
development evaluation. So at the very end of part one I'm also going to invite
the international development organization to attend part two
of the podcast where they will respond to this critique.
But this is only a starting point for me as the host.
I also will be inviting in part two on the podcast at least one and
possibly two evaluation experts to also respond to my critique
and provide alternatives to how to improve the evaluation of the project.
In part two it will take about 40 minutes so we won't be able to go through
all of the outcome indicators but we hopefully will be able to select a few
from each of the outcomes of the project. Usually these projects have five or six
outcomes and then for each outcome they have two or three
indicators. We'll probably just have to pick a few of them
and go through them to make sure we cover all of the outcomes.
And at the end of part two we're going to
answer the following question the host that's me
and the evaluation experts and that is the question
is the project in its current form with its current performance measurement
framework which they are required to follow and develop before they get
funding from the government of Canada is it adequate for the international
development organization to make the claim
that the project is achieving its outcomes. My answer
right now is no. There may be a few of those 12
organizations that you'll see in season one where
maybe some of the indicators get it right
but most of them fall into one of those five problem areas and for each episode
I'm going to post on a website the project's performance
measurement framework as well as a summary of how each of
the outcome indicators in that performance measurement framework falls
into one of those five problem areas.
So just to recap in part two of the episode I will be encouraging the
international development organization to respond to my critique
which includes offering alternative solutions to this problem of how to
improve their evaluation of their project. The
idea is to get a discussion going about how to improve
development evaluation.
So you might be wondering what we plan to cover in season one like I mentioned
we have 12 episodes introducing 12 international development
organizations each one having a project along with its performance measurement
framework and so these projects their cost to the
Canadian taxpayer all 12 of them is around 177 million
dollars.
What sort of services are these projects delivering?
Well some of the projects we'll be covering
focus a lot on training how to grow peanuts
how to tutor midwives how to change men's attitudes to support women's rights
and equality and raising awareness on accessing
reproductive health services and gender-based violence services.
These projects are being delivered in West Africa
in particular Ghana, Ivory Coast, Senegal, Burkina Faso and in East Africa
in Somalia, South Sudan, Tanzania and Kenya
as well as in Asia, Bangladesh and Vietnam.
And finally you may be wondering okay you're going to listen to the podcast
hopefully but in addition to that you may want to get a bit more involved
as you can sense from the podcast there's a bit of
advocacy involved where we're trying to advocate for improvements in development
evaluation. The government of Canada spends over
four billion dollars a year on development projects
all around the world. You may want to send me an email
regarding this podcast your comments and suggestions
on how the podcast can be improved. My email address is evaluate
canadaaid at gmail.com and finally I'm always looking for
evaluation experts who may wish to be on the evaluation
panel in the podcast part two.
And you can email me and I will be happy to send you the performance measurement
frameworks for season one. We've got four
already booked with evaluation experts but I can send
you the remaining eight performance measurement frameworks.
And for season two we will be getting an additional five
performance measurement frameworks. And if you're really interested
and you want a particular project for us to look at
on the podcast you can go to your search engine and enter
global affairs canada project browser and up will come the platform where all
of the projects thousands of them are listed.
And you can select a particular project that you have an interest in and you
will notice that even though there's a project
browser there's no performance measurement framework.
It's not available and so ideally what we want to see
is that performance measurement framework
available on the project browser platform as well as the international
development organizations putting their performance measurement
framework for each of their projects on their websites.
Ideally both the blank ones that we're discussing on the podcast
but as well the data that comes in on the outcome
indicators that they've listed. The irony of course is as you may have
figured this out is from this podcast most of those
outcome indicators are flawed so it wouldn't be very useful to use
the data coming in from those outcome indicators.
As the saying goes garbage in garbage out.
And that's the big issue that I am trying to raise
in this podcast. I look forward to hearing from you.
Thank you for your time. Goodbye and stay tuned for episode
one part one.