Welcome to the Improving Development Evaluation Podcast.
I'm your host David Wand and welcome to season two.
This is a trailer where we're gonna talk a little bit
about some of the changes and difference and focus
that we're gonna do in season two.
And I first of all wanna apologize
to the listeners out there where I had said two things
that we were gonna cover 12 episodes,
what I meant was 12 projects in season one
when actually we only managed to cover eight projects.
And I apologize for that.
And I also indicated that we would try to start season two
in September, but there's been a few delays due to,
what do you call it, economic circumstances.
So I am now finally getting in gear for season two.
So I thought I would give you some details
about how we've changed it a bit for season two.
And those changes are in just brief point form
and then I'll go into more detail.
First of all, I just wanna reiterate
for those who aren't familiar with the podcast
and haven't been following it,
the focus of this podcast is to expose those projects
funded by the government of Canada,
which is in the hundreds of millions of dollars,
if not billions, that are funding organizations
to deliver services in developing countries
that are making claims of development outcomes.
So we're not interested in looking at charity projects
and there's a lot of charity projects out there.
For example, money being sent by the government of Canada
through non-governmental organizations to, for example,
pay school fees for children in Africa.
That's a charity.
They don't care after they pay for the school fees.
Everybody goes home.
We're not interested in looking at those projects.
We're also not interested at looking at other projects
that are also vital and very important
and that is emergency funding.
I like to call it emergency.
People call it humanitarian.
It drives me crazy because when they say that,
what do they mean?
So this is money that's going overseas
to keep people alive in emergency situations
like in refugee camps for food, water, clothing, shelter,
medical care, emergency medical care.
That's fine too.
That's really important,
but that is not the focus of this podcast.
The focus of this podcast is to expose those projects
that like to wave the outcome flag
and that is claiming that their project services
are achieving these outcomes.
And as you've noticed from season one,
there's all sorts of projects out there
where they're not properly measuring
and they're not properly evaluating these project services
to support that claim.
So that's the focus of the podcast.
So what we're gonna do in season two
that's a little different is we're going to repeatedly relate
the flaws in these performance measurement frameworks
that the government of Canada requires
before they fund these projects
for services delivered overseas.
We're gonna connect those flaws to the development
assistance committee, DAC, DAC, evaluation criteria.
And for those who are familiar with it,
if you Google OECD, DAC criteria,
you'll notice they come up with six criteria
to properly evaluate a project.
And those six are one, relevance, two, coherence,
three, effectiveness, four, efficiency,
five, impact, and six, sustainability.
But for my podcast, I've kind of done a brief review
and we really are only looking at three of those six
that keep coming up because in the trailer for season one,
I identified five problem areas.
And those five problem areas
for these performance measurement frameworks
fall into three areas under the DAC criteria.
And they are one, relevance, two, coherence,
and three, impact.
So we're gonna be looking at in detail
how the performance measurement frameworks
fall into those three problem areas
related to the DAC criteria.
So I'm not gonna go into the detail right now about that,
but perhaps you'll get an idea of why it fall,
they fall into those three areas.
Particularly impact, because that's the big one.
Because the impact definition is basically claiming
that the project services and not other factors
are indeed responsible for achieving the outcomes.
And that the project beneficiaries
who show up to these services
don't figure it out on their own through YouTube
or talking to colleagues or whatever.
They actually have learned the outcomes
from the project, i.e. impact.
The second thing I'd like to talk about
that we're doing differently in season two
is despite the fact that I'm Canadian,
we can Canadians get access to information requests
on US Agency for International Development Projects,
USAID projects.
So that's pretty cool.
So I'm gonna expand to the USAID projects
and it took me a while, but a colleague down in the States
told me through an evaluation group
that the key magical word is activity MEL plan.
So that's an activity monitoring evaluation learning plan.
That's the document that is equivalent
to a Canadian performance measurement framework.
So what we're gonna try to do now
is get some activity MEL plans for projects
through an access to information request
in the United States.
I already sent a couple of months ago
an annual report from one organization
that received $5 million from the United States government
according to their annual report.
So I sent it to USAID and I said,
here's the annual report, 5 million bucks.
Give me the activity MEL plan that or MEL plans plural
that falls under this 5 million bucks.
They took about two months, they got back to me
and then they said, we can't find any activity MEL plan
for this 5 million bucks.
So I then sent back and I said, well, if you can't find it,
can you tell me where this money is coming from?
And that was about a month ago.
And they said, by the end of October,
they said they'd get back to me on first of all,
finding where the money came from
and hopefully secondly,
giving me the corresponding activity MEL plans
so that I can go through them and do the same thing
that we did in season one and see if any of the outcome
indicators don't smell right.
So that's another thing we're gonna be doing.
And number three, the format changes.
Instead of just doing two parts,
where part one I described the project and then in part two,
we go through all the outcome indicators
with evaluation experts, two different changes there.
I'm gonna combine parts one and part two,
which hopefully will lead to me covering more projects
in season two.
And also I've dropped evaluation experts
because it's basically the same issue.
So that's what I'm gonna do there, just do it by myself.
However, since I have an ambush mentality,
if you haven't noticed on this podcast,
where we're trying to confront organizations
that are waving this outcome flag
when they should be burning it
because they're really aren't measuring the outcomes
and they're either doing this fraudulently and knowingly
or they like to claim, oh, we don't have the money
to do it properly or we don't have the technical expertise,
which is another excuse they make.
And I do wanna point out
that this is not some right wing conservative podcast
in the sense that we wanna cancel aid.
Yes, people on the right side of the spectrum
wanna reduce aid relative to the left side of the spectrum,
but the left side of the spectrum should also be concerned
because if they want aid to continue
and receive continued political support
from people like me,
cause I'm on the left side, doesn't matter.
We should all be concerned
that our tax dollars are making a difference, i.e. impact.
So it has to be properly measured and evaluated.
It's not some right wing concept saying
that you want to have your programs that you pay for
properly measured, properly monitored, properly evaluated.
Everybody should do that because in the end,
the irony is that if you don't show that
that the programs are being effective and having impact,
the conservatives will use that
as an excuse to scrap the program.
So it's best that if you believe in it,
that you support it by showing the evidence
that the programs actually work.
So that's important also.
And finally, a few more points.
What we'll do in relation to that is that I will continue to,
if I find some organizations that have multiple projects
and they're doing the same wrong things
where they're not properly measuring the outcomes
using proper outcome indicators
and proper evaluation design,
then if I find that and I find it repeated going on
through various projects for the same organization,
then I may set up an ambush podcast
where I invite them to respond to my critique
and I also will have evaluation experts on board
that agree with me.
So let's be honest about this.
We are confronting these organizations
who are making these claims when these claims
are not being supported by the data that's coming in,
let alone the performance measurement framework
or the activity MEL plan, USAID that they're using.
So we'll still do that,
but I'll be more selective in the organizations I select
to invite to attend a part two of our podcast
where they can respond to the critique that we do.
I will, just like season one,
I will continue to offer to anyone if they want
to email me at evaluatecanadaaidatgmail.com
if they want any of the performance measurement frameworks
from season one, as well as my Excel summary
of the outcome indicators
and how each one of the outcome indicators are flawed.
I'm happy to send those out.
And I did receive a few emails
and I'm happy to continue to do that.
And I will also continue since there's clearly
an activist advocacy intent from this podcast,
which is to get these hundreds of millions of dollars
of projects paid for by the Canadian taxpayer
to start properly evaluating them,
is I will continue to send my critique
and the episodes to the Minister
for International Development,
as well as the shadow critics for the Conservative Party,
the New Democratic Party and the Green Party,
because like I said earlier,
proper evaluation should not be an issue
of the right or the left.
It should be an issue for all Canadians to be concerned,
to make sure that their dollars overseas
are making a difference.
And finally, I wanted to invite another format change
is that there is a growing industry
in the evaluation industry,
especially in international development
for three types of experts that are being used
to evaluate projects, which I'd like to learn more about
because I'm still having issues
with how these three different evaluation techniques
actually properly, validly measure
the achievement of outcomes,
and in particular show that the project has impact.
That is, in the absence of control groups
or comparison groups or repeated measures,
how does the project show that it is actually achieving
impact, making a difference, rather than other factors?
And those three areas of expertise are outcome harvesting,
most significant change, and appreciative inquiry.
So those are three evaluation techniques
that have come up quite frequently on various platforms,
claiming that they are proper ways
of evaluating impact of a project,
or at least showing relevance or coherence,
which we'll talk about in more detail
when we get to reviewing these performance measurement
frameworks or the USAID activity MEL plans.
So if there's anyone out there who has expertise
and would be willing to come on my podcast,
I'd be happy to interview you
so you could explain to our listeners
how outcome harvesting, most significant change,
or appreciative inquiry actually evaluate
these projects properly.
So feel free to email me at evaluatecanadaaid.gmail.com
and we'll set up a time,
because I would love to put that
as a separate episode on the podcast.
So thank you for your time and thank you for listening.
And soon we will begin introducing new projects
from the government of Canada and hopefully from USAID.
Bye for now.