Welcome to the Improving Development Evaluation Podcast.
I'm your host David Wand,
and welcome to episode eight, part two,
where we continue our discussion
of the international development organization
called United Nations Development Program, or UNDP.
You can learn more about the United Nations
Development Program at UNDP.org,
and we're looking at government of Canada funded project
cost to the taxpayer of $30 million in Bangladesh,
entitled Fighting Environmental Degradation
and Promoting Greener Energy.
And in summary, basically going to the Cox's Bazaar
refugee camp, but also communities outside of the camp,
giving them liquefied petroleum gas
to replace their need to go into the sacred bush
and get firewood.
So basically, that's the idea behind the project.
And as I will elaborate further,
there are lots of expected outcomes
from replacing firewood with LPG.
One of them is improved nutritional status
for the children because you can cook more frequently
nutritious meals.
And also, it's a safety issue for women.
They would not have to go out into the forest
and collect firewood, which reduces the risk
or eliminates it if they don't have to go out at all
because they have LPG at home,
reduces the risk of sexual assault and rape.
And the other thing it does is it reduces the harmful smoke
from burning firewood.
So those are some of the expected outcomes
of replacing the need to go out and get firewood
when you have it replaced by LPG.
But another expected outcome I'll talk about
in more detail is you hope,
since we're worried about the environment,
that in the longer term, as I mentioned in part one,
they'll replace the LPG with solar cookers or biogas.
And although that conflicts with perhaps
Canadian LPG companies,
because they're gonna lose business if that happens,
but ideally that's what you'd like to see,
even though the Canadian LPG companies,
assuming they're using Canadian LPG companies
to supply this camp, they're gonna lose a customer.
But in the long run, that's gonna benefit the environment.
So let's get with it.
So some background is on June 10th,
I sent the performance measurement framework
and the critique of the performance measurement framework,
which basically said that of the nine outcome indicators,
measuring the three outcomes
in the performance measurement framework,
I only found two out of the nine
that were properly measuring the outcomes.
And on June 10th, as I said,
I sent it to three departments of the UNDP.
I sent it to the Independent Evaluation Office in New York,
on June 10th, also sent it to
the United Nations Development Program in Bangladesh,
their country office,
because that's where the project is being implemented.
And I also sent it to the United Nations Development Program,
Asia and Pacific Regional Communications Advisor.
On June 18th, I got one email reply
from the Asia and Pacific Regional Communications Advisor,
and he advised me, he was on vacation,
so he couldn't respond.
He told me to email the Bureau
for External Relations and Advocacy, UNDP in New York,
which I did, and he confirmed the right email address.
I sent that also on June 18th,
but as of today, June 28th,
I've received no reply from any of those
three UNDP departments.
And I have also not received any written response
to my critique, even though in the email on June 10th,
I said that was another option.
If they didn't want to be on the podcast,
this part two, to discuss my critique,
they could send a written response,
but I haven't received that.
So on June 28th, which is today,
I'm now going to do part two, unfortunately, without them,
and without a written response.
And then at the end of that, I will, as I usually do,
send it to the Minister for International Development,
as well as the shadow critics from the Conservative Party,
the NDP, and the Bloc Québécois,
and also the Green Party, so everybody knows,
because this is not a political particular affiliation
that you need to have.
Everyone of every political persuasion
should be concerned about the fact
that Canadian government funds
are not being properly evaluated.
So let's get on with it.
So there are nine outcome indicators
measuring the three outcomes.
So we'll start with the first outcome, number one,
and I quote from the performance measurement framework.
Targeted households have safe and sufficient access
to cleaner cooking energy to meet their basic needs.
Problem there is, that's an output.
You're just delivering the LPG.
It's not an outcome.
It's after you deliver the gas,
what are the expected outcomes?
And I've already mentioned
some possible expected outcomes.
So that needs to be changed and reworded.
And it could, to some effect, be, like I've described,
forest areas where refugees harvest firewood is reduced
and replaced by seedlings.
That could be an example of an expected outcome.
But there's other expected outcomes I mentioned earlier.
But let's get to the outcome indicator
for this flawed outcome, number one.
And that outcome indicator, number one,
is percent of households who report
buying or collecting firewood.
The problem there is, it's all full of self-reporting bias,
but even more importantly, it is forget about it.
Don't even measure the indicator.
Don't waste your time asking households
how much firewood they bought or collected.
Go right to the chase and look at some
of the expected outcomes instead.
Weigh the children to see if their nutritional status
has improved.
Ask the women how many times they've had to go out
and collect firewood,
because you expect that to go down,
if they've got the LPG in their house,
in their location in the refugee camp, right?
So those are better indicators rather than just asking them
if they've been collecting firewood.
And the other one is, look at the children or the adults
to see if they have the LPG,
if smoke inhalation-related illnesses
related to burning firewood have gone down.
So those would be the basic things you would look at
instead of running around trying to figure out
how much firewood they've got in their location
or how much they're collecting.
So it's not a very good indicator,
and it creates a lot of waste in terms of measurement.
So you could just do what everybody else is doing,
which is weighing the children anyways, right?
So that's where I recommend instead of that indicator,
focusing on those as alternate indicators.
The second indicator,
this is one of the two that's really good,
and they've mentioned it,
the number of equivalent hectares
of natural forest protected.
That's an excellent indicator.
So I've noted that in the critique,
and they actually have a target of 10,700 hectares
of natural forest protected.
And that's actually in the performance measurement framework.
So that's excellent.
They even indicate in the performance measurement framework
that they will be planting seedlings.
So the key there is to make sure
that that target is reached,
and I'll talk about that a little bit later.
So that's an excellent indicator.
Forget about the first one.
I mean, that's just a waste of money.
And leading on to that, the third indicator
is the number of tons of carbon dioxide emissions abated.
So good luck with that.
I would again say forget about it.
Don't even waste Canadian taxpayer money measuring that.
Focus on the forest that they're going to
to collect firewood and monitor it
by making sure that seedlings are replacing
where they in the past at the baseline
ripped out the wood, chopping down the tree.
The other point is how are you gonna claim
that any reduction in carbon dioxide,
because it's floating in the air,
I don't know how they do this,
how do they claim that it's been abated?
The only way to show that,
and maybe that's what they're implying here,
is by the forest where they used to harvest the firewood
is now being reduced,
and they can monitor that area around the camp to show that.
So that's what they should be doing.
So take out that indicator.
And in fact, in outcome number two, this is what they do,
and I'll read the outcome.
Number two, environment and ecosystems rehabilitated.
Now that's the good part.
The next part is a bit flaky.
To strengthen community resilience to mitigate conflict
over natural resources, disasters, and climate shocks.
The first indicator for that outcome is,
and this is indicator number four,
is the percent of land ecosystems rehabilitated
in Cox's Bazaar through SAFE Plus Two program interventions.
SAFE is the name of the NGO locally
that has been hired by UNDP
to bring in the liquefied petroleum gas.
So that's an excellent indicator.
And in fact, if you look
at the performance measurement framework,
they actually have, I think it's called
geographic information system monitoring.
They use satellite imagery of that specific area
of the forest where the refugees used to go
to get the firewood,
and they're planting seedlings in there
to replace the firewood that was ripped out.
So they just need to track it
and to show that that target of 10,700 hectares
is being protected and sustained.
And that's excellent.
So that's a great indicator.
That's the second indicator, which is pretty good.
And the first one, which is equivalent basically to it,
is the number of equivalent hectares
natural forest protected.
It's basically the same thing.
So it's great that they're doing that.
So that's an excellent indicator.
Indicator number five, again, is a problem.
It's tons of carbon dioxide sequestered
through program interventions.
Again, how are you gonna claim
that the LPG coming in has actually sequestered the carbon?
Now their response could be, duh,
it is the seedlings that we're planting.
Okay, I get it.
But actually measuring the carbon dioxide sequestered,
waste of time and waste of money.
Just focus on the forest around the refugee camp,
which they have done, and show that it's been sustained.
But also, as I mentioned earlier,
look at the other expected outcomes from bringing in the LPG.
And then this is where it gets really a lot of waste
and fun and time and money.
The next indicator they've got
on the environment and ecosystems rehabilitated to, quote,
strengthen community resilience to mitigate conflict
over natural resources, disasters, and climate shocks.
The next indicator they have is, quote,
resilience capacity score increased compared to baseline.
The resilience capacity score,
it measures households' perception
of their resilience capacities to generic
or country-specific shocks and stressors.
And this again is where I say forget about it.
Don't ask about their perception
on fighting over firewood.
Don't do it.
Just look at, instead of asking for their perception
to become resilient, which is a waste of time
and money in measuring it,
it's also got self-reporting bias.
Weigh the children, because you're expecting
their nutritional status to go up.
Inspect the children, for lack of a better word,
to see if the smoke inhalation-related illnesses
related to burning firewood has gone down.
So all the symptoms of that have been reduced.
Also ask the women if they've had to go out
and get firewood, because if they,
of course they could lie,
but what you wanna look at is the frequency
of reporting going out and collecting firewood
has gone down.
Those are better measures, direct objective measures
to see that the LPG coming in is actually having
the desired outcome, right?
Forget about this resilience capacity score.
You don't need it.
So there's another outcome number three,
which is the resilience of Rohingya refugees,
especially women, is sustained.
And there they've got an indicator,
which is the resilience capacity score
is increased compared to the baseline.
Again, you could ask the women what their perception is
of fear of going out and collecting firewood,
but even better, just get to the frequency
that they're going out and collecting firewood,
because hopefully it's gone down.
You could even go a little step further,
but I wouldn't even try it,
which is looking at the number of reported,
because that's tricky,
reported incidents of sexual assault and rape
that have occurred in the area where the firewood
is being collected, right?
It's better to focus on the outcomes I described earlier.
And another indicator they've got,
this is indicator number eight,
is the average reduced coping strategy index.
The coping strategies index is a tool
that measures what people do
when they cannot access enough food.
It is a series of questions about how households
manage to cope with a shortfall in food for consumption
and results in a simple numeric score.
Now the problem here again is you have to look at the output.
Now in addition to bringing in liquefied petroleum gas,
taxpayers from Canada, $30 million,
also are spending money in this project on cash.
That is giving cash to buy food in the refugee camp
and also cash, I mean from the $30 million,
to train the refugees in growing their own food
in a community garden.
And they only measure this average reduced
coping strategy index once a year.
That's not good enough, that's the first problem.
The second problem is, hello,
if you're giving them cash to buy food,
don't worry about this coping strategy index,
figure out if they're buying the food with it
and you can do that at the camp.
I happen to be at the camp.
I've seen the World Food Program in action.
The refugees have a digital card
where they go into stores inside the camp and buy food
and it's registered on these cards.
So the World Food Program has detailed data
on the amount of food that's actually been bought.
And again, like I said earlier,
they can also weigh the children in these families
to see if their nutritional status is going up.
There's no need to have this average reduced
coping strategy index measured,
especially just once a year.
It's a waste of time and a waste of money
because the idea here is to increase
their nutritional status.
So you do it every six months, I would recommend.
Forget about this strategy index
and measure the amount of food they've actually bought
and or weigh the children.
And you can even weigh the adults to show
this is standard nutritional status measures.
You don't have to have this funky,
flaky coping strategy index.
Also measure the garden crop yields if you want
on the training on how to grow crops.
But I would even argue who cares about that.
Go to the end expected outcome,
which is increased weight gain,
nutritional status of the children.
And also the cash purchases of the food items.
I'm repeating myself here, but that's the idea.
So no need for this coping strategies index,
especially when the key outputs in this project are LPG
and cash for food and training to grow food.
So just measure the crop yields, food consumption,
and the weights of the children.
And that would be good enough.
The final indicator on this outcome
of the resilience of Rohingya refugees,
especially women is sustained.
Final outcome indicator is percent of households
with acceptable food consumption score.
Now again, it's only measured once per year.
And I would say you need to measure it more frequently.
And it's a consumption score.
Maybe they use the data from the World Food Program
to be fair to them, but we don't know
because they haven't given me a written response
or showed up for this part too.
So maybe they measure acceptable food consumption
with the data that comes from the World Food Program.
I'm not, I just don't know.
It's only measured once per year.
I would rather go in and measure every six months
the weights of the children.
And that would show their nutritional status going up.
So that's the end of episode eight, part two.
What I'm gonna do now is send this to the minister
and the shadow critics as well as UNDP.
I will copy them on that email so they're aware.
And if you are interested in receiving
the performance measurement framework
and the critique that I've verbally just described here,
you can send me an email at evaluatecanadaaidatgmail.com.
And I'd be happy to send it to you.
I can also send you any of the other
performance measurement frameworks for season one.
This ends season one for the podcast.
I will be returning in September for season two.
And before I do that, I will be doing one more episode,
sort of a prologue of season one,
which gives you an idea of how season two
is going to be slightly different than season one
and give you some hints as to what I'm expanding on
and maybe what I would be refocusing on.
And I'll probably launch that in a few weeks.
Thank you for listening and thank you for the feedback.
I've received some very positive emails
regarding potential learning opportunities on the podcast,
which I'm thinking about, but I'll elaborate more on that
in the prologue to season one.
So that's it.
Thank you for listening and stay tuned
for the prologue for season one and also season two,
which will start up again in September.
Bye for now.