1
00:00:00,000 --> 00:00:04,400
Welcome to the Improving Development Evaluation Podcast.

2
00:00:04,400 --> 00:00:10,200
I'm your host David Wand and welcome to season 1, episode 1, part 1.

3
00:00:10,200 --> 00:00:15,280
In this episode we're going to introduce you to an international development organization.

4
00:00:15,280 --> 00:00:20,940
That international development organization is Care Canada.

5
00:00:20,940 --> 00:00:28,000
You can learn more about Care Canada if you go to their website at www.care.ca.

6
00:00:28,000 --> 00:00:34,000
Their headquarters is located in Ottawa, Canada.

7
00:00:34,000 --> 00:00:39,200
The project we're going to be talking about that Care Canada has been funded by the government

8
00:00:39,200 --> 00:00:46,000
of Canada to deliver is called Advancing Women's Economic Empowerment.

9
00:00:46,000 --> 00:00:51,460
This project is being delivered in Vietnam and the cost of this project to the Canadian

10
00:00:51,460 --> 00:00:56,100
taxpayer is $3,400,000.

11
00:00:56,100 --> 00:01:01,600
And in this part 1 I'm going to give you a brief description of the project services

12
00:01:01,600 --> 00:01:07,440
that are being delivered and a critique of Care Canada's performance measurement framework

13
00:01:07,440 --> 00:01:11,180
that they have developed to measure the project's performance.

14
00:01:11,180 --> 00:01:17,360
That is, the achievement of the project's outcomes using the outcome indicators in that

15
00:01:17,360 --> 00:01:20,120
performance measurement framework.

16
00:01:20,120 --> 00:01:24,520
But before I describe the project services that are being delivered for this project

17
00:01:24,520 --> 00:01:33,160
by Care Canada and their partners in Vietnam, I'm going to go to my summary and conclude

18
00:01:33,160 --> 00:01:38,780
that in this performance measurement framework for this project there are 19 outcome indicators

19
00:01:38,780 --> 00:01:42,000
that are measuring 9 outcomes.

20
00:01:42,000 --> 00:01:48,560
My review of all 19 outcome indicators has led me to conclude that all of these indicators

21
00:01:48,560 --> 00:01:51,120
are not measuring the outcomes.

22
00:01:51,120 --> 00:01:57,640
So the organization, Care Canada, cannot make the claim that it's achieving its project

23
00:01:57,640 --> 00:02:03,520
outcomes with the services they're delivering in the project simply because their outcome

24
00:02:03,520 --> 00:02:09,120
indicators are not properly measuring the outcomes for this project.

25
00:02:09,120 --> 00:02:14,660
As you probably know from the trailer for this podcast, those 19 outcome indicators

26
00:02:14,660 --> 00:02:18,240
fall into one of five problem areas.

27
00:02:18,240 --> 00:02:26,080
In this project, 11 of them, those outcome indicators, fall into the how do you feel

28
00:02:26,080 --> 00:02:31,800
self-reporting problem where people are asked how they feel about achieving the outcome

29
00:02:31,800 --> 00:02:38,040
rather than being objectively measured on whether they achieved that outcome.

30
00:02:38,040 --> 00:02:43,440
Another seven of the outcome indicators fall into problem area number five, which is there's

31
00:02:43,440 --> 00:02:50,040
no comparison group of people outside of the project that are also measured on the same

32
00:02:50,040 --> 00:02:56,280
outcome, which means even if the people in the project are achieving the outcomes, we

33
00:02:56,280 --> 00:03:00,800
don't know if they're achieving them any better than another group of people that could

34
00:03:00,800 --> 00:03:04,840
also achieve the outcome outside of the project.

35
00:03:04,840 --> 00:03:10,360
And finally, we have one indicator that is falling into problem number four, where it's

36
00:03:10,360 --> 00:03:14,840
simply not being measured frequently enough.

37
00:03:14,840 --> 00:03:19,320
If you wish to receive Care Canada's performance measurement framework for this project and

38
00:03:19,320 --> 00:03:28,320
my summary of the outcome indicators for this project, send me an email to evaluatecanadaaid

39
00:03:28,320 --> 00:03:32,800
at gmail.com and I would be happy to send those documents to you.

40
00:03:32,800 --> 00:03:38,260
I plan to post all of these documents for each episode on a website that is currently

41
00:03:38,260 --> 00:03:39,920
being developed.

42
00:03:39,920 --> 00:03:46,000
So now that I've concluded that the PMF is flawed, let's get into more detail about the

43
00:03:46,000 --> 00:03:47,560
project services.

44
00:03:47,560 --> 00:03:52,600
So there are four target groups that this project delivers services to.

45
00:03:52,600 --> 00:03:59,320
The first target group are poor, rural, ethnic minority women, and the services being delivered

46
00:03:59,320 --> 00:04:06,800
to them are training on negotiation and household decision making, action plans being developed

47
00:04:06,800 --> 00:04:14,480
for them to promote their economic rights and are promoted by male champions.

48
00:04:14,480 --> 00:04:19,440
Dialogues are held between these women and men on household decision making and gender

49
00:04:19,440 --> 00:04:23,560
based violence and unequal caregiving burden.

50
00:04:23,560 --> 00:04:31,460
These dialogues are also held on the same topics between these women and communal officers.

51
00:04:31,460 --> 00:04:36,080
Community events are also held, attended by women and men to promote the economic rights

52
00:04:36,080 --> 00:04:37,880
of these women.

53
00:04:37,880 --> 00:04:42,360
And gender based violence services that are available for these women are promoted to

54
00:04:42,360 --> 00:04:43,800
these women.

55
00:04:43,800 --> 00:04:49,800
And finally, a feasibility study is completed on child care services and household time

56
00:04:49,800 --> 00:04:52,360
saving appliances.

57
00:04:52,360 --> 00:04:57,020
And then there's a variety of trainings that are also delivered to these women, such as

58
00:04:57,020 --> 00:05:04,480
training on financial literacy, savings group operations, digital record keeping, and business

59
00:05:04,480 --> 00:05:06,280
plan development.

60
00:05:06,280 --> 00:05:12,880
These women also receive unpaid care time saving options.

61
00:05:12,880 --> 00:05:19,180
And also studies and policy briefs are produced on the benefits of reducing unpaid work for

62
00:05:19,180 --> 00:05:21,620
these women.

63
00:05:21,620 --> 00:05:27,100
Training is also delivered to these women on climate appropriate agricultural practices

64
00:05:27,100 --> 00:05:32,000
and on high quality tea production and safe migration.

65
00:05:32,000 --> 00:05:36,880
A second target group in this project delivers services.

66
00:05:36,880 --> 00:05:43,100
And that target group are civil society organizations and media organizations.

67
00:05:43,100 --> 00:05:50,280
The services they provide are they promote the economic rights for these poor rural ethnic

68
00:05:50,280 --> 00:05:56,620
minority women, such as including success stories of women entrepreneurs, which are

69
00:05:56,620 --> 00:05:59,400
promoted in the media.

70
00:05:59,400 --> 00:06:05,160
These civil society organizations and media organizations also attend learning events

71
00:06:05,160 --> 00:06:12,920
paid for by this project, where these civil society organizations and media organizations

72
00:06:12,920 --> 00:06:20,160
are expected to learn on how to work with these women whose economic rights they're

73
00:06:20,160 --> 00:06:21,160
promoting.

74
00:06:21,160 --> 00:06:26,320
These civil society organizations and media organizations are also expected to produce

75
00:06:26,320 --> 00:06:28,560
documentation.

76
00:06:28,560 --> 00:06:35,920
During the promotion of these women's economic rights, this documentation is sent to policymakers.

77
00:06:35,920 --> 00:06:42,600
Also round tables are organized by this project between these civil society organizations

78
00:06:42,600 --> 00:06:49,000
and policymakers on how to promote the economic rights of these women.

79
00:06:49,000 --> 00:06:53,000
The third target group are private sector individuals.

80
00:06:53,000 --> 00:06:59,000
And they receive from this project training on women's economic rights and gender responsive

81
00:06:59,000 --> 00:07:01,360
value chain approaches.

82
00:07:01,360 --> 00:07:07,240
And finally, a fourth target group are what we call duty bearers or what they refer to

83
00:07:07,240 --> 00:07:12,200
as village leaders and communal officers.

84
00:07:12,200 --> 00:07:17,820
The services the project delivers to them is training on gender responsive programs

85
00:07:17,820 --> 00:07:19,280
for women.

86
00:07:19,280 --> 00:07:22,560
So that gives you an idea of the services the project delivers.

87
00:07:22,560 --> 00:07:26,120
Now, there's nine outcomes in this project.

88
00:07:26,120 --> 00:07:33,800
And they are outcome number one, increased entrepreneurship capacity of women led grassroots

89
00:07:33,800 --> 00:07:39,360
social enterprises to pursue economic opportunities in Vietnam.

90
00:07:39,360 --> 00:07:46,520
Outcome number two, increased gender responsive capacity of poor rural and economic minorities,

91
00:07:46,520 --> 00:07:52,000
especially women to save and borrow money to support productive and sustainable economic

92
00:07:52,000 --> 00:07:54,280
activities in Vietnam.

93
00:07:54,280 --> 00:08:00,880
Outcome number three, increased gender responsive capacity of poor rural and ethnic minorities,

94
00:08:00,880 --> 00:08:07,400
especially women to earn income from climate smart and sustainable on farm and off farm

95
00:08:07,400 --> 00:08:09,440
livelihoods in Vietnam.

96
00:08:09,440 --> 00:08:15,080
Outcome number four, increased willingness of poor rural and ethnic minority women and

97
00:08:15,080 --> 00:08:23,960
men duty bearers, in particular, commune officers to address the issue of women's unequal caregiving

98
00:08:23,960 --> 00:08:27,020
burden and economic rights in Vietnam.

99
00:08:27,020 --> 00:08:32,960
Outcome number five, increased ability of ethnic minority women to meaningfully participate

100
00:08:32,960 --> 00:08:39,160
and make decisions related to economic activities at the household level and in their livelihoods.

101
00:08:39,160 --> 00:08:45,040
Outcome number six, increased capacity of civil society organizations, including women's

102
00:08:45,040 --> 00:08:51,280
rights organizations, private sector companies and media organizations to promote poor rural

103
00:08:51,280 --> 00:08:55,280
and ethnic minority women economic rights in Vietnam.

104
00:08:55,280 --> 00:09:02,040
Outcome number seven, increased participation by poor and ethnic minority women in paid

105
00:09:02,040 --> 00:09:04,280
economic activities.

106
00:09:04,280 --> 00:09:10,160
Outcome number eight, enhanced promotion, economic rights for poor rural women, especially

107
00:09:10,160 --> 00:09:12,440
ethnic minority women in Vietnam.

108
00:09:12,440 --> 00:09:19,640
And finally, outcome number nine, improved economic well-being for poor rural women,

109
00:09:19,640 --> 00:09:22,960
especially ethnic minority women in Vietnam.

110
00:09:22,960 --> 00:09:28,920
So we can't go into all the outcome indicators, but I'm going to give you just three of those

111
00:09:28,920 --> 00:09:34,720
outcome indicators as to why they fall into one of those five problem areas that we outlined

112
00:09:34,720 --> 00:09:36,080
in the trailer.

113
00:09:36,080 --> 00:09:42,040
So in part two of this episode, where Care Canada will be invited to respond to this

114
00:09:42,040 --> 00:09:47,360
critique, we will cover all nine outcomes with at least one outcome indicator to be

115
00:09:47,360 --> 00:09:49,380
discussed for each outcome.

116
00:09:49,380 --> 00:09:54,420
If we have more time, great, we'll go into further outcome indicators, but we're going

117
00:09:54,420 --> 00:09:59,320
to focus on making sure we cover at least one outcome indicator for each of the nine

118
00:09:59,320 --> 00:10:05,680
outcomes so you get an idea of right across the entire spectrum of outcomes, the nine

119
00:10:05,680 --> 00:10:07,560
of them for this project.

120
00:10:07,560 --> 00:10:12,920
At the end of part two, the evaluation panel, that's me and two evaluation experts, will

121
00:10:12,920 --> 00:10:19,480
be concluding as to whether Care Canada can make the claim that its project services are

122
00:10:19,480 --> 00:10:21,860
achieving its project outcomes.

123
00:10:21,860 --> 00:10:27,920
As host, I conclude Care Canada cannot make this claim, but I want our listeners to hear

124
00:10:27,920 --> 00:10:33,700
what Care Canada has to say as well as the other evaluation experts.

125
00:10:33,700 --> 00:10:39,640
So let's look at the outcome increased capacity of civil society organizations, including

126
00:10:39,640 --> 00:10:45,160
women's rights organizations, private sector companies, and media organizations to promote

127
00:10:45,160 --> 00:10:50,600
poor, rural, and ethnic minority women's economic rights in Vietnam.

128
00:10:50,600 --> 00:10:51,960
So that's the outcome.

129
00:10:51,960 --> 00:10:59,080
One of the indicators to measure this increased capacity to promote is number of community,

130
00:10:59,080 --> 00:11:05,120
civil society organizations, media organizations, and network members, private sector companies,

131
00:11:05,120 --> 00:11:13,200
who report an improvement in their perception, opinion related to ethnic minority women's

132
00:11:13,200 --> 00:11:15,120
economic rights.

133
00:11:15,120 --> 00:11:20,080
You probably can tell already from this indicator that it's flawed, but it's interestingly

134
00:11:20,080 --> 00:11:22,600
flawed in two ways.

135
00:11:22,600 --> 00:11:28,160
First, reporting on the number of civil society organizations who report an improvement in

136
00:11:28,160 --> 00:11:34,920
their perception or opinion related to ethnic minority women's economic rights, it's got

137
00:11:34,920 --> 00:11:39,600
nothing to do with increased capacity to promote their rights.

138
00:11:39,600 --> 00:11:46,320
They're just asking them for their opinion, but the outcome is increased capacity to promote.

139
00:11:46,320 --> 00:11:50,960
And if you look at the project services, they are expected to improve their capacity to

140
00:11:50,960 --> 00:11:57,800
promote because if you remember from the description of the project, they are showing up at a cost

141
00:11:57,800 --> 00:12:04,520
to you, the Canadian taxpayer to learning events where they're expected to learn how

142
00:12:04,520 --> 00:12:09,560
to work with the women whose economic rights they're promoting.

143
00:12:09,560 --> 00:12:13,680
They're also being paid by you, the Canadian taxpayer to promote the economic rights of

144
00:12:13,680 --> 00:12:14,680
these women.

145
00:12:14,680 --> 00:12:19,400
And of course, they are going to say they have improved their opinion on promoting these

146
00:12:19,400 --> 00:12:21,920
women's economic rights.

147
00:12:21,920 --> 00:12:27,480
They're also expected to produce documentation on how they're promoting these women's economics

148
00:12:27,480 --> 00:12:34,400
rights with that documentation being sent to policymakers also at a cost to the Canadian

149
00:12:34,400 --> 00:12:41,680
taxpayer and are showing up to round tables where they sit with policymakers on how they're

150
00:12:41,680 --> 00:12:45,320
going to promote women's economic rights.

151
00:12:45,320 --> 00:12:50,160
So it's clear the project funds are being spent to increase their capacity, technical

152
00:12:50,160 --> 00:12:53,720
capacity on how best to promote.

153
00:12:53,720 --> 00:12:56,480
But the outcome indicator is not measuring that at all.

154
00:12:56,480 --> 00:13:01,200
It's just measuring how they feel, their perception, their opinion.

155
00:13:01,200 --> 00:13:02,820
We don't want their opinion.

156
00:13:02,820 --> 00:13:04,100
We don't want their perception.

157
00:13:04,100 --> 00:13:08,900
We want to know if they're technically capable, thanks to all that money spent showing up

158
00:13:08,900 --> 00:13:14,360
at these learning events, showing up at those round tables, producing documentation that

159
00:13:14,360 --> 00:13:18,040
shows they know how to promote the economic rights of these women.

160
00:13:18,040 --> 00:13:22,440
And more importantly, that they've significantly increased their capacity.

161
00:13:22,440 --> 00:13:25,120
So the indicators got nothing to do with the outcome.

162
00:13:25,120 --> 00:13:27,880
That's the first problem.

163
00:13:27,880 --> 00:13:33,740
Even if capacity is defined not as technical capacity to promote, but rather an improved

164
00:13:33,740 --> 00:13:40,160
willingness to promote, they are going to report yes, because they are paid by the project

165
00:13:40,160 --> 00:13:42,380
to promote.

166
00:13:42,380 --> 00:13:44,200
And even you can go a step further.

167
00:13:44,200 --> 00:13:47,640
Let's say we've got an objective measure that they've improved their willingness.

168
00:13:47,640 --> 00:13:52,080
There could be another group of civil society organizations out there in another part of

169
00:13:52,080 --> 00:13:57,440
the country that have got nothing to do with the project and their willingness to promote

170
00:13:57,440 --> 00:13:59,240
has also gotten.

171
00:13:59,240 --> 00:14:07,640
So we'll never know on several levels whether these organizations have increased their technical

172
00:14:07,640 --> 00:14:11,320
capacity or even their willingness to promote.

173
00:14:11,320 --> 00:14:17,640
Another indicator we're going to look at is for the outcome enhanced promotion of economic

174
00:14:17,640 --> 00:14:22,480
rights for poor rural women, especially ethnic minority women in Vietnam.

175
00:14:22,480 --> 00:14:28,720
One of the indicators for that is the average total number and proportion of daily hours

176
00:14:28,720 --> 00:14:32,680
spent on unpaid domestic care work.

177
00:14:32,680 --> 00:14:36,520
Now if you remember from the project services, one of the things they deliver to these women

178
00:14:36,520 --> 00:14:41,840
is unpaid care time saving options.

179
00:14:41,840 --> 00:14:47,120
We don't know what those options are, but we suspect and we logically expect them to

180
00:14:47,120 --> 00:14:52,820
reduce the number of hours spent on unpaid domestic work for these women.

181
00:14:52,820 --> 00:14:57,640
If they're given these unpaid care time saving options.

182
00:14:57,640 --> 00:14:59,620
Unpaid care time saving options.

183
00:14:59,620 --> 00:15:03,840
It means they don't have to spend as much time on unpaid care themselves.

184
00:15:03,840 --> 00:15:06,400
Sounds good to me.

185
00:15:06,400 --> 00:15:09,200
Problem is we don't have any comparison group.

186
00:15:09,200 --> 00:15:15,240
What about another group of women who are not in the project in some other part of the

187
00:15:15,240 --> 00:15:21,360
country in Vietnam who could say, ah, let's get together and set up a communal child care

188
00:15:21,360 --> 00:15:22,360
center.

189
00:15:22,360 --> 00:15:26,480
We'll all send our children there and just one of us will watch all the children together

190
00:15:26,480 --> 00:15:31,160
collectively while the rest of us reduce our hours of unpaid child care.

191
00:15:31,160 --> 00:15:36,960
So we also as a group perform well on the outcome indicator with our hours also going

192
00:15:36,960 --> 00:15:41,600
down like the project group of women.

193
00:15:41,600 --> 00:15:47,000
So there's nothing wrong with the indicator, but if Care Canada wants to claim that their

194
00:15:47,000 --> 00:15:53,520
project service of delivering these time saving options is making a greater reduction in hours

195
00:15:53,520 --> 00:15:59,480
spent on unpaid child care by this service, they have to compare it with another group

196
00:15:59,480 --> 00:16:01,400
outside of the project.

197
00:16:01,400 --> 00:16:06,340
And finally, one more indicator we're going to look at is related to the outcome improve

198
00:16:06,340 --> 00:16:12,440
economic well-being for poor rural women, especially ethnic minority women in Vietnam.

199
00:16:12,440 --> 00:16:18,720
One of the indicators there in the performance measurement framework is percent and number

200
00:16:18,720 --> 00:16:25,760
of poor rural and ethnic minority women and men who report an improvement in their economic

201
00:16:25,760 --> 00:16:26,760
well-being.

202
00:16:26,760 --> 00:16:34,420
Now, we don't know exactly what Care Canada asks them in terms of economic well-being.

203
00:16:34,420 --> 00:16:39,560
So to be fair, maybe at the podcast part two, they'll bring up something like a world bank

204
00:16:39,560 --> 00:16:42,120
survey or something, but we just don't know.

205
00:16:42,120 --> 00:16:46,640
So what I'm suggesting here is maybe they should measure their incomes or measure their

206
00:16:46,640 --> 00:16:52,620
assets rather than asking them to self-report whether they feel their economic well-being

207
00:16:52,620 --> 00:16:54,080
has improved.

208
00:16:54,080 --> 00:16:58,200
Maybe it's too intrusive, but I would think measuring their incomes or assets over time

209
00:16:58,200 --> 00:17:02,440
before, during, and after the project would be a better measure of showing that their

210
00:17:02,440 --> 00:17:05,520
actual economic well-being has improved.

211
00:17:05,520 --> 00:17:10,080
That world bank survey that you often hear about with the percentages of those living

212
00:17:10,080 --> 00:17:13,960
on $2 a day or less comes to mind.

213
00:17:13,960 --> 00:17:18,920
And again, even if their incomes or assets increase for the project women, what about

214
00:17:18,920 --> 00:17:24,680
a comparison group of women not in the project and measuring their incomes or assets as well

215
00:17:24,680 --> 00:17:28,840
over the same time period so you can compare the two groups.

216
00:17:28,840 --> 00:17:33,720
So those are just three examples from three outcome indicators that fall into these problem

217
00:17:33,720 --> 00:17:35,540
areas.

218
00:17:35,540 --> 00:17:43,920
If you have any comments or suggestions for this episode, feel free to email me at evaluatecanadaaid.gmail.com.

219
00:17:43,920 --> 00:17:49,120
And if there are any evaluation experts out there who'd like to be on the podcast to look

220
00:17:49,120 --> 00:17:54,400
at other performance measurement frameworks and in particular the outcome indicators,

221
00:17:54,400 --> 00:17:59,160
just like I have here, email me and I'd be happy to send you the remaining performance

222
00:17:59,160 --> 00:18:04,600
measurement frameworks that we have lined up for season one where we don't have evaluation

223
00:18:04,600 --> 00:18:06,200
experts so far.

224
00:18:06,200 --> 00:18:12,480
Right now that's eight performance measurement frameworks that we're looking for experts

225
00:18:12,480 --> 00:18:14,340
to comment on.

226
00:18:14,340 --> 00:18:18,880
What we're going to do now is invite a Care Canada representative to attend part two of

227
00:18:18,880 --> 00:18:24,700
this episode to respond to this critique here in part one, along with two evaluation experts

228
00:18:24,700 --> 00:18:27,840
that will attend part two of this podcast.

229
00:18:27,840 --> 00:18:43,280
Thank you for listening and stay tuned for Care Canada, part two, episode one, season one.