1
00:00:00,000 --> 00:00:04,800
I'm your host David Wand and welcome to episode six, part one.

2
00:00:05,320 --> 00:00:09,960
In this episode six, part one, I'll be introducing you to a new

3
00:00:10,200 --> 00:00:15,320
international development organization and in particular, one of their projects.

4
00:00:16,520 --> 00:00:22,120
And giving you a summary of the issues with that performance measurement

5
00:00:22,120 --> 00:00:28,160
framework, along with a description of the services that the project is delivering.

6
00:00:28,160 --> 00:00:35,040
So that new international development organization is colleges and institutes Canada.

7
00:00:35,680 --> 00:00:40,240
You can learn more about this international development organization at their website,

8
00:00:40,720 --> 00:00:45,960
www.collegesinstitutes.ca.

9
00:00:46,880 --> 00:00:49,760
The project is entitled, A Thousand Women.

10
00:00:49,920 --> 00:00:53,280
I am woman, I exist, I participate.

11
00:00:53,280 --> 00:00:58,280
The cost of this project to the Canadian taxpayer is $18 million.

12
00:00:59,400 --> 00:01:05,320
And this project services are being delivered in Senegal, in West Africa.

13
00:01:06,120 --> 00:01:10,320
I've acquired the performance measurement framework for this project through an

14
00:01:10,320 --> 00:01:16,000
access to information request, because currently it is not available to the public.

15
00:01:16,320 --> 00:01:20,720
Although I am advocating for these performance measurement frameworks to be

16
00:01:20,720 --> 00:01:24,640
put on the government of Canada project browser website.

17
00:01:25,720 --> 00:01:31,000
Before I describe the project in detail as to what the services are they deliver, I

18
00:01:31,120 --> 00:01:38,680
will come to the conclusion that of the nine outcomes in this project, there are 17

19
00:01:38,920 --> 00:01:45,120
outcome indicators and only one of those outcome indicators properly measures

20
00:01:45,120 --> 00:01:47,920
whether or not the outcome has been achieved.

21
00:01:47,920 --> 00:01:54,040
Therefore, I have led to conclude that this performance measurement framework in its

22
00:01:54,040 --> 00:02:00,320
current design is unable to support any claim by colleges and institutes Canada

23
00:02:00,640 --> 00:02:03,280
that this project is achieving its outcome.

24
00:02:03,280 --> 00:02:10,720
So briefly, before I get into the description of the project, if you wish to receive the

25
00:02:10,720 --> 00:02:16,920
performance measurement framework for this project, as well as the summary of the

26
00:02:16,920 --> 00:02:23,560
outcome indicators, all 17 and how each one of them fails to properly measure the

27
00:02:23,560 --> 00:02:31,440
outcome for the project, you can email me at evaluatecanadaaid at gmail.com.

28
00:02:31,880 --> 00:02:37,240
And I'll make sure to put that information into the notes that are associated with

29
00:02:37,240 --> 00:02:40,080
this podcast, part one, episode six.

30
00:02:40,080 --> 00:02:51,600
So for this project of $18 million, there are 11 target groups that receive a variety

31
00:02:52,680 --> 00:02:54,280
of services from this project.

32
00:02:54,600 --> 00:02:56,440
So let me go through them with you.

33
00:02:56,600 --> 00:03:01,680
The first target group are male leaders who receive training in leadership.

34
00:03:02,600 --> 00:03:08,040
The second target group are vulnerable women and they are mentored by women leaders.

35
00:03:08,040 --> 00:03:11,280
And they're also trained by women leaders on leadership.

36
00:03:11,760 --> 00:03:17,000
And they also have partnerships established and support measures to support their

37
00:03:17,000 --> 00:03:20,880
employment, integration and business development.

38
00:03:23,000 --> 00:03:27,200
They also receive training on female entrepreneurship.

39
00:03:28,800 --> 00:03:34,000
The third target group is male leaders and women leaders who are also trained on

40
00:03:34,000 --> 00:03:35,000
leadership.

41
00:03:35,000 --> 00:03:37,840
The fourth target group is communities.

42
00:03:38,000 --> 00:03:42,520
They receive campaigns that promote, I quote, gender equality.

43
00:03:43,080 --> 00:03:48,240
The fifth target group are associations of entrepreneurs and business people.

44
00:03:48,680 --> 00:03:53,760
For this target group of associations and entrepreneurs and business people, they are

45
00:03:53,800 --> 00:03:57,960
promoting to this target group, quote, the participation of women.

46
00:03:58,760 --> 00:04:02,960
The sixth target group, they refer to as target entrepreneurs.

47
00:04:02,960 --> 00:04:05,680
They refer to as target audiences.

48
00:04:06,240 --> 00:04:11,720
To these audiences, awareness campaigns are delivered to, quote, raise awareness of the

49
00:04:11,720 --> 00:04:17,080
importance of women's participation in all decision making bodies.

50
00:04:17,760 --> 00:04:20,560
The seventh target group are regions.

51
00:04:20,560 --> 00:04:27,560
And in these regions, advocacy actions, quote, to promote women as full partners in the

52
00:04:27,560 --> 00:04:30,120
development of communities are reached.

53
00:04:30,120 --> 00:04:36,920
Target group eight are the actual project staff and they receive training to support

54
00:04:37,240 --> 00:04:41,400
the women target groups in accessing the project services.

55
00:04:41,640 --> 00:04:47,280
That is access to training on leadership, entrepreneurship and integration into

56
00:04:47,280 --> 00:04:48,280
employment.

57
00:04:48,800 --> 00:04:55,200
The ninth target group are civil society organizations, women's groups and economic

58
00:04:55,200 --> 00:04:56,200
interest groups.

59
00:04:56,200 --> 00:05:02,200
They receive a consultation mechanism between these three groups that is put in place

60
00:05:02,200 --> 00:05:03,200
and is functional.

61
00:05:04,080 --> 00:05:09,880
The 10th target group are government partners where they also receive a consultation

62
00:05:09,880 --> 00:05:15,040
mechanism between these government partners that is put in place and is functional.

63
00:05:15,520 --> 00:05:21,480
And finally, the 11th target group are teachers and directors of training centers.

64
00:05:21,480 --> 00:05:27,000
These teachers are trained on the mechanism for validating acquired experience.

65
00:05:27,320 --> 00:05:31,280
They're also trained on the, and I quote, gender approach.

66
00:05:31,600 --> 00:05:36,320
So that gives you an idea of the description of the services that are delivered under this

67
00:05:36,320 --> 00:05:40,640
project, as well as the target groups that receive these services.

68
00:05:40,720 --> 00:05:47,160
And as I mentioned earlier, when we reviewed the 17 outcome indicators, only one of those

69
00:05:47,160 --> 00:05:54,520
17 outcome indicators was designed in such a way that we could claim that the outcome

70
00:05:54,520 --> 00:05:58,440
indicator for the outcome was properly measuring that outcome.

71
00:05:58,920 --> 00:06:06,120
So for part two, I'll be inviting an evaluation expert to attend to go through those

72
00:06:06,120 --> 00:06:09,080
outcome indicators, at least nine of them.

73
00:06:09,080 --> 00:06:14,440
So we make sure we cover all nine outcomes and critique each outcome indicator to explain

74
00:06:14,440 --> 00:06:18,040
in detail how it fails to properly measure the outcome.

75
00:06:18,280 --> 00:06:24,600
We will also be inviting a representative from colleges and institutes Canada to attend

76
00:06:24,600 --> 00:06:27,960
part two of the podcast to respond to our critique.

77
00:06:28,000 --> 00:06:32,880
And just to reiterate, there are nine outcomes for this project from the performance

78
00:06:32,880 --> 00:06:38,360
measurement framework, and there are 17 outcome indicators that attempt to measure the

79
00:06:38,360 --> 00:06:40,400
achievement of those nine outcomes.

80
00:06:40,400 --> 00:06:48,880
Before I finish, I'm going to look just at three outcome indicators and explain why they

81
00:06:48,880 --> 00:06:51,680
fail to properly measure the outcomes.

82
00:06:51,720 --> 00:06:56,200
So one of the outcomes in the performance measurement framework is, and I quote,

83
00:06:56,360 --> 00:07:02,400
strengthening the empowerment of women among the most vulnerable across Senegal.

84
00:07:03,200 --> 00:07:07,040
And they use what's called the female empowerment framework.

85
00:07:07,040 --> 00:07:12,320
What is fantastic about this measurement is they actually do use a comparison group

86
00:07:12,480 --> 00:07:18,320
that is women who are not in this project receiving that spectrum of services that I

87
00:07:18,320 --> 00:07:19,680
described to you earlier.

88
00:07:20,160 --> 00:07:25,840
And they're also measuring their levels of female empowerment before the project starts,

89
00:07:26,080 --> 00:07:28,320
as well as after the project starts.

90
00:07:28,320 --> 00:07:32,960
However, there are two problems with the measurement of women.

91
00:07:32,960 --> 00:07:37,120
Even though they're using a comparison group, which is a good sign.

92
00:07:37,120 --> 00:07:40,240
First of all, they're only measuring it once per year.

93
00:07:40,480 --> 00:07:46,000
So this means, as I mentioned in my trailer, it would be very easy for the project women

94
00:07:46,480 --> 00:07:53,360
to achieve their outcomes by themselves in between the trainings that they receive in

95
00:07:53,360 --> 00:07:54,240
the project.

96
00:07:55,280 --> 00:07:57,840
Now, you could compare that with the comparison group.

97
00:07:57,840 --> 00:08:03,920
But it would be better if they did it more frequently.

98
00:08:03,920 --> 00:08:10,160
But the second reason there's a problem with this indicator, even though they're using

99
00:08:10,160 --> 00:08:16,720
a comparison group, is the female empowerment index looks at six dimensions.

100
00:08:16,720 --> 00:08:22,400
They are one, violence against women, two, employment, three, education, four,

101
00:08:22,400 --> 00:08:28,000
reproductive health care, five, decision making, and six, access to contraceptives.

102
00:08:28,000 --> 00:08:33,120
Now, for this project, there's clearly a focus on employment and decision making.

103
00:08:33,120 --> 00:08:38,400
The problem is, when you look at the project services related to employment for these women

104
00:08:38,880 --> 00:08:47,360
and the services related to improving their decision making, there are two indicators

105
00:08:47,360 --> 00:08:49,360
in the performance measurement framework.

106
00:08:49,360 --> 00:08:55,600
Related to these two areas, the first one is percent of women in employment or self-employment

107
00:08:55,600 --> 00:08:56,880
after training.

108
00:08:57,360 --> 00:09:03,280
The problem here is there's no comparison group there for that service being delivered,

109
00:09:03,280 --> 00:09:04,080
the training.

110
00:09:05,040 --> 00:09:10,000
Also, there's no measure of their incomes after the training, which would be a better

111
00:09:10,000 --> 00:09:16,480
measure of whether or not, even if they do get employed, are they making any money?

112
00:09:16,480 --> 00:09:21,120
Are they making any money from that employment?

113
00:09:21,120 --> 00:09:26,880
And is it due to the project training that they receive, for example, in entrepreneurial

114
00:09:26,880 --> 00:09:28,560
skills or how to set up a business?

115
00:09:29,520 --> 00:09:31,680
So they didn't use a comparison group there.

116
00:09:31,680 --> 00:09:37,440
So we need to know for the female empowerment index, did they also ask that comparison group

117
00:09:38,000 --> 00:09:44,400
for the project women, by the way, you attended some training on entrepreneurship.

118
00:09:44,400 --> 00:09:47,280
Did you earn any income after you received the training?

119
00:09:47,920 --> 00:09:54,320
It's not clear if they actually asked them that because we don't know if the same two

120
00:09:54,320 --> 00:09:59,840
comparison groups, that is the project women and the non-project women for the female

121
00:09:59,840 --> 00:10:06,160
empowerment index, if they're also using that same group of women for those two groups for

122
00:10:06,160 --> 00:10:10,240
all the other parts of the project, such as training them in entrepreneurship.

123
00:10:10,240 --> 00:10:13,040
We hope they are, but we don't know.

124
00:10:13,040 --> 00:10:18,160
It's not clear because in the performance measurement framework, it doesn't show a

125
00:10:18,160 --> 00:10:24,480
comparison group on this indicator of percent of women in employment, self-employment after

126
00:10:24,480 --> 00:10:25,520
the project training.

127
00:10:26,560 --> 00:10:31,600
The other indicator that they use that's related to decision making is the percent of women

128
00:10:31,600 --> 00:10:38,560
targeted by the project who participate in decision making bodies in their communities.

129
00:10:38,560 --> 00:10:42,240
The problem here is it is only measured once per year.

130
00:10:43,120 --> 00:10:50,000
So again, these women could easily increase their participation by themselves without

131
00:10:50,000 --> 00:10:50,960
the project training.

132
00:10:52,160 --> 00:11:00,000
There also is no comparison group at the time that they're receiving training in decision

133
00:11:00,000 --> 00:11:00,480
making.

134
00:11:00,480 --> 00:11:05,280
A better measure of whether or not they're participating in decision making is not to

135
00:11:05,280 --> 00:11:09,120
ask the women themselves because they're self-reporting bias.

136
00:11:09,120 --> 00:11:15,520
The project women, of course, are going to say, yes, social desirability bias.

137
00:11:15,520 --> 00:11:23,040
Yes, I have increased my participation in the community in decision making bodies.

138
00:11:23,040 --> 00:11:30,000
A better measure would be to see if these women have increased their membership on decision

139
00:11:30,000 --> 00:11:33,840
making bodies, either municipal, elected, or state.

140
00:11:33,840 --> 00:11:40,720
Either municipal, elected councils, or school boards, or whatever they consider to be a

141
00:11:40,720 --> 00:11:42,160
decision making body.

142
00:11:42,160 --> 00:11:47,760
Have these project women actually become members in positions of power where they can make

143
00:11:47,760 --> 00:11:48,320
decisions.

144
00:11:48,880 --> 00:11:53,440
That would be a better objective measure of whether or not they've increased their

145
00:11:53,440 --> 00:11:56,480
participation on decision making bodies.

146
00:11:57,040 --> 00:11:58,800
So that's the problem there.

147
00:11:58,800 --> 00:12:05,200
Even though the female empowerment index, they're using comparison groups at the top,

148
00:12:05,200 --> 00:12:06,720
at the end of the project.

149
00:12:06,720 --> 00:12:12,560
We don't know if all the trainings they received during the project are making a significant

150
00:12:12,560 --> 00:12:18,160
difference compared to the non-project women because they were not using comparison groups.

151
00:12:18,160 --> 00:12:23,920
Now, hopefully, College and Institutes Canada will attend part two of the podcast and will

152
00:12:23,920 --> 00:12:31,120
explain perhaps, maybe when we were using the female empowerment index, we were also

153
00:12:31,120 --> 00:12:35,760
using these other questions on these other indicators at the same time.

154
00:12:35,760 --> 00:12:41,040
And then we could show that even when the project was delivering these services, the

155
00:12:41,040 --> 00:12:47,360
women were also reporting on their incomes, their employment levels, also to show that

156
00:12:47,360 --> 00:12:53,840
their employment was going up and their ability to participate in decision making was also

157
00:12:53,840 --> 00:12:56,560
going up compared to the non-project women.

158
00:12:56,560 --> 00:13:02,000
So then now that you have a brief description of the project, as well as my conclusion that

159
00:13:02,000 --> 00:13:05,440
the outcome indicators, at least three of them that I've talked to you about there,

160
00:13:06,080 --> 00:13:11,920
are not properly designed to show that the outcomes were actually achieved for the project.

161
00:13:12,480 --> 00:13:18,720
Now, we're going to send this part one to College and Institutes Canada to invite them

162
00:13:18,720 --> 00:13:26,720
to attend part two and there will also be an evaluation expert from Sweden who will

163
00:13:26,720 --> 00:13:34,000
be attending the podcast to take turns with me in explaining to College and Institutes Canada

164
00:13:34,000 --> 00:13:38,800
why we believe the outcome indicators are flawed, but more importantly, we're also

165
00:13:38,800 --> 00:13:44,720
going to suggest how these indicators could be improved so that the overall evaluation

166
00:13:44,720 --> 00:13:46,720
of the project could be improved.

167
00:13:46,720 --> 00:13:53,120
Thank you for listening and stay tuned for episode six, part two, featuring College and

168
00:13:53,120 --> 00:13:59,120
Institutes Canada and the flaws in their performance measurement framework and how we can improve

169
00:13:59,120 --> 00:14:15,920
those indicators leading to a better evaluation of the project.