1 00:00:00,000 --> 00:00:03,440 Welcome to the Improving Development Evaluation Podcast. 2 00:00:03,920 --> 00:00:09,320 I'm your host, David Wand, and welcome to Season One, Episode Two, Part One. 3 00:00:09,760 --> 00:00:13,680 In this episode, we're going to introduce you to an international 4 00:00:13,680 --> 00:00:15,040 development organization. 5 00:00:15,040 --> 00:00:35,440 That international development organization is the the Canadian Bureau of International Education. 6 00:00:45,040 --> 00:00:58,880 You can learn more about the Canadian Bureau of International Education if you go to their website at www.cbie.ca. 7 00:00:59,560 --> 00:01:02,520 Their headquarters is located in Ottawa, Canada. 8 00:01:03,880 --> 00:01:10,520 The project we're going to be talking about today in Part One is for Episode Two. 9 00:01:10,520 --> 00:01:23,360 The Canadian Bureau of International Education has been funded by the government of Canada to deliver this project, and it is called African Leaders of Tomorrow Scholarships Fund. 10 00:01:23,960 --> 00:01:34,560 This project is being delivered in Canada, and the cost of this project to the Canadian taxpayer is $5,127,124. 11 00:01:34,880 --> 00:01:36,600 Of course, it could be a little less, I think. 12 00:01:36,600 --> 00:01:39,720 It depends on the actual expenditures. 13 00:01:39,720 --> 00:01:56,520 And in this Part One, I'm going to give you a brief description of the project services that are being delivered and a critique of the Canadian Bureau of International Education's PMF, Performance Measurement Framework, that they have developed to measure the project's performance. 14 00:01:56,520 --> 00:02:04,440 That is, the achievement of the project's outcomes using the outcome indicators in that performance measurement framework. 15 00:02:04,440 --> 00:02:25,680 But before I describe the project services that are being delivered for this project by the Canadian Bureau of International Education in Canada, I'm going to go to my summary and conclude that in this performance measurement framework for this project, there are 11 outcome indicators that are measuring six outcomes. 16 00:02:25,680 --> 00:02:37,200 There is a seventh outcome, but its outcome indicators will not be reviewed, as I will explain a bit later, because this outcome is actually an output or service. 17 00:02:37,200 --> 00:02:41,200 It's not what you expect after you deliver the service. 18 00:02:41,200 --> 00:02:43,200 It actually is the service. 19 00:02:43,200 --> 00:02:45,200 So that's why we're not going to look at it. 20 00:02:45,200 --> 00:02:57,200 So let's start. My review of all 11 outcome indicators has led me to conclude that all of these outcome indicators are not measuring the six outcomes. 21 00:02:57,200 --> 00:03:17,200 So the organization, the Canadian Bureau of International Education, cannot make the claim that it's achieving its project outcomes with the services they're delivering in the project simply because their outcome indicators are not properly measuring the outcomes for this project. 22 00:03:17,200 --> 00:03:27,200 As you probably know from the trailer for this podcast, those 11 outcome indicators fall into one of five problem areas. 23 00:03:27,200 --> 00:03:53,200 Now for this project, eight of those outcome indicators fall into the problem area of how do you feel self-reporting, where people are asked how they feel about achieving the outcome rather than being objectively measured on whether they achieve that outcome. 24 00:03:53,200 --> 00:04:19,200 Another two of the outcome indicators fall into problem area number five, which is there's no comparison group of people outside of the project that are also measured on the same outcome, which means even if people in the project are achieving the outcomes, we don't know if they're achieving them any better than another group of people that could also achieve the outcome outside of the project. 25 00:04:19,200 --> 00:04:25,200 Some people refer to this with a fancy word called impact. 26 00:04:25,200 --> 00:04:28,200 But I won't bore you with that word. I'll just stick to the facts. 27 00:04:28,200 --> 00:04:41,200 And finally, we have one outcome indicator that is falling into problem number three, where the outcome indicator is measuring something that is not related to the achievement of the outcome at all. 28 00:04:41,200 --> 00:04:52,200 So now that I concluded that the PMF is flawed, that means the performance measurement framework is flawed. Let's get into more detail about the project services. 29 00:04:52,200 --> 00:05:03,200 And just to remind you, if you wish to get more detail about this project, you can go to your search engine and enter Global Affairs Canada project browser into your search engine. 30 00:05:03,200 --> 00:05:13,200 And what will come up is a platform with all the projects listed there, you can go to the partner list and select Canadian Bureau of International Education. 31 00:05:13,200 --> 00:05:17,200 There you will see a description of this project. 32 00:05:17,200 --> 00:05:32,200 If you wish to get the Canadian Bureau of International Education performance measurement framework for this project, as well as my summary of the 11 outcome indicators for this project and where they fall into one of those five problem areas, you can email me 33 00:05:32,200 --> 00:05:53,200 at evaluatecanadaaid at gmail.com, and I will send it to you. Also, in that summary, I will include the link to the project description at the Global Affairs Canada project browser platform where you can find a description of the project and the services they deliver. 34 00:05:53,200 --> 00:06:05,200 And as you know from the trailer, you will not see on that platform at Global Affairs Canada, the performance measurement framework. Hence, I think the need for this podcast. 35 00:06:05,200 --> 00:06:09,200 So let's start with a brief description of the project. 36 00:06:09,200 --> 00:06:19,200 The target group that is receiving the services delivered from this project are Africans who are entering a master's program in public administration in Canada. 37 00:06:19,200 --> 00:06:33,200 Between 120 and 130 students are selected and placed into this master's program in public administration in Canada with these students coming from 23 African countries. 38 00:06:33,200 --> 00:06:49,200 The project delivers the following services to these students. Number one, mentoring by Canadian public administration practitioners to those students from Africa, who are attending a master's program in public administration in Canada. 39 00:06:49,200 --> 00:06:53,200 Number two, delivering learning events to these students. 40 00:06:53,200 --> 00:07:08,200 Number three, providing online platforms for the students so that they can network with other students and other people. Number four, matching these students with Canadian practitioners for internship placements. 41 00:07:08,200 --> 00:07:14,200 Number five, developing guidelines to support these students when they are interns. 42 00:07:14,200 --> 00:07:43,200 Number six, monitoring these students in their academic programs. And finally, number seven, a variety of administrative services that the project delivers, such as helping the students get their study permits, paying for promoting the program to get international student applications, using a website, processing their applications online, and selecting and placing the candidates in these master's programs in public administration in Canada. 43 00:07:43,200 --> 00:08:03,200 Additional funds are provided by MasterCard Foundation to pay for the foreign international student tuition fees for this project, which are quite high, as you probably know if you've been following the news, like around $18,000 a pop per year. 44 00:08:03,200 --> 00:08:12,200 So, there you go. So that gives you an idea of the services the project delivers. Now, let's get into the outcomes. 45 00:08:12,200 --> 00:08:31,200 Let's get into the outcomes for this project. There are seven outcomes, and they are outcome number one, increased leadership skills of female and male African leaders of tomorrow scholars and public administration and public policy outcome number two, increased access of female 46 00:08:31,200 --> 00:08:45,200 African leaders of tomorrow scholars to opportunities for exchange and collaboration within the framework of Canadian and Pan African public administration public policy networks. 47 00:08:45,200 --> 00:09:02,200 This is not an outcome, but direct services, increased access with no expected outcome from accessing those services. Therefore, its indicators will not be reviewed and this outcome will not be reviewed. So we have six outcomes that will be reviewed. 48 00:09:02,200 --> 00:09:23,200 Outcome number three, enhanced understanding of female and male African leaders of tomorrow scholars, respecting public administration, public policy development and implementation processes, including the integration of gender equality considerations. 49 00:09:23,200 --> 00:09:35,200 Outcome number four, increased knowledge of female and male African leaders of tomorrow scholars in governance, public administration, public policy and gender equality. 50 00:09:35,200 --> 00:09:53,200 Outcome number five, increased effectiveness of female and male African leaders of tomorrow scholars to provide leadership within public administration, public policy networks, both Canadian and Pan African networks. 51 00:09:53,200 --> 00:10:11,200 Outcome number six, increased effectiveness of female and male African leaders of tomorrow scholars to contribute to public administration, public policy development in key sectors in sub Saharan Africa. 52 00:10:11,200 --> 00:10:25,200 Keep in mind that one again in sub Saharan Africa, so they're expected after they graduate to go back to their home country and work in the public sector, the government sector in their respective country. 53 00:10:25,200 --> 00:10:38,200 Outcome number seven, strengthened economic and social development of key public administration and policy sectors in sub Saharan Africa. 54 00:10:38,200 --> 00:10:52,200 So, part two of the podcast will cover at least six outcome indicators to ensure that all six outcomes, not including outcome two, because it's increased access that's not an outcome, that's an output. 55 00:10:52,200 --> 00:10:56,200 And it is not an outcome of the project are covered. 56 00:10:56,200 --> 00:11:10,200 So, a Canadian Bureau of International Education representative will be invited to respond to the evaluation panels critique of those outcome indicators and provide alternative methods of evaluation for the project. 57 00:11:10,200 --> 00:11:23,200 At the end of part two, the evaluation panel will be concluding as to whether the Canadian Bureau of International Education can make the claim that its project services are achieving its project outcomes. 58 00:11:23,200 --> 00:11:28,200 As host, I conclude that the Canadian Bureau of International Education cannot make this claim. 59 00:11:28,200 --> 00:11:38,200 But I want our listeners to hear what the Canadian Bureau of International Education has to say, as well as the other evaluation experts. 60 00:11:38,200 --> 00:11:56,200 Now I'm going to look at just three outcome indicators to give you an idea of why they fall into one of these five problem areas and in part two, we are going to make sure we go through all six outcomes, looking at least at one outcome indicator for each of those outcomes. 61 00:11:56,200 --> 00:12:15,200 So, let's look at the outcome number three enhanced understanding of female and male African leaders of tomorrow scholars, respecting public administration, public policy development and implementation processes including the integration of gender equality considerations. 62 00:12:15,200 --> 00:12:30,200 One of the outcome indicators to measure this enhanced understanding is percentage of female and male African leaders of tomorrow scholars, reporting that they're mentoring relationships with Canadian leaders. 63 00:12:30,200 --> 00:12:43,200 Practitioners have strengthened their knowledge of public administration, public policy development and implementation processes, including the integration of gender equality considerations. 64 00:12:43,200 --> 00:12:56,200 Measuring strength and knowledge can be best done objectively through testing, rather than self reporting, especially with something as specific as quote, gender equality, which needs to be clearly defined. 65 00:12:56,200 --> 00:13:04,200 So, self reporting based on a mentoring relationship would not be a valid way of measuring that knowledge. 66 00:13:04,200 --> 00:13:10,200 It would not be a valid way of measuring a strengthening of that knowledge. 67 00:13:10,200 --> 00:13:23,200 Let's look at the outcome number four increased knowledge of female and male African leaders of tomorrow scholars in governance public administration, public policy and gender equality. 68 00:13:23,200 --> 00:13:39,200 One of the outcome indicators for this outcome is percentage of female and male African leaders of tomorrow scholars who have successfully completed a master's level public administration, public policy program in Canada. 69 00:13:39,200 --> 00:13:48,200 Now on the surface you would think this indicator would be a good measure of increased knowledge, since a master's degree in public administration. 70 00:13:48,200 --> 00:13:56,200 After all, is directly relevant to knowledge specific to quote governance public administration and public policy. 71 00:13:56,200 --> 00:14:19,200 However, two points to keep in mind here are that gender equality is very specific. So we need to look further to see of courses specific to gender equality were completed and fit with gender equality policies and programs back in the students African country, where they are expected to return to after graduation to use their 72 00:14:19,200 --> 00:14:41,200 master's degree in public administration to contribute to and I quote governance public administration, public policy, getting the NPA would not be considered directly related to quote increased knowledge of gender equality, unless the NPA required the completion of courses specific to 73 00:14:41,200 --> 00:14:56,200 gender equality. For example, with these students be able to calculate the United Nations Development Program UNDP gender inequality index for their country, using a particular data set. 74 00:14:56,200 --> 00:15:11,200 And would they be able to use that calculation to support their development of policies to address gender inequality. This would seem to be a good measure of their quote increased knowledge of gender equality. 75 00:15:11,200 --> 00:15:26,200 Finally, a second point is current Canadian immigration policy encourages international students to remain in Canada, after graduation. 76 00:15:26,200 --> 00:15:43,200 This is a good example of how these immigration targets, as these immigrants are young, highly educated with Canadian master's degrees in public administration, and have become accustomed to Canada and offer a skill set to replace the large group of public servants in Canada who 77 00:15:43,200 --> 00:16:01,200 are in the process of being able to apply for a degree in public administration. This conflicts directly with outcome number six of this project, which wants these masters graduates in public administration to return back to their African country, who and I quote contribute to public 78 00:16:01,200 --> 00:16:07,200 administration in key sectors in sub Saharan Africa. 79 00:16:07,200 --> 00:16:23,200 This may raise the need to introduce the indicator here in the project percent of NPA graduates who return to their home country, and are employed in the government sector in their home country. 80 00:16:23,200 --> 00:16:42,200 This will become seven strengthen economic and social development of key public administration and policy sectors in sub Saharan Africa graduates themselves are asked if you look at the performance measurement framework for this indicator to self report their quote 81 00:16:42,200 --> 00:16:50,200 level of position in Africa, as either entry, middle or senior position in Africa. 82 00:16:50,200 --> 00:17:11,200 These positions should be defined and applied across all 23 African countries, where these students are coming from, and then graduates could be interviewed based on those criteria, rather than self reporting, that would be a better way of seeing if they are actually in the 83 00:17:11,200 --> 00:17:22,200 senior levels of government. Also, what about masters graduates in public administration outside of the project. 84 00:17:22,200 --> 00:17:42,200 There could be thousands of Africans who go to universities in their own countries graduate with NPAs. They could be in higher senior positions than this project, showing better strength and development of the public sector than this Canadian Bureau of International Education 85 00:17:42,200 --> 00:18:00,200 program. This outcome indicator has self reporting bias. 86 00:18:00,200 --> 00:18:17,200 This is a very good example of how this program is being used in the public sector compared to other NPA graduates that did not participate in this project. So those are just three examples from three outcome indicators that fall into those problem areas. 87 00:18:17,200 --> 00:18:30,200 If you have any critical feedback, any critical criticism, or critical criticism or critical critique an episode, feel free to email me at evaluatecandidateatgmail.com. 88 00:18:30,200 --> 00:18:34,800 delivered by the Canadian Bureau of International Education, 89 00:18:34,800 --> 00:18:38,720 as well as my summary of all of the outcome indicators 90 00:18:38,720 --> 00:18:40,560 on which problem area 91 00:18:40,560 --> 00:18:43,140 that each of the outcome indicators falls into, 92 00:18:43,140 --> 00:18:44,500 feel free to email me. 93 00:18:46,120 --> 00:18:51,120 And finally, if there are any evaluation experts out there 94 00:18:51,280 --> 00:18:52,780 who'd like to be on the podcast 95 00:18:52,780 --> 00:18:55,040 to look at other performance measurement frameworks, 96 00:18:55,040 --> 00:18:57,920 and in particular the outcome indicators, 97 00:18:57,920 --> 00:19:00,760 to see if they fall into these five problem areas, 98 00:19:00,760 --> 00:19:02,900 email me and I'd be happy to send you 99 00:19:02,900 --> 00:19:05,800 the remaining performance measurement frameworks 100 00:19:05,800 --> 00:19:07,480 that we have lined up for season one. 101 00:19:07,480 --> 00:19:11,820 We have 12, where we don't have evaluation experts so far. 102 00:19:12,800 --> 00:19:15,160 And what we're going to do now, 103 00:19:15,160 --> 00:19:16,840 at the end of this part one, 104 00:19:16,840 --> 00:19:20,160 is send it to the Canadian Bureau 105 00:19:20,160 --> 00:19:24,540 of International Education representative, 106 00:19:24,540 --> 00:19:28,420 and invite them to attend part two of this episode, 107 00:19:28,420 --> 00:19:31,560 to respond to this critique here in part one, 108 00:19:32,820 --> 00:19:35,140 along with an evaluation expert 109 00:19:35,140 --> 00:19:37,880 that will also attend part two of this podcast, 110 00:19:38,740 --> 00:19:41,140 to also provide a critique 111 00:19:41,140 --> 00:19:43,720 of their performance measurement framework. 112 00:19:43,720 --> 00:19:44,720 Thank you for listening, 113 00:19:44,720 --> 00:19:47,480 and stay tuned for Canadian Bureau 114 00:19:47,480 --> 00:19:49,440 of International Education, 115 00:19:49,440 --> 00:19:54,440 part two, episode two, season one.