WEBVTT 00:00:00.000 --> 00:00:03.399 So, welcome to the Improving Development Evaluation 00:00:03.399 --> 00:00:07.019 Podcast. I'm your host, David Wand, and in this 00:00:07.019 --> 00:00:10.560 episode, we're going to feature Humber Polytechnic 00:00:10.560 --> 00:00:13.740 International Development Institute. You can 00:00:13.740 --> 00:00:18.960 learn more about the Humber Polytechnic International 00:00:18.960 --> 00:00:21.699 Development Institute if you go to their website, 00:00:22.019 --> 00:00:25.239 which will be in the episode notes. It's humber 00:00:25.239 --> 00:00:29.559 .ca forward slash global forward slash international 00:00:29.559 --> 00:00:34.920 hyphen development hyphen institute hyphen IDI. 00:00:36.179 --> 00:00:38.159 And we're going to look at a project that they 00:00:38.159 --> 00:00:42.719 received $4 ,950 ,000 from the government of 00:00:42.719 --> 00:00:45.820 Canada. That's you guys, some of you listening, 00:00:45.939 --> 00:00:49.420 the Canadian taxpayers. And it's for a project 00:00:49.420 --> 00:00:53.109 that they delivered in Bhutan. called Education 00:00:53.109 --> 00:00:57.530 and Skills Training. And I'll also put in the 00:00:57.530 --> 00:01:01.670 episode notes, the link to the project description. 00:01:01.789 --> 00:01:03.770 Of course, I'm going to give you a bit of background 00:01:03.770 --> 00:01:09.209 on what this project and the $4 ,950 ,000 services 00:01:09.209 --> 00:01:12.689 it actually delivered. And as usual, when it's 00:01:12.689 --> 00:01:17.090 done, I'll be sending my critique, this link 00:01:17.090 --> 00:01:19.390 to the episode, along with the performance measurement 00:01:19.390 --> 00:01:23.650 framework. to the Humber Polytechnic to invite 00:01:23.650 --> 00:01:26.349 them to respond to my critique if they wish. 00:01:26.549 --> 00:01:29.549 And of course, I will always send it to the minister 00:01:29.549 --> 00:01:32.549 responsible for international development, which 00:01:32.549 --> 00:01:35.989 is I think called the Secretary of State for 00:01:35.989 --> 00:01:38.870 International Development. I'll also send it 00:01:38.870 --> 00:01:40.709 to, of course, all the other political party 00:01:40.709 --> 00:01:43.730 shadow critics responsible for international 00:01:43.730 --> 00:01:47.780 development. And I'm always recommending... that 00:01:47.780 --> 00:01:50.540 instead of us trying to chase the performance 00:01:50.540 --> 00:01:53.620 measurement framework, that all of these agencies, 00:01:53.640 --> 00:01:57.599 including Humber, Polytechnic, put on their website 00:01:57.599 --> 00:02:00.620 their performance measurement framework, as well 00:02:00.620 --> 00:02:04.019 as the data, which we aren't getting right now. 00:02:04.280 --> 00:02:07.439 But I do have some good news. There is an international 00:02:07.439 --> 00:02:09.780 organization that actually is starting to do 00:02:09.780 --> 00:02:12.879 it. So in future episodes, I will be critiquing 00:02:12.879 --> 00:02:15.879 their performance measurement framework that 00:02:15.879 --> 00:02:19.639 is on their website. So without further delay, 00:02:19.879 --> 00:02:23.539 let's get into this project in Bhutan. And one 00:02:23.539 --> 00:02:26.939 last point. If you're listening to this podcast 00:02:26.939 --> 00:02:32.060 on Spotify, Spotify has now added a comment section. 00:02:32.259 --> 00:02:35.680 So feel free to leave your comments in response 00:02:35.680 --> 00:02:39.180 to any of my episodes that are up there. I look 00:02:39.180 --> 00:02:42.819 forward to reading your comments. And if you 00:02:42.819 --> 00:02:45.719 ever want any of my performance measurement frameworks 00:02:45.719 --> 00:02:48.930 along with my critiques, that are produced in 00:02:48.930 --> 00:02:54.349 writing. You can send me an email at evaluatecanadaaid 00:02:54.349 --> 00:02:57.909 at gmail .com. That'll be in the episode notes. 00:02:58.110 --> 00:03:01.909 And feel free to request, and I'm happy to send 00:03:01.909 --> 00:03:05.030 them to you. So this project in Bhutan, delivered 00:03:05.030 --> 00:03:08.629 by Humber Polytechnic International Development 00:03:08.629 --> 00:03:11.620 Institute. They basically have six target groups 00:03:11.620 --> 00:03:16.960 that they deliver services to with this $4 ,950 00:03:16.960 --> 00:03:21.280 ,000. Target group number one are technical vocation 00:03:21.280 --> 00:03:24.879 institution teachers. Target group number two, 00:03:24.960 --> 00:03:29.240 technical vocation institute curriculum developers. 00:03:29.639 --> 00:03:33.840 Number three, technical vocation industry representatives. 00:03:34.139 --> 00:03:38.719 Number four, technical vocation institutes. Number 00:03:38.719 --> 00:03:43.020 five, all the staff in the technical vocation 00:03:43.020 --> 00:03:47.300 institution. And number six, parents and community 00:03:47.300 --> 00:03:51.879 leaders at large. So you might be wondering, 00:03:52.020 --> 00:03:56.060 what services do these target groups get? Well, 00:03:56.139 --> 00:03:58.900 the technical vocation teachers, they're receiving 00:03:58.900 --> 00:04:04.080 training on gender equality to incorporate within 00:04:04.080 --> 00:04:08.539 industry. They're also receiving training on 00:04:08.539 --> 00:04:11.080 career counseling for students, and in particular 00:04:11.080 --> 00:04:14.439 women students and students who have disabilities. 00:04:15.099 --> 00:04:22.180 They're also being trained on ability to coach 00:04:22.180 --> 00:04:25.920 students in self -employment to be gender sensitive 00:04:25.920 --> 00:04:30.300 and environmentally responsible. The industry 00:04:30.300 --> 00:04:33.759 staff target group. Industry representatives, 00:04:33.899 --> 00:04:36.220 they're being trained on how to best accommodate 00:04:36.220 --> 00:04:40.420 women and people with disability students when 00:04:40.420 --> 00:04:43.939 they graduate from these technical vocation institutes. 00:04:44.519 --> 00:04:48.660 A third target group, technical vocation institute 00:04:48.660 --> 00:04:51.500 curriculum developers, they're being trained 00:04:51.500 --> 00:04:54.920 by Humber Polytechnic to be able to incorporate 00:04:54.920 --> 00:04:58.100 gender equality into the student curriculum. 00:04:59.120 --> 00:05:02.240 And the technical vocation institutions by themselves 00:05:02.240 --> 00:05:05.800 as another target group, they are, it's not really 00:05:05.800 --> 00:05:07.639 training, but it just says in the performance 00:05:07.639 --> 00:05:14.899 measurement framework that their job is to set 00:05:14.899 --> 00:05:19.560 up incubation centers to enable students to pilot 00:05:19.560 --> 00:05:23.259 business ideas. They're also responsible for, 00:05:23.339 --> 00:05:27.279 with this funding, to set up resource centers. 00:05:27.839 --> 00:05:32.339 to deliver knowledge to students. Another target 00:05:32.339 --> 00:05:36.160 group are all of the staff at the Technical Vocation 00:05:36.160 --> 00:05:38.459 Institutes where they do this training in technical 00:05:38.459 --> 00:05:41.600 vocations. That training is on gender equality, 00:05:42.000 --> 00:05:45.899 also on gender sensitive, disability inclusive 00:05:45.899 --> 00:05:50.160 admissions, advising and counseling, and also 00:05:50.160 --> 00:05:54.720 gender inclusive recruitment. And finally, the 00:05:54.720 --> 00:05:58.180 last target group which are parents and community 00:05:58.180 --> 00:06:01.079 leaders in the public at large, is promoting 00:06:01.079 --> 00:06:05.199 technical vocation to them for people with disabilities 00:06:05.199 --> 00:06:10.699 through events. And also for women, promoting 00:06:10.699 --> 00:06:14.379 technical vocation to women through education 00:06:14.379 --> 00:06:17.300 fairs and education forums. So that gives you 00:06:17.300 --> 00:06:20.699 an idea of who is getting these various services 00:06:20.699 --> 00:06:23.939 and these various target groups. So before we 00:06:23.939 --> 00:06:27.899 get into... the outcomes and the outcome indicators 00:06:27.899 --> 00:06:31.519 to see if all these wonderful services were achieving 00:06:31.519 --> 00:06:36.879 the outcomes, we need to digress a bit and say 00:06:36.879 --> 00:06:41.540 to maybe the staff, although Bhutan is not really 00:06:41.540 --> 00:06:47.019 that hot and humid in the global south, perhaps. 00:06:47.100 --> 00:06:49.279 I've been to Nepal, but I haven't been to Bhutan. 00:06:49.459 --> 00:06:52.319 There may be times. where it is hot and humid. 00:06:52.439 --> 00:06:54.819 So people living in the Global South, or you 00:06:54.819 --> 00:06:57.660 have to go to the Global South to, believe it 00:06:57.660 --> 00:07:01.420 or not, evaluate a project, or maybe you just 00:07:01.420 --> 00:07:04.560 live there right now, I highly recommend two 00:07:04.560 --> 00:07:08.500 products to protect yourself from the sun in 00:07:08.500 --> 00:07:11.040 hot and humid climates that are mostly going 00:07:11.040 --> 00:07:15.519 on in the Global South. Sunglasses you can get 00:07:15.519 --> 00:07:20.629 at akila .la, A -K -I -L -A dot L -A. You can 00:07:20.629 --> 00:07:23.569 get prescription lenses also, all made from 50 00:07:23.569 --> 00:07:26.769 % recycled material. You get a 10 % discount 00:07:26.769 --> 00:07:32.129 if you use my link in the episode notes. I will 00:07:32.129 --> 00:07:35.850 get a 15 % commission when you purchase any product 00:07:35.850 --> 00:07:41.490 using that promotional link. So you have your 00:07:41.490 --> 00:07:44.069 sunglasses to protect you from the sun and the 00:07:44.069 --> 00:07:46.290 global south, but you need the right clothes 00:07:46.290 --> 00:07:49.110 to be comfortable in that hot. and human climate. 00:07:49.230 --> 00:07:53.029 So I would recommend my second product, and that's 00:07:53.029 --> 00:07:56.509 men's polos, tees, and button -ups, but I'm quite 00:07:56.509 --> 00:08:00.069 sure women would find them acceptable too. And 00:08:00.069 --> 00:08:03.990 you can go to hypernaturalstyle .com. That's 00:08:03.990 --> 00:08:08.269 hypernaturalstyle .com. And you get 20 % off 00:08:08.269 --> 00:08:11.709 any product when you use my promotional link, 00:08:11.889 --> 00:08:15.100 which you'll see in the episode notes. And if 00:08:15.100 --> 00:08:17.779 you do that and make a purchase, I get a 20 % 00:08:17.779 --> 00:08:21.860 commission on the total purchase you made when 00:08:21.860 --> 00:08:25.420 you use my promotional link. Just go to the link 00:08:25.420 --> 00:08:28.860 and you'll go straight to the company's website 00:08:28.860 --> 00:08:32.399 for purchase. So what is so cool about these 00:08:32.399 --> 00:08:35.980 Hypernatural t -shirts? They have a patent pending 00:08:35.980 --> 00:08:40.200 Hypercool Jade technology that when it's hot, 00:08:40.299 --> 00:08:43.019 lowers your body temperature by three to five 00:08:43.019 --> 00:08:47.519 degrees. They're also breathable thanks to Supina 00:08:47.519 --> 00:08:50.779 cotton. And they are sold in premium retailers 00:08:50.779 --> 00:08:54.460 like Nordstrom. And were voted the best men's 00:08:54.460 --> 00:08:58.480 polo shirts in 2024 in Men's Journal, Forbes, 00:08:58.659 --> 00:09:04.940 and Esquire. So check those out. And now we're 00:09:04.940 --> 00:09:07.700 going to get into the Performance Measurement 00:09:07.700 --> 00:09:14.240 Framework, PMF. And we'll say, let's start off 00:09:14.240 --> 00:09:17.679 with, you have an idea of the target groups, 00:09:17.779 --> 00:09:20.500 the services they receive. So let's go through 00:09:20.500 --> 00:09:24.179 the various outcomes. So what you're going to 00:09:24.179 --> 00:09:33.139 discover first to begin with is there are 29 00:09:33.139 --> 00:09:37.159 outcome indicators. determining whether or not 00:09:37.159 --> 00:09:42.360 11 expected outcomes in this $4 ,950 ,000 project 00:09:42.360 --> 00:09:45.580 were actually achieved. So we're going to go 00:09:45.580 --> 00:09:48.320 through those 29 outcome indicators for those 00:09:48.320 --> 00:09:51.600 11 expected outcomes, and then we'll have a conclusion 00:09:51.600 --> 00:09:57.759 as to how many of those were achieved. I should 00:09:57.759 --> 00:10:04.320 also point out to you that Humber Polytechnic 00:10:05.070 --> 00:10:08.269 offers a postgraduate diploma program in public 00:10:08.269 --> 00:10:11.149 administration, which I had the opportunity about, 00:10:11.230 --> 00:10:14.429 what is it, 2008. I taught two courses there, 00:10:14.549 --> 00:10:18.009 and one of them was supposed service quality. 00:10:18.600 --> 00:10:21.340 But now they finally have realized how bad that 00:10:21.340 --> 00:10:23.940 was. That's what you get when an MBA is running 00:10:23.940 --> 00:10:26.259 the school of business and doesn't understand 00:10:26.259 --> 00:10:29.940 that when you're evaluating human services, it's 00:10:29.940 --> 00:10:32.100 more than just asking the human being after they 00:10:32.100 --> 00:10:34.360 got the service, whether they have been customer 00:10:34.360 --> 00:10:38.279 satisfied with the service they received. There's 00:10:38.279 --> 00:10:40.940 a whole field of evaluation and psychometrics 00:10:40.940 --> 00:10:43.659 that you have to be involved with, as you've 00:10:43.659 --> 00:10:47.320 noticed from my podcast. But now they have improved. 00:10:48.279 --> 00:10:52.100 the postgraduate Humber Polytechnic program, 00:10:52.320 --> 00:10:55.440 diploma program in public administration. They 00:10:55.440 --> 00:10:58.100 have two new courses instead, which get it right. 00:10:58.220 --> 00:11:01.299 They have one in project design and one in project 00:11:01.299 --> 00:11:04.179 evaluation. And the first one, project design, 00:11:04.340 --> 00:11:07.779 guess what? They teach the students how to design 00:11:07.779 --> 00:11:11.320 a performance measurement framework, how to properly 00:11:11.320 --> 00:11:14.379 design an outcome indicator so it reliably... 00:11:14.809 --> 00:11:18.090 and validly measures the outcome that the project 00:11:18.090 --> 00:11:21.070 is trying to achieve. Why do I bring it up here? 00:11:21.269 --> 00:11:24.350 Well, when I finish my critique, I'm going to 00:11:24.350 --> 00:11:27.490 send it to them and suggest that they share it 00:11:27.490 --> 00:11:31.990 with the instructor for that course. It's BUS5014, 00:11:32.429 --> 00:11:38.129 Project Design and Planning, so that maybe their 00:11:38.129 --> 00:11:41.570 instructor will incorporate it into the curriculum 00:11:41.570 --> 00:11:46.120 as an example. A real case study of not any project, 00:11:46.220 --> 00:11:50.679 but actually a project that the Humber Polytechnic 00:11:50.679 --> 00:11:54.539 is doing itself. So it's relevant because they're 00:11:54.539 --> 00:11:57.379 going to Humber Polytechnic to study this. So 00:11:57.379 --> 00:11:59.639 you would think it would be logical and helpful 00:11:59.639 --> 00:12:03.700 for them to incorporate this performance measurement 00:12:03.700 --> 00:12:07.580 framework as well as my critique. But of course, 00:12:07.580 --> 00:12:10.909 they're free to disagree with my critique. And 00:12:10.909 --> 00:12:13.289 hopefully we'll come on the podcast and respond 00:12:13.289 --> 00:12:16.009 to what I'm going to tell you right now as I 00:12:16.009 --> 00:12:21.070 go through the 29 outcome indicators. So let's 00:12:21.070 --> 00:12:26.929 start with the first outcome. And this is the 00:12:26.929 --> 00:12:29.730 Bhutan Education and Skills Training Project 00:12:29.730 --> 00:12:34.570 delivered by Humber Polytechnic. So increased 00:12:34.570 --> 00:12:37.970 opportunities for paid employment for technical 00:12:37.970 --> 00:12:42.059 vocation. Graduates, especially women and people 00:12:42.059 --> 00:12:44.480 with disabilities, that's the expected outcome. 00:12:44.799 --> 00:12:47.879 They have three indicators to see if they've 00:12:47.879 --> 00:12:50.200 actually achieved that increased opportunity 00:12:50.200 --> 00:12:54.059 for paid employment. The first one is the percent 00:12:54.059 --> 00:12:57.440 of targeted students who access a career service 00:12:57.440 --> 00:13:02.450 and by field of study. Yes, they can record the 00:13:02.450 --> 00:13:05.889 percent of students who actually accessed a service 00:13:05.889 --> 00:13:09.250 who were women and or had a disability. If you 00:13:09.250 --> 00:13:12.250 look at their PMF, their quote analysis of records 00:13:12.250 --> 00:13:15.509 at the TVEN Institute needs to add analysis of 00:13:15.509 --> 00:13:19.669 records using standardized criteria of what is 00:13:19.669 --> 00:13:22.750 needed to get a yes, this student did access 00:13:22.750 --> 00:13:25.950 a career service. Right now, we do not know if 00:13:25.950 --> 00:13:28.960 the field officer responsible for this analysis 00:13:28.960 --> 00:13:31.820 is in a conflict of interest employed by the 00:13:31.820 --> 00:13:33.919 project. So that's a problem right now. If you 00:13:33.919 --> 00:13:36.659 look at the PMF, there's somebody inside the 00:13:36.659 --> 00:13:40.519 project being paid to tick off the box that says, 00:13:40.639 --> 00:13:43.759 yep, you've got access to a career service. Ideally, 00:13:43.759 --> 00:13:46.100 what they should be doing instead is bringing 00:13:46.100 --> 00:13:49.639 in an external evaluator who by themselves independently 00:13:49.639 --> 00:13:53.519 figures out, if I go into that institute, vocational 00:13:53.519 --> 00:13:56.700 institute, training institute, what checklist 00:13:56.700 --> 00:13:59.419 do they have to meet for me to say yeah they've 00:13:59.419 --> 00:14:02.720 got a career service going and yes the students 00:14:02.720 --> 00:14:04.980 actually accessed it so that would be better 00:14:04.980 --> 00:14:09.059 so I don't give that one a plus for the indicator 00:14:09.059 --> 00:14:10.980 I just think it's got a conflict of interest 00:14:10.980 --> 00:14:14.940 and we don't know that it's quite simply that 00:14:14.940 --> 00:14:18.100 the other thing I pointed out is instead of looking 00:14:18.100 --> 00:14:21.159 at the records of the institute why not ask the 00:14:21.159 --> 00:14:25.759 students directly did you get access To a career 00:14:25.759 --> 00:14:28.500 service, yes or no? I think that would be a better 00:14:28.500 --> 00:14:31.840 data source than the records of the institute, 00:14:31.899 --> 00:14:34.519 which they currently have in the PMF. So that's 00:14:34.519 --> 00:14:38.200 a no -no. The second indicator was percent of 00:14:38.200 --> 00:14:43.100 industry TVET partnerships that are active. But 00:14:43.100 --> 00:14:45.980 remember, the outcome is increased opportunities 00:14:45.980 --> 00:14:49.799 for paid employment for TVET graduates. Paid 00:14:49.799 --> 00:14:52.940 employment, not just activity, right? So here 00:14:52.940 --> 00:14:56.399 I had a beef with that. What about the percent 00:14:56.399 --> 00:14:58.820 of total partnerships that actually have paid 00:14:58.820 --> 00:15:00.460 apprenticeships? What do they mean by active? 00:15:00.620 --> 00:15:03.440 Remember, the outcome statement is increased 00:15:03.440 --> 00:15:08.480 opportunities for paid employment. Current data 00:15:08.480 --> 00:15:11.799 source is just reviewing the meeting minutes, 00:15:11.940 --> 00:15:15.820 if you go to the PMF, between the TVET institution 00:15:15.820 --> 00:15:18.440 and the company. That's not good enough, since 00:15:18.440 --> 00:15:21.580 the expected outcome is paid employment for the 00:15:21.580 --> 00:15:24.299 TVET graduate. So that's not very good. You don't 00:15:24.299 --> 00:15:26.320 want to just look at the minutes. You want to 00:15:26.320 --> 00:15:30.220 ask even the students directly, are you getting 00:15:30.220 --> 00:15:32.860 a paid apprenticeship? That would be a better 00:15:32.860 --> 00:15:35.659 measure, a better indicator for whether or not 00:15:35.659 --> 00:15:38.100 they've actually increased opportunities for 00:15:38.100 --> 00:15:41.379 paid employment for TVET graduates, right? So 00:15:41.379 --> 00:15:44.820 that's a no -no. The third indicator is percent 00:15:44.820 --> 00:15:48.259 of TVET institutions that have a career service 00:15:48.259 --> 00:15:51.700 specific to women and people with disabilities. 00:15:53.039 --> 00:15:54.919 So it's very similar to the first indicator, 00:15:55.200 --> 00:15:58.340 but they're now being more precise with it's 00:15:58.340 --> 00:16:02.220 to women and people with disabilities. Same problem 00:16:02.220 --> 00:16:05.000 that I mentioned previously, right? They've got 00:16:05.000 --> 00:16:07.759 to ask the women and the people with disabilities 00:16:07.759 --> 00:16:11.179 directly. That would be better. So that's not 00:16:11.179 --> 00:16:15.120 very good. So the next outcome is increased. 00:16:15.580 --> 00:16:21.820 Whoops, excuse me. Yes. They have a career service 00:16:21.820 --> 00:16:24.200 specific to women and people with disabilities 00:16:24.200 --> 00:16:28.080 or they do not. So again, the same problem. There's 00:16:28.080 --> 00:16:30.740 a bias. They got the field officer doing it instead 00:16:30.740 --> 00:16:34.259 of an independent external evaluator. So what 00:16:34.259 --> 00:16:36.740 I've already repeating myself here, paid employment 00:16:36.740 --> 00:16:39.700 for graduates suggests a better indicator would 00:16:39.700 --> 00:16:43.220 be the percent of graduates who got paid employment, 00:16:43.500 --> 00:16:46.740 not the percent of TVN institutions that have 00:16:46.740 --> 00:16:51.240 a career service for the target groups. Now, 00:16:51.259 --> 00:16:52.759 you could argue they need to have the career 00:16:52.759 --> 00:16:55.799 service first before they get the paid apprenticeship. 00:16:56.480 --> 00:17:00.120 Yes, maybe, possibly. But that's not what the 00:17:00.120 --> 00:17:03.360 outcome is saying. The outcome is clear. Increased 00:17:03.360 --> 00:17:06.579 opportunities for paid employment. So cut to 00:17:06.579 --> 00:17:09.339 the chase. Go straight to that. Find out how 00:17:09.339 --> 00:17:12.220 many of the graduates, percent, actually got 00:17:12.220 --> 00:17:15.900 paid employment or paid apprenticeships. That 00:17:15.900 --> 00:17:20.140 would be more valid measure of the outcome. The 00:17:20.140 --> 00:17:24.880 next outcome is improved capacity of TVET students, 00:17:25.319 --> 00:17:27.759 especially women and people with disabilities, 00:17:27.799 --> 00:17:31.839 to engage in entrepreneurial activities, including 00:17:31.839 --> 00:17:36.400 in green business. And they have, let me see, 00:17:36.519 --> 00:17:40.200 three indicators for that outcome. Let's look 00:17:40.200 --> 00:17:43.319 at the first one. Percent of targeted students 00:17:43.319 --> 00:17:46.700 who have developed a business plan. And I've 00:17:46.700 --> 00:17:48.880 mentioned this in previous episodes with other 00:17:48.880 --> 00:17:53.240 organizations. Same problem. Even if the percent 00:17:53.240 --> 00:17:56.240 increases, the quality of the business plan could 00:17:56.240 --> 00:18:00.299 be poor. Remember, the outcome is improved capacity 00:18:00.299 --> 00:18:06.420 of students to engage. Well, okay, they develop 00:18:06.420 --> 00:18:09.079 a business plan. You could argue that's sufficient 00:18:09.079 --> 00:18:12.660 engagement, but really, no. They need to produce 00:18:12.660 --> 00:18:15.759 a good one. That's why... They have this technical 00:18:15.759 --> 00:18:18.380 vocational institute to train them on that, right? 00:18:19.740 --> 00:18:23.480 So the PMF indicates no criteria on what is considered 00:18:23.480 --> 00:18:26.480 a good business plan. Remember, the outcome is 00:18:26.480 --> 00:18:29.019 improved capacity of these students to engage 00:18:29.019 --> 00:18:32.039 in entrepreneurial activity. So they produce 00:18:32.039 --> 00:18:35.319 a business plan. which is improved capacity compared 00:18:35.319 --> 00:18:38.640 to no business plan at all. But those business 00:18:38.640 --> 00:18:40.559 plans should be checked on meeting a minimum 00:18:40.559 --> 00:18:43.819 level of quality. Second, there is again a conflict 00:18:43.819 --> 00:18:46.480 of interest. As the M &E officer paid by the 00:18:46.480 --> 00:18:50.240 project, not an external evaluator, is responsible 00:18:50.240 --> 00:18:55.380 for getting this percent. Third, this percent 00:18:55.380 --> 00:18:58.240 is only measured once per year and not until 00:18:58.240 --> 00:19:02.660 year two, and not until year three. Too little. 00:19:03.000 --> 00:19:05.700 too late if they want to show improved capacity 00:19:05.700 --> 00:19:09.420 due to the project training rather than other 00:19:09.420 --> 00:19:13.039 factors. The second indicator for improved capacity 00:19:13.039 --> 00:19:16.740 of TVET students to engage is percent of graduates 00:19:16.740 --> 00:19:20.400 who identify a high level of satisfaction with 00:19:20.400 --> 00:19:24.099 the quality of entrepreneurship support programs 00:19:24.099 --> 00:19:30.279 provided by TVET institutions. Again, if you're 00:19:30.279 --> 00:19:34.450 asking the graduates, So satisfaction has got 00:19:34.450 --> 00:19:38.109 nothing to do with improved capacity. Sure, they 00:19:38.109 --> 00:19:41.309 might be satisfied, but we're not measuring that 00:19:41.309 --> 00:19:44.029 in the outcome, right? We want to know if they 00:19:44.029 --> 00:19:47.589 technically are able to produce, among other 00:19:47.589 --> 00:19:50.890 things, a good business plan, right? So I would 00:19:50.890 --> 00:19:53.970 take this out. Or improved capacity to engage. 00:19:54.289 --> 00:19:58.309 When they say capacity, I see that as technical 00:19:58.309 --> 00:20:02.240 capacity to produce a business plan. or whatever, 00:20:02.440 --> 00:20:05.940 right? So I would take satisfaction out. Of course, 00:20:05.960 --> 00:20:07.940 it's a good indicator telling people maybe there's 00:20:07.940 --> 00:20:09.920 something wrong with the program, but it's not 00:20:09.920 --> 00:20:13.640 related to the outcome. Third indicator is percent 00:20:13.640 --> 00:20:16.619 of student business plans that focus on green 00:20:16.619 --> 00:20:20.779 business. Okay, again, same problem as above. 00:20:21.119 --> 00:20:24.420 PMF does not mention any criteria as to what 00:20:24.420 --> 00:20:27.940 constitutes, quote, a good green business plan 00:20:27.940 --> 00:20:31.279 of high quality. Now, perhaps this is assumed, 00:20:31.420 --> 00:20:36.519 but we do not know. If it does exist, that should 00:20:36.519 --> 00:20:39.500 be mentioned in the PMF. Again, not a valid indicator 00:20:39.500 --> 00:20:41.779 due to conflict of interest being measured by 00:20:41.779 --> 00:20:44.619 someone within the project. It should be an external 00:20:44.619 --> 00:20:47.740 evaluator instead. And again, too little, too 00:20:47.740 --> 00:20:51.359 late, and only measure it once per year and only 00:20:51.359 --> 00:20:53.500 starting in year three of the project. Now, they 00:20:53.500 --> 00:20:55.579 could argue they're not doing business plans 00:20:55.579 --> 00:20:58.759 until year three of the project. Good. Fair enough. 00:20:59.849 --> 00:21:02.029 One could argue that even an increase in the 00:21:02.029 --> 00:21:05.690 percent of students producing low -quality green 00:21:05.690 --> 00:21:08.549 business plans is better than no business plans 00:21:08.549 --> 00:21:13.750 at all. I suggest for $4 .9 million in training, 00:21:13.910 --> 00:21:16.990 these students on business plans, we should at 00:21:16.990 --> 00:21:19.490 least measure and evaluate whether these green 00:21:19.490 --> 00:21:23.269 business plans meet minimum quality standards. 00:21:24.069 --> 00:21:28.150 So that's not a good indicator. You can't just 00:21:28.150 --> 00:21:31.869 say, Focus on green business. We've got to see 00:21:31.869 --> 00:21:36.349 if it's good, good quality, using some external 00:21:36.349 --> 00:21:40.710 criteria of what constitutes a good green business 00:21:40.710 --> 00:21:43.990 plan. Next outcome, increased effectiveness of 00:21:43.990 --> 00:21:47.569 TVET partner institutions to deliver inclusive, 00:21:47.849 --> 00:21:50.369 industry -relevant, environmentally responsive 00:21:50.369 --> 00:21:55.529 skills training programs to students. First indicator 00:21:55.529 --> 00:21:59.150 here for this outcome is percent of TVET instructors 00:21:59.150 --> 00:22:03.630 who apply at least one named teaching technique 00:22:03.630 --> 00:22:07.170 learned through project activities. It's a great 00:22:07.170 --> 00:22:09.650 indicator, but the same problem. Got the wrong 00:22:09.650 --> 00:22:13.269 person measuring it. The PMS shows that the project 00:22:13.269 --> 00:22:17.029 M &E staff will be observing the TVET instructors 00:22:17.029 --> 00:22:21.410 to see if they are applying these teaching techniques. 00:22:22.009 --> 00:22:24.190 This is a conflict of interest. They mention 00:22:24.190 --> 00:22:28.269 a survey in the PMF, but it is not clear if the 00:22:28.269 --> 00:22:31.710 TVET instructors get the survey to complete themselves, 00:22:32.210 --> 00:22:36.430 which would be unacceptable due to bias, or if 00:22:36.430 --> 00:22:39.730 they actually surveyed the students to ask them 00:22:39.730 --> 00:22:43.130 if their instructor delivered one of these teaching 00:22:43.130 --> 00:22:46.450 techniques. And again, it is only done in year 00:22:46.450 --> 00:22:52.200 four and year five. That sounds a bit late. for 00:22:52.200 --> 00:22:55.000 this project. But maybe that's only those years 00:22:55.000 --> 00:22:58.839 that the students receive this, quote, industry 00:22:58.839 --> 00:23:01.359 -relevant, environmentally responsive skills 00:23:01.359 --> 00:23:05.240 training, the students receiving it. So either 00:23:05.240 --> 00:23:07.420 way, I wouldn't accept this indicator because 00:23:07.420 --> 00:23:10.740 the data source is wrong. An external evaluator 00:23:10.740 --> 00:23:13.519 should be observing these TVET instructors to 00:23:13.519 --> 00:23:16.220 see if they are applying these teaching techniques, 00:23:16.400 --> 00:23:20.339 preferably using a pre -developed checklist before 00:23:20.339 --> 00:23:23.079 they enter the classroom to observe the TVET 00:23:23.079 --> 00:23:28.779 instructor, right? Next indicator for this outcome 00:23:28.779 --> 00:23:33.259 is percent of TVET instructors who receive positive 00:23:33.259 --> 00:23:37.220 course evaluations. Now, we have to assume that 00:23:37.220 --> 00:23:40.640 positive evaluation is equivalent to deliver 00:23:40.640 --> 00:23:43.960 relevant, inclusive, environmentally responsible, 00:23:44.400 --> 00:23:48.759 responsive skills training. The data source is 00:23:48.759 --> 00:23:52.160 TVET institution records. I guess the course 00:23:52.160 --> 00:23:55.940 evaluations completed by the students. The design 00:23:55.940 --> 00:23:58.319 of these course evaluations for the students 00:23:58.319 --> 00:24:01.019 to complete should be completed and delivered 00:24:01.019 --> 00:24:05.319 to the students by an external evaluator, not 00:24:05.319 --> 00:24:08.079 the Project M &E staff, which again is a conflict 00:24:08.079 --> 00:24:10.859 of interest. So you'll want to make sure the 00:24:10.859 --> 00:24:13.779 course evaluation form is specific to student 00:24:13.779 --> 00:24:17.710 ratings of the instructor. specific to delivering, 00:24:17.809 --> 00:24:21.509 quote, relevant, inclusive, environmentally responsive 00:24:21.509 --> 00:24:25.210 skills training. So that should be put in the 00:24:25.210 --> 00:24:28.809 PMF. It's not there. We don't know. Could just 00:24:28.809 --> 00:24:31.069 be the M &E officer in the project ticking off 00:24:31.069 --> 00:24:34.170 a box and saying, oh yeah, it's positive. We 00:24:34.170 --> 00:24:36.930 don't know. So it should be more explicit. And 00:24:36.930 --> 00:24:38.890 again, an external evaluator should be doing 00:24:38.890 --> 00:24:42.849 that, not the project M &E officer. Because there's 00:24:42.849 --> 00:24:45.609 a potential for bias. So just eliminate it, right? 00:24:46.119 --> 00:24:49.059 Next outcome, strengthened capacity of targeted 00:24:49.059 --> 00:24:53.140 TVET partner institutions to develop gender responsive 00:24:53.140 --> 00:24:55.980 programs that meet economic and environmental 00:24:55.980 --> 00:24:59.119 needs. So here, first indicator for that outcome, 00:24:59.500 --> 00:25:03.900 percent of courses developed through that industry 00:25:03.900 --> 00:25:07.480 partners confirm as relevant to emerging economic 00:25:07.480 --> 00:25:12.119 sectors. From the indicator, it is clear that 00:25:12.119 --> 00:25:14.769 it will be up to these industry partners. to 00:25:14.769 --> 00:25:18.069 declare and confirm that each course indeed has 00:25:18.069 --> 00:25:20.890 met some sort of criteria for them to conclude, 00:25:21.089 --> 00:25:24.250 quote, that course is relevant to emerging economic 00:25:24.250 --> 00:25:28.329 sectors. The problem is that the PMF shows the 00:25:28.329 --> 00:25:32.029 data source as, quote, meeting records with industry 00:25:32.029 --> 00:25:35.710 partners and the Project M &E staff, conflict 00:25:35.710 --> 00:25:38.250 of interest, reviewing those meeting records 00:25:38.250 --> 00:25:40.990 to determine if the course is indeed, quote, 00:25:41.069 --> 00:25:44.549 relevant. This is not acceptable, right? for 00:25:44.549 --> 00:25:49.150 those two reasons. So instead, an external evaluator 00:25:49.150 --> 00:25:51.829 should survey the industry partners to get either 00:25:51.829 --> 00:25:56.390 their prior approval before the course is delivered, 00:25:56.549 --> 00:25:59.410 that it is relevant, they can review the course 00:25:59.410 --> 00:26:03.089 syllabus. It says in the indicator, quote, developed, 00:26:03.130 --> 00:26:07.789 not yet delivered. So that would seem to be the 00:26:07.789 --> 00:26:10.269 right approach. They're developing these courses 00:26:10.269 --> 00:26:14.069 to make sure they are gender responsive. Right? 00:26:14.789 --> 00:26:20.430 So they need to survey the industry people, say, 00:26:20.549 --> 00:26:24.609 is this relevant? Is it gender responsive? And 00:26:24.609 --> 00:26:27.849 the external evaluator needs to come up with 00:26:27.849 --> 00:26:31.730 a questionnaire and criteria so that they can 00:26:31.730 --> 00:26:36.789 conclude after that that the industry partners 00:26:36.789 --> 00:26:39.470 do indeed confirm that the course is relevant. 00:26:40.930 --> 00:26:45.240 The second... indicator is percent of courses 00:26:45.240 --> 00:26:48.460 developed that incorporate quote a gender -related 00:26:48.460 --> 00:26:53.339 concept strategy or skill so again the data source 00:26:53.339 --> 00:26:57.779 course curriculum and its review is good but 00:26:57.779 --> 00:27:00.140 again we've got the same problem we mentioned 00:27:00.140 --> 00:27:03.900 before conflict of interest right project M &E 00:27:03.900 --> 00:27:06.059 staff are reviewing it which is a conflict of 00:27:06.059 --> 00:27:08.990 interest also No mention of pre -established 00:27:08.990 --> 00:27:12.190 criteria to be used when reviewing that course 00:27:12.190 --> 00:27:15.529 content to conclude whether course content, quote, 00:27:15.589 --> 00:27:20.009 incorporates that gender concept. So what would 00:27:20.009 --> 00:27:22.410 be better is getting an external evaluator to 00:27:22.410 --> 00:27:24.950 do the review as well as develop the criteria 00:27:24.950 --> 00:27:30.049 to be used for that review. And another indicator 00:27:30.049 --> 00:27:33.750 for this outcome, percent of courses developed 00:27:33.750 --> 00:27:37.180 that incorporate an environmental. sustainability 00:27:37.180 --> 00:27:40.460 concept, strategy, or skill. Same problem as 00:27:40.460 --> 00:27:43.420 before. They need to get an external evaluator 00:27:43.420 --> 00:27:46.019 in there to do the review as well as to develop 00:27:46.019 --> 00:27:49.640 the criteria to be used for that review to lead 00:27:49.640 --> 00:27:52.200 to the conclusion of whether course content incorporates, 00:27:52.519 --> 00:27:55.819 quote, environmental sustainability concept, 00:27:56.160 --> 00:27:59.099 right? So you can get the pattern here. There's 00:27:59.099 --> 00:28:03.000 conflicts of interest going on with some of these 00:28:03.000 --> 00:28:06.880 indicators, which is not acceptable. So next 00:28:06.880 --> 00:28:10.480 outcome, increased effectiveness of national 00:28:10.480 --> 00:28:14.220 TVET agencies and TVET institutions to manage 00:28:14.220 --> 00:28:18.019 a reformed gender and disability inclusive and 00:28:18.019 --> 00:28:23.059 environmentally responsive TVET system. So the 00:28:23.059 --> 00:28:26.660 first indicator is level of confidence among 00:28:26.660 --> 00:28:30.539 TVET officials in their ability to manage a reformed 00:28:30.539 --> 00:28:33.579 TVET system that is gender sensitive, disability 00:28:33.579 --> 00:28:37.690 inclusive. and environmentally responsible. So, 00:28:37.769 --> 00:28:40.829 technical ability to manage an increased effectiveness 00:28:40.829 --> 00:28:44.349 of an agency or institution cannot be measured 00:28:44.349 --> 00:28:48.450 by TVET managers self -reporting their confidence 00:28:48.450 --> 00:28:52.450 in their ability to manage. This self -reporting 00:28:52.450 --> 00:28:55.349 is biased and fails to measure the technical 00:28:55.349 --> 00:28:59.490 ability to manage. In addition, they collected 00:28:59.490 --> 00:29:02.549 this self -reporting through focus groups and 00:29:02.549 --> 00:29:08.140 surveys. A waste of money, right? And what could 00:29:08.140 --> 00:29:11.119 they do instead? Well, way back before any of 00:29:11.119 --> 00:29:14.480 you were born, including myself, way back in 00:29:14.480 --> 00:29:18.180 the 1950s, the Educational Testing Service developed 00:29:18.180 --> 00:29:21.720 the in -basket test. It's a test and it still 00:29:21.720 --> 00:29:25.160 exists today, I'm guessing. It's an example of 00:29:25.160 --> 00:29:27.279 where you can test the competency of managers, 00:29:27.519 --> 00:29:30.220 where they have to prioritize tasks they're given. 00:29:30.839 --> 00:29:32.920 defer it to somebody else, do it right away, 00:29:33.240 --> 00:29:38.339 make somebody else do it, delegate. That's just 00:29:38.339 --> 00:29:41.480 an example. But if you're going to train people 00:29:41.480 --> 00:29:44.420 on how to manage, especially with this specific 00:29:44.420 --> 00:29:47.720 content, gender and disability inclusiveness, 00:29:47.759 --> 00:29:51.460 environmental responsibility, you need to design 00:29:51.460 --> 00:29:55.819 a test that can test these managers if they're 00:29:55.819 --> 00:29:59.220 competent to do that, rather than just asking 00:29:59.220 --> 00:30:01.619 them. their confidence level, right? So that's 00:30:01.619 --> 00:30:05.119 just an example. Now, after using that objective 00:30:05.119 --> 00:30:08.079 measure, you find the percent of managers isn't 00:30:08.079 --> 00:30:11.940 going up. Then, and only then, you could hold 00:30:11.940 --> 00:30:16.900 a focus group and ask them, you guys in this 00:30:16.900 --> 00:30:18.900 room, the percent has barely gone up. What's 00:30:18.900 --> 00:30:21.960 going on? What's the problem with your ability 00:30:21.960 --> 00:30:26.700 to manage? Why was there no percent increase 00:30:26.700 --> 00:30:29.200 in those who passed? on the in -basket test. 00:30:30.039 --> 00:30:33.039 That could be an example, right? So that indicator 00:30:33.039 --> 00:30:37.500 is not very good. The next indicator for this 00:30:37.500 --> 00:30:41.460 outcome was percent of revised courses that are 00:30:41.460 --> 00:30:46.099 initiated within their designated timeline. So 00:30:46.099 --> 00:30:48.579 initiating a revised course, perhaps to address 00:30:48.579 --> 00:30:52.079 low confidence reported in their ability to manage, 00:30:52.279 --> 00:30:56.259 I don't know, is still an output rather than 00:30:56.259 --> 00:30:59.180 an expected outcome. This is not related to the 00:30:59.180 --> 00:31:02.019 expected outcome of improving technical ability 00:31:02.019 --> 00:31:06.319 to manage. Now, I suppose you could say it's 00:31:06.319 --> 00:31:09.079 the step before, right? They've learned their 00:31:09.079 --> 00:31:12.279 lesson. They know they're not managing. And so 00:31:12.279 --> 00:31:14.140 they have to turn around and revise the course. 00:31:14.599 --> 00:31:18.039 So that could be argued that, you know, if the 00:31:18.039 --> 00:31:22.180 percent goes up and they do that, that's a measure, 00:31:22.319 --> 00:31:25.940 again, of technical ability. But then somebody 00:31:25.940 --> 00:31:28.799 could argue. Well, we've seen the revised course 00:31:28.799 --> 00:31:33.200 and it's crap. But even if it's not crap and 00:31:33.200 --> 00:31:35.279 it's really good, what is the outcome again in 00:31:35.279 --> 00:31:40.960 this? It's to manage a system. It's a management 00:31:40.960 --> 00:31:44.880 skill they're looking at here. And maybe that's 00:31:44.880 --> 00:31:47.500 a measure of management is producing a high quality 00:31:47.500 --> 00:31:51.759 revised course. So maybe again, they need an 00:31:51.759 --> 00:31:55.569 external evaluator to come in and say. This is 00:31:55.569 --> 00:31:57.890 what these managers have to reach if they're 00:31:57.890 --> 00:32:00.650 going to revise these courses. And I haven't 00:32:00.650 --> 00:32:03.029 checked the data source on this, but I'm assuming 00:32:03.029 --> 00:32:08.130 it's not good. I think I'd have to check, but 00:32:08.130 --> 00:32:10.289 I'm going to assume same problem. They don't 00:32:10.289 --> 00:32:12.750 have an evaluator coming in to look at those 00:32:12.750 --> 00:32:16.849 courses and say, yep, these have improved with 00:32:16.849 --> 00:32:20.130 respect to the outcome. Therefore, the management 00:32:20.130 --> 00:32:22.970 skill has improved, but we don't know at this 00:32:22.970 --> 00:32:27.230 point. Another outcome, enhanced knowledge and 00:32:27.230 --> 00:32:30.069 attitudes about TVET as a viable education and 00:32:30.069 --> 00:32:32.990 employment pathway that is inclusive of women, 00:32:33.130 --> 00:32:35.950 people with disabilities, and vulnerable populations. 00:32:36.569 --> 00:32:40.109 So, first indicator for this outcome, increased 00:32:40.109 --> 00:32:43.690 level of support among community members for 00:32:43.690 --> 00:32:47.069 TVET education. If you look at the performance 00:32:47.069 --> 00:32:49.910 measurement framework, a sample is taken from 00:32:49.910 --> 00:32:52.859 the community members at large to see if the 00:32:52.859 --> 00:32:56.440 percent supporting TVET as a viable pathway has 00:32:56.440 --> 00:33:01.400 increased over time. This is great. It's a great 00:33:01.400 --> 00:33:03.880 indicator. Problem is, there's no comparison 00:33:03.880 --> 00:33:06.599 or control groups, so they cannot support their 00:33:06.599 --> 00:33:10.200 claim that their project was responsible for 00:33:10.200 --> 00:33:13.779 any observed increase in the percent. In addition, 00:33:14.039 --> 00:33:17.140 there is no mention of a one -tailed, right -tailed 00:33:17.140 --> 00:33:19.900 hypothesis test using the sampling Z distribution 00:33:19.900 --> 00:33:23.960 of a simple single proportion. So any observed 00:33:23.960 --> 00:33:26.019 increase in the percent over time could fail 00:33:26.019 --> 00:33:29.440 to be statistically significant with a p -value 00:33:29.440 --> 00:33:35.799 less than 0 .05, right? So either way, they've 00:33:35.799 --> 00:33:38.599 got a problem. At the very minimum, they should 00:33:38.599 --> 00:33:42.079 be doing what any second year undergraduate social 00:33:42.079 --> 00:33:44.660 science major will tell you. They should be comparing 00:33:44.660 --> 00:33:49.339 the percent sample of the population that they've 00:33:49.339 --> 00:33:52.940 just done the community members sampling to see 00:33:52.940 --> 00:33:58.519 if the percent has gone up enough to reach a 00:33:58.519 --> 00:34:01.440 critical value where they can reject the null 00:34:01.440 --> 00:34:04.299 and go with the alternate and say yeah the percent 00:34:04.299 --> 00:34:06.980 has gone up and it's been statistically significant 00:34:06.980 --> 00:34:10.599 and we're at least 95 percent confident that 00:34:10.599 --> 00:34:13.739 there is that observed increase is statistically 00:34:13.739 --> 00:34:18.239 significant otherwise why do the project right 00:34:18.750 --> 00:34:21.010 If you don't do the project and the percent still 00:34:21.010 --> 00:34:25.050 goes up, why spend almost $5 million? You got 00:34:25.050 --> 00:34:28.250 to show it's statistically significant. It's 00:34:28.250 --> 00:34:31.030 effective. Or some people say, not only effective, 00:34:31.349 --> 00:34:37.050 it has impact, right? So that's a problem. Next 00:34:37.050 --> 00:34:40.130 indicator, percent of women and people with disabilities 00:34:40.130 --> 00:34:44.590 from sample communities who view TVET as a viable 00:34:44.590 --> 00:34:49.510 education pathway. Now, From the PMF, a sample 00:34:49.510 --> 00:34:51.329 is taken from women and people with disabilities 00:34:51.329 --> 00:34:55.030 to see if the percent supporting TVET as a viable 00:34:55.030 --> 00:34:58.110 pathway has increased over time. Again, great. 00:34:58.449 --> 00:35:01.570 But again, no comparison groups, no control groups. 00:35:01.949 --> 00:35:05.929 They still need to do a statistical test to see 00:35:05.929 --> 00:35:09.349 if that percent over time has statistically significantly 00:35:09.349 --> 00:35:12.869 increased. Otherwise, there's no point. They 00:35:12.869 --> 00:35:15.989 got to prove that the project has made a statistically 00:35:15.989 --> 00:35:20.639 significant difference. Right? And ideally, you 00:35:20.639 --> 00:35:24.460 want to have a comparison group or a control 00:35:24.460 --> 00:35:28.500 group. Now, they could be delivering this project 00:35:28.500 --> 00:35:32.480 to every single TVAN institution in the country. 00:35:32.599 --> 00:35:37.139 And maybe they are. That's fine. Okay, so no 00:35:37.139 --> 00:35:39.800 point in having a control group. But they still 00:35:39.800 --> 00:35:41.619 have to show that the percentages they've got 00:35:41.619 --> 00:35:44.719 have statistically increased significantly. Otherwise, 00:35:45.079 --> 00:35:50.650 they should shut the program down. Another indicator 00:35:50.650 --> 00:35:55.170 is the percent of TVET institutions that implement 00:35:55.170 --> 00:35:59.269 new recruitment and counseling strategies and 00:35:59.269 --> 00:36:04.849 systems. Now, again, the outcome there is, sorry, 00:36:05.010 --> 00:36:07.429 I went to, there's a new outcome. So let me repeat 00:36:07.429 --> 00:36:10.070 that. There's a new outcome here for that indicator. 00:36:10.250 --> 00:36:13.650 It is strengthened capacity of TVET institutions 00:36:13.650 --> 00:36:17.250 to recruit and counsel students, especially youth, 00:36:17.389 --> 00:36:20.800 women, and people with disabilities. So the first 00:36:20.800 --> 00:36:24.039 indicator for that outcome is percent of TVET 00:36:24.039 --> 00:36:27.239 institutions that implement new recruitment and 00:36:27.239 --> 00:36:30.900 counseling strategies and systems. Again, great 00:36:30.900 --> 00:36:34.119 indicator, but wrong people to see if this percent 00:36:34.119 --> 00:36:36.639 has increased. The PMF shows that individuals 00:36:36.639 --> 00:36:39.440 representing these TVET institutions will be 00:36:39.440 --> 00:36:42.460 asked, did you implement new strategies to recruit 00:36:42.460 --> 00:36:45.260 youth, women, people with disabilities? Yes or 00:36:45.260 --> 00:36:48.599 no? This is done through a survey and interviews. 00:36:49.840 --> 00:36:53.059 Of course, they will say yes due to self -reporting 00:36:53.059 --> 00:36:57.059 social desirability bias. So they shouldn't be 00:36:57.059 --> 00:36:59.780 involved in this. Again, if you look at the PMF, 00:36:59.880 --> 00:37:05.920 it's the institutions, excuse me, serving. Of 00:37:05.920 --> 00:37:08.239 course, they will say yes due to self -reporting. 00:37:08.320 --> 00:37:12.639 So I have to look again at the PMF, but it's 00:37:12.639 --> 00:37:16.360 not mentioning as the data collector, person 00:37:16.360 --> 00:37:22.900 responsible is being. a external evaluator, which 00:37:22.900 --> 00:37:25.440 is what you should be doing instead. Going into 00:37:25.440 --> 00:37:28.880 the TVET institution using a predetermined checklist 00:37:28.880 --> 00:37:33.639 that the institution actually has met the criteria 00:37:33.639 --> 00:37:36.079 of, yes, this TVET has indeed implemented a recruitment 00:37:36.079 --> 00:37:40.199 system. And even if they did, does the institution 00:37:40.199 --> 00:37:43.000 have a system in place to counsel these three 00:37:43.000 --> 00:37:47.119 groups? So external evaluator is what you need 00:37:47.119 --> 00:37:50.110 to do instead. The second indicator for this 00:37:50.110 --> 00:37:53.510 outcome is level of confidence among counseling 00:37:53.510 --> 00:37:57.070 staff at TVAN institutions in their ability to 00:37:57.070 --> 00:37:59.590 effectively counsel students, including women, 00:37:59.690 --> 00:38:01.750 youth, and people with disabilities. Again, as 00:38:01.750 --> 00:38:06.570 you know, you can't do that, especially when 00:38:06.570 --> 00:38:09.469 you explicitly state ability to effectively counsel. 00:38:10.070 --> 00:38:13.789 I mean, wow. You don't ask them how they're feeling. 00:38:13.949 --> 00:38:17.300 You do an external evaluation. of their ability 00:38:17.300 --> 00:38:20.980 to counsel students. There's a whole pile of 00:38:20.980 --> 00:38:23.619 psychometric tools out there to measure. I mean, 00:38:23.639 --> 00:38:26.880 just go to any university in Canada that has 00:38:26.880 --> 00:38:28.739 a master's program in counseling psychology. 00:38:29.340 --> 00:38:32.179 I'm quite sure there's a whole bunch of external 00:38:32.179 --> 00:38:35.500 measures, observation of their counseling with 00:38:35.500 --> 00:38:37.840 the student, rather than asking the counselor 00:38:37.840 --> 00:38:40.739 themselves to report on their self -confidence, 00:38:40.900 --> 00:38:44.659 right? So get an external evaluator, not a good 00:38:44.659 --> 00:38:49.769 indicator. And the last one, same problem. Level 00:38:49.769 --> 00:38:52.429 of confidence among recruitment staff at TVET 00:38:52.429 --> 00:38:54.769 institutions in their ability to effectively 00:38:54.769 --> 00:38:58.369 recruit. So that's the other skill. Counseling, 00:38:58.369 --> 00:39:01.010 now they're looking at recruitment skills. Again, 00:39:01.130 --> 00:39:03.610 same problem. Self -reporting confidence levels 00:39:03.610 --> 00:39:05.929 is not a valid measure of whether recruitment 00:39:05.929 --> 00:39:09.070 staff have the ability to effectively recruit 00:39:09.070 --> 00:39:14.989 students. So that's not good. Next. Expected 00:39:14.989 --> 00:39:18.730 outcome, enhanced access to economic opportunities 00:39:18.730 --> 00:39:22.449 in the labour market for TVET graduates, especially 00:39:22.449 --> 00:39:26.809 women and people with disabilities in high opportunity 00:39:26.809 --> 00:39:31.030 sectors. First indicator, percent of graduates 00:39:31.030 --> 00:39:34.050 who complete entrepreneurial training, who start 00:39:34.050 --> 00:39:38.510 their own business. Great indicator, great measurement 00:39:38.510 --> 00:39:42.000 through tracking. of graduates if you see their 00:39:42.000 --> 00:39:46.079 pmf it's called a tracers study but all that 00:39:46.079 --> 00:39:48.420 money to track these graduates just to ask hey 00:39:48.420 --> 00:39:51.400 dude have you got a business yes or no wouldn't 00:39:51.400 --> 00:39:54.119 you want to ask hey are you making any money 00:39:54.119 --> 00:39:58.219 show me the money so a better indicator would 00:39:58.219 --> 00:40:00.920 be the mean monthly income earned from that business 00:40:00.920 --> 00:40:04.659 and track it over time to see if it has increased 00:40:04.659 --> 00:40:08.420 also no need to have a focus group as they talk 00:40:08.420 --> 00:40:10.730 about in the performance measurement framework, 00:40:10.849 --> 00:40:13.949 at least not at this point, until you first use 00:40:13.949 --> 00:40:17.530 this income indicator. Then if income has not 00:40:17.530 --> 00:40:20.869 increased, you could hold a focus group and ask 00:40:20.869 --> 00:40:23.690 them, what's going on, dudes? You've started 00:40:23.690 --> 00:40:27.650 a business. It's great, but you're not making 00:40:27.650 --> 00:40:31.730 any money from it. So what do we need to do to 00:40:31.730 --> 00:40:35.349 identify what are the problems as to why? you're 00:40:35.349 --> 00:40:37.670 not making any money. Is it because they didn't 00:40:37.670 --> 00:40:39.969 get the labor market analysis right in the first 00:40:39.969 --> 00:40:45.670 place? There's thousands of seamstresses already 00:40:45.670 --> 00:40:47.630 out there in the market that we can't compete 00:40:47.630 --> 00:40:49.849 with, even though I went to this institute and 00:40:49.849 --> 00:40:52.289 learned how to sew and graduate. I can't get 00:40:52.289 --> 00:40:56.489 any income. Maybe the United States or Canada 00:40:56.489 --> 00:40:59.070 are dumping used clothing into the market, which 00:40:59.070 --> 00:41:03.980 I can't compete against. Whatever, right? So 00:41:03.980 --> 00:41:08.000 a better valid indicator of enhanced access would 00:41:08.000 --> 00:41:10.940 not be just starting up a business, but more 00:41:10.940 --> 00:41:16.480 money, right? And the next indicator, level of 00:41:16.480 --> 00:41:19.199 satisfaction among graduates with their economic 00:41:19.199 --> 00:41:22.539 opportunities. Again, they might not be satisfied. 00:41:22.579 --> 00:41:24.340 It tells you, yeah, there's something wrong. 00:41:24.579 --> 00:41:27.280 But what you really want to do is focus on the 00:41:27.280 --> 00:41:31.400 income, right? And of course, we're assuming. 00:41:32.079 --> 00:41:34.159 They've acquired the necessary skill levels. 00:41:34.360 --> 00:41:38.219 I can't see anywhere where there's a certification 00:41:38.219 --> 00:41:41.619 testing program in the institute to make sure 00:41:41.619 --> 00:41:45.320 they've met a minimum standard to say that they're 00:41:45.320 --> 00:41:50.079 a carpenter or a plumber or a mason or a seamstress. 00:41:51.099 --> 00:41:56.780 Right? Or tailor, right? So if we track the income, 00:41:56.920 --> 00:42:01.079 we don't need the satisfaction levels. And remember, 00:42:01.239 --> 00:42:04.239 not other income sources, right? Got to make 00:42:04.239 --> 00:42:06.860 sure that they're actually making money from 00:42:06.860 --> 00:42:10.320 the training that this $5 million project gave 00:42:10.320 --> 00:42:14.539 them in a particular technical vocation, right? 00:42:14.699 --> 00:42:18.139 Got to be very clear about that. It's a big problem 00:42:18.139 --> 00:42:20.820 in Africa where they'll have multiple sources 00:42:20.820 --> 00:42:23.519 of income and then you just ask them and they 00:42:23.519 --> 00:42:26.179 give you a summary number and you think, wait 00:42:26.179 --> 00:42:28.360 a sec, didn't we train you how to be a welder? 00:42:28.880 --> 00:42:30.860 How much money do you make from welding? Nothing. 00:42:30.980 --> 00:42:35.260 Okay. And you'll notice the PMF does indicate 00:42:35.260 --> 00:42:38.179 that the project training will align with, quote, 00:42:38.300 --> 00:42:41.559 labor market needs. So satisfaction should be 00:42:41.559 --> 00:42:44.239 related to income rather than the frustration 00:42:44.239 --> 00:42:47.840 of not getting demand for their services after 00:42:47.840 --> 00:42:51.539 graduation, right? So the next indicator is percent 00:42:51.539 --> 00:42:54.659 of industry partners that hire a TVET graduate 00:42:54.659 --> 00:42:57.809 in the last year by field of study. and economic 00:42:57.809 --> 00:43:00.849 sector. Again, great indicator, but even better 00:43:00.849 --> 00:43:03.550 would be income levels. Because they could be 00:43:03.550 --> 00:43:07.289 hired, but paid a really bad wage, right? Or 00:43:07.289 --> 00:43:09.809 nothing at all. They could be an unpaid intern. 00:43:09.969 --> 00:43:12.650 We don't know what they mean by hired. So cut 00:43:12.650 --> 00:43:16.630 to the chase, find out the incomes. And we need 00:43:16.630 --> 00:43:18.530 to show that the income levels of the graduates 00:43:18.530 --> 00:43:21.190 are higher than those who did not get trained 00:43:21.190 --> 00:43:24.369 in technical vocation. Perhaps people that taught 00:43:24.369 --> 00:43:28.639 themselves to be a carpenter. Plumber or seamstress, 00:43:28.659 --> 00:43:34.000 right? Or tailor. Next outcome is strengthened 00:43:34.000 --> 00:43:36.880 effectiveness of TVET agencies and institutions 00:43:36.880 --> 00:43:40.239 to provide inclusive and environmentally responsive 00:43:40.239 --> 00:43:44.260 TVET education to trainees, especially women 00:43:44.260 --> 00:43:47.500 in high opportunity sectors. First indicator, 00:43:47.820 --> 00:43:51.239 percent of students who enter a TVET institution 00:43:51.239 --> 00:43:54.440 who graduate. Now remember, it's strengthened 00:43:54.440 --> 00:43:58.780 effectiveness. of TVET agencies and institutions 00:43:58.780 --> 00:44:01.960 to provide inclusive and environmentally responsive 00:44:01.960 --> 00:44:06.059 TVET education to trainees. It's very precise, 00:44:06.360 --> 00:44:09.320 but the first indicator has got nothing to do 00:44:09.320 --> 00:44:10.940 with that. It's just saying whether or not they 00:44:10.940 --> 00:44:12.960 graduate or not. We don't want to know that. 00:44:13.099 --> 00:44:16.780 We want to know they not only graduated, but 00:44:16.780 --> 00:44:18.820 they are familiar with inclusive and environmentally 00:44:18.820 --> 00:44:23.460 responsive education, particularly women, right? 00:44:24.559 --> 00:44:27.559 And also, if you read the outcome, it's not about 00:44:27.559 --> 00:44:30.800 the student. It's about the institution being 00:44:30.800 --> 00:44:34.719 inclusive and environmentally responsive in their 00:44:34.719 --> 00:44:38.619 education to the students. Right. So I would 00:44:38.619 --> 00:44:41.940 take it out. Who cares what the percent is of 00:44:41.940 --> 00:44:44.559 students who graduate? It's completely unrelated 00:44:44.559 --> 00:44:48.199 to the outcome. That's gone. Now, the next one 00:44:48.199 --> 00:44:52.519 indicator for this is. level of satisfaction 00:44:52.519 --> 00:44:56.420 among TVET graduates with the quality of their 00:44:56.420 --> 00:44:59.599 TVET education. That's a little closer. It's 00:44:59.599 --> 00:45:03.320 a good indicator since it asks the student if 00:45:03.320 --> 00:45:06.699 they are satisfied that the institutions indeed 00:45:06.699 --> 00:45:09.860 are delivering training that is, quote, inclusive 00:45:09.860 --> 00:45:12.960 and environmentally responsive. So you have to 00:45:12.960 --> 00:45:15.559 make sure that that satisfaction survey questionnaire 00:45:15.559 --> 00:45:19.199 that goes to the students contains questions 00:45:19.199 --> 00:45:23.139 specific. to that expected outcome regarding 00:45:23.139 --> 00:45:27.480 what we just said inclusive and environmentally 00:45:27.480 --> 00:45:30.900 responsive the only issue here is that this tracer 00:45:30.900 --> 00:45:34.659 study is delivered again by the project m e staff 00:45:34.659 --> 00:45:39.239 rather than an external evaluator ideally you 00:45:39.239 --> 00:45:41.800 want an external evaluator to be hired to review 00:45:41.800 --> 00:45:43.960 the course content develop questions for the 00:45:43.960 --> 00:45:47.400 students that measure satisfaction levels specific 00:45:48.030 --> 00:45:50.730 to being, quote, inclusive and environmentally 00:45:50.730 --> 00:45:54.570 responsive. But on the whole, it's a good indicator. 00:45:56.929 --> 00:45:59.650 So I've skipped the other indicator because I 00:45:59.650 --> 00:46:03.570 don't think it's relevant. So just one. So here's 00:46:03.570 --> 00:46:07.130 a new outcome. Increased participation of learners, 00:46:07.349 --> 00:46:10.670 particularly among youth, women, and learners 00:46:10.670 --> 00:46:14.210 in vulnerable situations in TVET education in 00:46:14.210 --> 00:46:17.579 high -opportunity sectors. First outcome indicator 00:46:17.579 --> 00:46:20.460 for this outcome is number of enrolled students 00:46:20.460 --> 00:46:24.519 annually in public TVET institutions. So here 00:46:24.519 --> 00:46:27.380 the only issue is whether this project is necessary 00:46:27.380 --> 00:46:31.920 to achieve increased enrollment in TVET institutions 00:46:31.920 --> 00:46:35.400 and increased percent within this enrollment 00:46:35.400 --> 00:46:38.199 of those who are youth, women, rural, or people 00:46:38.199 --> 00:46:40.780 with disabilities. What about another region? 00:46:41.239 --> 00:46:43.139 within the country where this project is not 00:46:43.139 --> 00:46:45.820 operating, and that regional government has a 00:46:45.820 --> 00:46:48.500 policy in place with the same expected outcomes. 00:46:49.760 --> 00:46:52.480 Now, as I mentioned earlier, if this project 00:46:52.480 --> 00:46:55.340 covers the entire country, then this is not important. 00:46:56.500 --> 00:46:59.440 A review of the project browser indicates a countrywide 00:46:59.440 --> 00:47:03.400 project reaching all TVET institutions. That's 00:47:03.400 --> 00:47:06.059 what I'm assuming. So that's a good indicator. 00:47:07.039 --> 00:47:09.949 It's a good indicator. You want to... you want 00:47:09.949 --> 00:47:12.769 to get those enrollments up, right? And they've 00:47:12.769 --> 00:47:16.250 done some community announcements to try to get 00:47:16.250 --> 00:47:19.610 that enrollment up. Now, the next one is percent 00:47:19.610 --> 00:47:21.750 of graduates who are employed in their field 00:47:21.750 --> 00:47:26.190 within six months of graduation. Again, percent 00:47:26.190 --> 00:47:28.070 of graduates employed in their field is good, 00:47:28.130 --> 00:47:30.869 but an even better indicator would be their actual 00:47:30.869 --> 00:47:34.389 income from that employment. But I'm still accepting 00:47:34.389 --> 00:47:37.769 this indicator as a good one. And the disaggregation 00:47:37.769 --> 00:47:41.250 is really good. because you want to see those 00:47:41.250 --> 00:47:43.630 percents for women, people with disabilities, 00:47:43.710 --> 00:47:46.349 since that's a focus of the project. But again, 00:47:46.530 --> 00:47:49.550 there is no control or comparison group. So if 00:47:49.550 --> 00:47:52.469 those percents go up, they need to show that 00:47:52.469 --> 00:47:55.170 they're going up statistically significantly, 00:47:55.670 --> 00:47:58.530 right? So they need to take a sample of those 00:47:58.530 --> 00:48:02.030 graduates at different points in time, find out 00:48:02.030 --> 00:48:04.030 what percent of the sample actually say yeah. 00:48:04.699 --> 00:48:07.159 I got employed within six months of graduation. 00:48:07.420 --> 00:48:10.119 It's a simple yes. And compare that over time 00:48:10.119 --> 00:48:15.420 using a one -tailed, right -tailed alternative 00:48:15.420 --> 00:48:20.739 hypothesis with a P value less than 0 .05, right? 00:48:22.380 --> 00:48:26.500 The second indicator here is level of satisfaction 00:48:26.500 --> 00:48:30.179 within the industry of the ability of TVET graduates 00:48:30.179 --> 00:48:35.289 to fill labor market needs. in an environmentally 00:48:35.289 --> 00:48:40.070 responsive manner. We are assuming that the industry 00:48:40.070 --> 00:48:43.050 people who are interviewed support environmentally 00:48:43.050 --> 00:48:45.929 responsive manner. If the assumption is correct, 00:48:46.110 --> 00:48:49.329 the survey used here in the PMF would be acceptable, 00:48:49.550 --> 00:48:52.949 and this indicator complements the claim. The 00:48:52.949 --> 00:48:55.530 graduates are indeed being environmentally responsive. 00:48:56.010 --> 00:48:59.550 Again, an external evaluator, not the project 00:48:59.550 --> 00:49:02.690 staff, would be better for designing and delivering. 00:49:03.150 --> 00:49:05.389 the survey questionnaire. But I still went ahead 00:49:05.389 --> 00:49:07.989 and said this is a good indicator, despite that. 00:49:10.130 --> 00:49:13.969 The other indicator is, finally, level of community 00:49:13.969 --> 00:49:19.610 support for TVET as a means to promote women's 00:49:19.610 --> 00:49:21.710 equality. Because, you know, some people may 00:49:21.710 --> 00:49:24.429 dispute whether women should be involved in technical 00:49:24.429 --> 00:49:27.849 locations, right? And again, the indicator here 00:49:27.849 --> 00:49:31.630 is a percent surveyed from the PMF. And the percent 00:49:31.630 --> 00:49:34.309 could increase, but not enough to support effectiveness 00:49:34.309 --> 00:49:37.469 at P less than 0 .05. So there's no control group, 00:49:37.550 --> 00:49:40.750 no comparison group. So at the very least, they've 00:49:40.750 --> 00:49:45.530 got to do a proper sampling, two different points 00:49:45.530 --> 00:49:50.429 in time, and do a proper statistical test. And 00:49:50.429 --> 00:49:53.630 any undergraduate student majoring in the social 00:49:53.630 --> 00:49:57.710 sciences will learn the importance of inferential 00:49:57.710 --> 00:50:00.679 statistics and showing that any increase in the 00:50:00.679 --> 00:50:03.739 percent needs to be statistically significant 00:50:03.739 --> 00:50:08.300 to make the claim that it has increased not just 00:50:08.300 --> 00:50:12.480 due to chance, right? So my summary from all 00:50:12.480 --> 00:50:16.340 of this, the 29 outcome indicators for the 11 00:50:16.340 --> 00:50:20.659 expected outcomes, is that only four of the 29 00:50:20.659 --> 00:50:23.860 outcome indicators are valid measures of the 00:50:23.860 --> 00:50:28.360 project's 11 expected outcomes. So only two out 00:50:28.360 --> 00:50:33.449 of the 11, project outcomes could Humber Polytechnic 00:50:33.449 --> 00:50:36.570 claim that they've achieved their outcomes. So 00:50:36.570 --> 00:50:39.170 thank you for listening. That's the end of my 00:50:39.170 --> 00:50:43.909 review. And like I said, if you want to get the 00:50:43.909 --> 00:50:47.309 PMF and my critique in writing, you can just 00:50:47.309 --> 00:50:50.949 email me at evaluatecanadaid at gmail .com and 00:50:50.949 --> 00:50:53.530 I'll be happy to send it to you. Feel free to 00:50:53.530 --> 00:50:57.389 make any comments in the Spotify platform. where 00:50:57.389 --> 00:50:59.710 you now can make comments on any of the episodes. 00:50:59.949 --> 00:51:04.190 And stay tuned. My next episode is going to deal 00:51:04.190 --> 00:51:07.449 with an international organization that has finally 00:51:07.449 --> 00:51:13.150 decided to publish its PMF on its website, along 00:51:13.150 --> 00:51:18.789 with the data for their percent indicators. Unfortunately, 00:51:18.909 --> 00:51:22.730 in Canada, as you can see, we don't have any 00:51:22.730 --> 00:51:26.650 PMFs on any of these. organizations like Humber 00:51:26.650 --> 00:51:29.769 Polytechnic that are funded by the government 00:51:29.769 --> 00:51:31.829 of Canada. It should be a requirement that their 00:51:31.829 --> 00:51:35.550 PMF is on their website along with the data that 00:51:35.550 --> 00:51:37.829 goes with it. If you go to the project browser, 00:51:38.090 --> 00:51:42.150 they have data there, but it's absolutely useless 00:51:42.150 --> 00:51:44.769 data because it's not related to their outcomes 00:51:44.769 --> 00:51:47.070 or their outcome indicators. This is why I have 00:51:47.070 --> 00:51:50.670 to go and ask for the PMF through an access to 00:51:50.670 --> 00:51:54.590 information request. Thank you for listening. 00:51:55.599 --> 00:51:58.699 Stay tuned for the next episode. Bye for now.