So, welcome to the Improving Development Evaluation
Podcast. I'm your host, David Wand, and in this
episode, we're going to feature Humber Polytechnic
International Development Institute. You can
learn more about the Humber Polytechnic International
Development Institute if you go to their website,
which will be in the episode notes. It's humber
.ca forward slash global forward slash international
hyphen development hyphen institute hyphen IDI.
And we're going to look at a project that they
received $4 ,950 ,000 from the government of
Canada. That's you guys, some of you listening,
the Canadian taxpayers. And it's for a project
that they delivered in Bhutan. called Education
and Skills Training. And I'll also put in the
episode notes, the link to the project description.
Of course, I'm going to give you a bit of background
on what this project and the $4 ,950 ,000 services
it actually delivered. And as usual, when it's
done, I'll be sending my critique, this link
to the episode, along with the performance measurement
framework. to the Humber Polytechnic to invite
them to respond to my critique if they wish.
And of course, I will always send it to the minister
responsible for international development, which
is I think called the Secretary of State for
International Development. I'll also send it
to, of course, all the other political party
shadow critics responsible for international
development. And I'm always recommending... that
instead of us trying to chase the performance
measurement framework, that all of these agencies,
including Humber, Polytechnic, put on their website
their performance measurement framework, as well
as the data, which we aren't getting right now.
But I do have some good news. There is an international
organization that actually is starting to do
it. So in future episodes, I will be critiquing
their performance measurement framework that
is on their website. So without further delay,
let's get into this project in Bhutan. And one
last point. If you're listening to this podcast
on Spotify, Spotify has now added a comment section.
So feel free to leave your comments in response
to any of my episodes that are up there. I look
forward to reading your comments. And if you
ever want any of my performance measurement frameworks
along with my critiques, that are produced in
writing. You can send me an email at evaluatecanadaaid
at gmail .com. That'll be in the episode notes.
And feel free to request, and I'm happy to send
them to you. So this project in Bhutan, delivered
by Humber Polytechnic International Development
Institute. They basically have six target groups
that they deliver services to with this $4 ,950
,000. Target group number one are technical vocation
institution teachers. Target group number two,
technical vocation institute curriculum developers.
Number three, technical vocation industry representatives.
Number four, technical vocation institutes. Number
five, all the staff in the technical vocation
institution. And number six, parents and community
leaders at large. So you might be wondering,
what services do these target groups get? Well,
the technical vocation teachers, they're receiving
training on gender equality to incorporate within
industry. They're also receiving training on
career counseling for students, and in particular
women students and students who have disabilities.
They're also being trained on ability to coach
students in self -employment to be gender sensitive
and environmentally responsible. The industry
staff target group. Industry representatives,
they're being trained on how to best accommodate
women and people with disability students when
they graduate from these technical vocation institutes.
A third target group, technical vocation institute
curriculum developers, they're being trained
by Humber Polytechnic to be able to incorporate
gender equality into the student curriculum.
And the technical vocation institutions by themselves
as another target group, they are, it's not really
training, but it just says in the performance
measurement framework that their job is to set
up incubation centers to enable students to pilot
business ideas. They're also responsible for,
with this funding, to set up resource centers.
to deliver knowledge to students. Another target
group are all of the staff at the Technical Vocation
Institutes where they do this training in technical
vocations. That training is on gender equality,
also on gender sensitive, disability inclusive
admissions, advising and counseling, and also
gender inclusive recruitment. And finally, the
last target group which are parents and community
leaders in the public at large, is promoting
technical vocation to them for people with disabilities
through events. And also for women, promoting
technical vocation to women through education
fairs and education forums. So that gives you
an idea of who is getting these various services
and these various target groups. So before we
get into... the outcomes and the outcome indicators
to see if all these wonderful services were achieving
the outcomes, we need to digress a bit and say
to maybe the staff, although Bhutan is not really
that hot and humid in the global south, perhaps.
I've been to Nepal, but I haven't been to Bhutan.
There may be times. where it is hot and humid.
So people living in the Global South, or you
have to go to the Global South to, believe it
or not, evaluate a project, or maybe you just
live there right now, I highly recommend two
products to protect yourself from the sun in
hot and humid climates that are mostly going
on in the Global South. Sunglasses you can get
at akila .la, A -K -I -L -A dot L -A. You can
get prescription lenses also, all made from 50
% recycled material. You get a 10 % discount
if you use my link in the episode notes. I will
get a 15 % commission when you purchase any product
using that promotional link. So you have your
sunglasses to protect you from the sun and the
global south, but you need the right clothes
to be comfortable in that hot. and human climate.
So I would recommend my second product, and that's
men's polos, tees, and button -ups, but I'm quite
sure women would find them acceptable too. And
you can go to hypernaturalstyle .com. That's
hypernaturalstyle .com. And you get 20 % off
any product when you use my promotional link,
which you'll see in the episode notes. And if
you do that and make a purchase, I get a 20 %
commission on the total purchase you made when
you use my promotional link. Just go to the link
and you'll go straight to the company's website
for purchase. So what is so cool about these
Hypernatural t -shirts? They have a patent pending
Hypercool Jade technology that when it's hot,
lowers your body temperature by three to five
degrees. They're also breathable thanks to Supina
cotton. And they are sold in premium retailers
like Nordstrom. And were voted the best men's
polo shirts in 2024 in Men's Journal, Forbes,
and Esquire. So check those out. And now we're
going to get into the Performance Measurement
Framework, PMF. And we'll say, let's start off
with, you have an idea of the target groups,
the services they receive. So let's go through
the various outcomes. So what you're going to
discover first to begin with is there are 29
outcome indicators. determining whether or not
11 expected outcomes in this $4 ,950 ,000 project
were actually achieved. So we're going to go
through those 29 outcome indicators for those
11 expected outcomes, and then we'll have a conclusion
as to how many of those were achieved. I should
also point out to you that Humber Polytechnic
offers a postgraduate diploma program in public
administration, which I had the opportunity about,
what is it, 2008. I taught two courses there,
and one of them was supposed service quality.
But now they finally have realized how bad that
was. That's what you get when an MBA is running
the school of business and doesn't understand
that when you're evaluating human services, it's
more than just asking the human being after they
got the service, whether they have been customer
satisfied with the service they received. There's
a whole field of evaluation and psychometrics
that you have to be involved with, as you've
noticed from my podcast. But now they have improved.
the postgraduate Humber Polytechnic program,
diploma program in public administration. They
have two new courses instead, which get it right.
They have one in project design and one in project
evaluation. And the first one, project design,
guess what? They teach the students how to design
a performance measurement framework, how to properly
design an outcome indicator so it reliably...
and validly measures the outcome that the project
is trying to achieve. Why do I bring it up here?
Well, when I finish my critique, I'm going to
send it to them and suggest that they share it
with the instructor for that course. It's BUS5014,
Project Design and Planning, so that maybe their
instructor will incorporate it into the curriculum
as an example. A real case study of not any project,
but actually a project that the Humber Polytechnic
is doing itself. So it's relevant because they're
going to Humber Polytechnic to study this. So
you would think it would be logical and helpful
for them to incorporate this performance measurement
framework as well as my critique. But of course,
they're free to disagree with my critique. And
hopefully we'll come on the podcast and respond
to what I'm going to tell you right now as I
go through the 29 outcome indicators. So let's
start with the first outcome. And this is the
Bhutan Education and Skills Training Project
delivered by Humber Polytechnic. So increased
opportunities for paid employment for technical
vocation. Graduates, especially women and people
with disabilities, that's the expected outcome.
They have three indicators to see if they've
actually achieved that increased opportunity
for paid employment. The first one is the percent
of targeted students who access a career service
and by field of study. Yes, they can record the
percent of students who actually accessed a service
who were women and or had a disability. If you
look at their PMF, their quote analysis of records
at the TVEN Institute needs to add analysis of
records using standardized criteria of what is
needed to get a yes, this student did access
a career service. Right now, we do not know if
the field officer responsible for this analysis
is in a conflict of interest employed by the
project. So that's a problem right now. If you
look at the PMF, there's somebody inside the
project being paid to tick off the box that says,
yep, you've got access to a career service. Ideally,
what they should be doing instead is bringing
in an external evaluator who by themselves independently
figures out, if I go into that institute, vocational
institute, training institute, what checklist
do they have to meet for me to say yeah they've
got a career service going and yes the students
actually accessed it so that would be better
so I don't give that one a plus for the indicator
I just think it's got a conflict of interest
and we don't know that it's quite simply that
the other thing I pointed out is instead of looking
at the records of the institute why not ask the
students directly did you get access To a career
service, yes or no? I think that would be a better
data source than the records of the institute,
which they currently have in the PMF. So that's
a no -no. The second indicator was percent of
industry TVET partnerships that are active. But
remember, the outcome is increased opportunities
for paid employment for TVET graduates. Paid
employment, not just activity, right? So here
I had a beef with that. What about the percent
of total partnerships that actually have paid
apprenticeships? What do they mean by active?
Remember, the outcome statement is increased
opportunities for paid employment. Current data
source is just reviewing the meeting minutes,
if you go to the PMF, between the TVET institution
and the company. That's not good enough, since
the expected outcome is paid employment for the
TVET graduate. So that's not very good. You don't
want to just look at the minutes. You want to
ask even the students directly, are you getting
a paid apprenticeship? That would be a better
measure, a better indicator for whether or not
they've actually increased opportunities for
paid employment for TVET graduates, right? So
that's a no -no. The third indicator is percent
of TVET institutions that have a career service
specific to women and people with disabilities.
So it's very similar to the first indicator,
but they're now being more precise with it's
to women and people with disabilities. Same problem
that I mentioned previously, right? They've got
to ask the women and the people with disabilities
directly. That would be better. So that's not
very good. So the next outcome is increased.
Whoops, excuse me. Yes. They have a career service
specific to women and people with disabilities
or they do not. So again, the same problem. There's
a bias. They got the field officer doing it instead
of an independent external evaluator. So what
I've already repeating myself here, paid employment
for graduates suggests a better indicator would
be the percent of graduates who got paid employment,
not the percent of TVN institutions that have
a career service for the target groups. Now,
you could argue they need to have the career
service first before they get the paid apprenticeship.
Yes, maybe, possibly. But that's not what the
outcome is saying. The outcome is clear. Increased
opportunities for paid employment. So cut to
the chase. Go straight to that. Find out how
many of the graduates, percent, actually got
paid employment or paid apprenticeships. That
would be more valid measure of the outcome. The
next outcome is improved capacity of TVET students,
especially women and people with disabilities,
to engage in entrepreneurial activities, including
in green business. And they have, let me see,
three indicators for that outcome. Let's look
at the first one. Percent of targeted students
who have developed a business plan. And I've
mentioned this in previous episodes with other
organizations. Same problem. Even if the percent
increases, the quality of the business plan could
be poor. Remember, the outcome is improved capacity
of students to engage. Well, okay, they develop
a business plan. You could argue that's sufficient
engagement, but really, no. They need to produce
a good one. That's why... They have this technical
vocational institute to train them on that, right?
So the PMF indicates no criteria on what is considered
a good business plan. Remember, the outcome is
improved capacity of these students to engage
in entrepreneurial activity. So they produce
a business plan. which is improved capacity compared
to no business plan at all. But those business
plans should be checked on meeting a minimum
level of quality. Second, there is again a conflict
of interest. As the M &E officer paid by the
project, not an external evaluator, is responsible
for getting this percent. Third, this percent
is only measured once per year and not until
year two, and not until year three. Too little.
too late if they want to show improved capacity
due to the project training rather than other
factors. The second indicator for improved capacity
of TVET students to engage is percent of graduates
who identify a high level of satisfaction with
the quality of entrepreneurship support programs
provided by TVET institutions. Again, if you're
asking the graduates, So satisfaction has got
nothing to do with improved capacity. Sure, they
might be satisfied, but we're not measuring that
in the outcome, right? We want to know if they
technically are able to produce, among other
things, a good business plan, right? So I would
take this out. Or improved capacity to engage.
When they say capacity, I see that as technical
capacity to produce a business plan. or whatever,
right? So I would take satisfaction out. Of course,
it's a good indicator telling people maybe there's
something wrong with the program, but it's not
related to the outcome. Third indicator is percent
of student business plans that focus on green
business. Okay, again, same problem as above.
PMF does not mention any criteria as to what
constitutes, quote, a good green business plan
of high quality. Now, perhaps this is assumed,
but we do not know. If it does exist, that should
be mentioned in the PMF. Again, not a valid indicator
due to conflict of interest being measured by
someone within the project. It should be an external
evaluator instead. And again, too little, too
late, and only measure it once per year and only
starting in year three of the project. Now, they
could argue they're not doing business plans
until year three of the project. Good. Fair enough.
One could argue that even an increase in the
percent of students producing low -quality green
business plans is better than no business plans
at all. I suggest for $4 .9 million in training,
these students on business plans, we should at
least measure and evaluate whether these green
business plans meet minimum quality standards.
So that's not a good indicator. You can't just
say, Focus on green business. We've got to see
if it's good, good quality, using some external
criteria of what constitutes a good green business
plan. Next outcome, increased effectiveness of
TVET partner institutions to deliver inclusive,
industry -relevant, environmentally responsive
skills training programs to students. First indicator
here for this outcome is percent of TVET instructors
who apply at least one named teaching technique
learned through project activities. It's a great
indicator, but the same problem. Got the wrong
person measuring it. The PMS shows that the project
M &E staff will be observing the TVET instructors
to see if they are applying these teaching techniques.
This is a conflict of interest. They mention
a survey in the PMF, but it is not clear if the
TVET instructors get the survey to complete themselves,
which would be unacceptable due to bias, or if
they actually surveyed the students to ask them
if their instructor delivered one of these teaching
techniques. And again, it is only done in year
four and year five. That sounds a bit late. for
this project. But maybe that's only those years
that the students receive this, quote, industry
-relevant, environmentally responsive skills
training, the students receiving it. So either
way, I wouldn't accept this indicator because
the data source is wrong. An external evaluator
should be observing these TVET instructors to
see if they are applying these teaching techniques,
preferably using a pre -developed checklist before
they enter the classroom to observe the TVET
instructor, right? Next indicator for this outcome
is percent of TVET instructors who receive positive
course evaluations. Now, we have to assume that
positive evaluation is equivalent to deliver
relevant, inclusive, environmentally responsible,
responsive skills training. The data source is
TVET institution records. I guess the course
evaluations completed by the students. The design
of these course evaluations for the students
to complete should be completed and delivered
to the students by an external evaluator, not
the Project M &E staff, which again is a conflict
of interest. So you'll want to make sure the
course evaluation form is specific to student
ratings of the instructor. specific to delivering,
quote, relevant, inclusive, environmentally responsive
skills training. So that should be put in the
PMF. It's not there. We don't know. Could just
be the M &E officer in the project ticking off
a box and saying, oh yeah, it's positive. We
don't know. So it should be more explicit. And
again, an external evaluator should be doing
that, not the project M &E officer. Because there's
a potential for bias. So just eliminate it, right?
Next outcome, strengthened capacity of targeted
TVET partner institutions to develop gender responsive
programs that meet economic and environmental
needs. So here, first indicator for that outcome,
percent of courses developed through that industry
partners confirm as relevant to emerging economic
sectors. From the indicator, it is clear that
it will be up to these industry partners. to
declare and confirm that each course indeed has
met some sort of criteria for them to conclude,
quote, that course is relevant to emerging economic
sectors. The problem is that the PMF shows the
data source as, quote, meeting records with industry
partners and the Project M &E staff, conflict
of interest, reviewing those meeting records
to determine if the course is indeed, quote,
relevant. This is not acceptable, right? for
those two reasons. So instead, an external evaluator
should survey the industry partners to get either
their prior approval before the course is delivered,
that it is relevant, they can review the course
syllabus. It says in the indicator, quote, developed,
not yet delivered. So that would seem to be the
right approach. They're developing these courses
to make sure they are gender responsive. Right?
So they need to survey the industry people, say,
is this relevant? Is it gender responsive? And
the external evaluator needs to come up with
a questionnaire and criteria so that they can
conclude after that that the industry partners
do indeed confirm that the course is relevant.
The second... indicator is percent of courses
developed that incorporate quote a gender -related
concept strategy or skill so again the data source
course curriculum and its review is good but
again we've got the same problem we mentioned
before conflict of interest right project M &E
staff are reviewing it which is a conflict of
interest also No mention of pre -established
criteria to be used when reviewing that course
content to conclude whether course content, quote,
incorporates that gender concept. So what would
be better is getting an external evaluator to
do the review as well as develop the criteria
to be used for that review. And another indicator
for this outcome, percent of courses developed
that incorporate an environmental. sustainability
concept, strategy, or skill. Same problem as
before. They need to get an external evaluator
in there to do the review as well as to develop
the criteria to be used for that review to lead
to the conclusion of whether course content incorporates,
quote, environmental sustainability concept,
right? So you can get the pattern here. There's
conflicts of interest going on with some of these
indicators, which is not acceptable. So next
outcome, increased effectiveness of national
TVET agencies and TVET institutions to manage
a reformed gender and disability inclusive and
environmentally responsive TVET system. So the
first indicator is level of confidence among
TVET officials in their ability to manage a reformed
TVET system that is gender sensitive, disability
inclusive. and environmentally responsible. So,
technical ability to manage an increased effectiveness
of an agency or institution cannot be measured
by TVET managers self -reporting their confidence
in their ability to manage. This self -reporting
is biased and fails to measure the technical
ability to manage. In addition, they collected
this self -reporting through focus groups and
surveys. A waste of money, right? And what could
they do instead? Well, way back before any of
you were born, including myself, way back in
the 1950s, the Educational Testing Service developed
the in -basket test. It's a test and it still
exists today, I'm guessing. It's an example of
where you can test the competency of managers,
where they have to prioritize tasks they're given.
defer it to somebody else, do it right away,
make somebody else do it, delegate. That's just
an example. But if you're going to train people
on how to manage, especially with this specific
content, gender and disability inclusiveness,
environmental responsibility, you need to design
a test that can test these managers if they're
competent to do that, rather than just asking
them. their confidence level, right? So that's
just an example. Now, after using that objective
measure, you find the percent of managers isn't
going up. Then, and only then, you could hold
a focus group and ask them, you guys in this
room, the percent has barely gone up. What's
going on? What's the problem with your ability
to manage? Why was there no percent increase
in those who passed? on the in -basket test.
That could be an example, right? So that indicator
is not very good. The next indicator for this
outcome was percent of revised courses that are
initiated within their designated timeline. So
initiating a revised course, perhaps to address
low confidence reported in their ability to manage,
I don't know, is still an output rather than
an expected outcome. This is not related to the
expected outcome of improving technical ability
to manage. Now, I suppose you could say it's
the step before, right? They've learned their
lesson. They know they're not managing. And so
they have to turn around and revise the course.
So that could be argued that, you know, if the
percent goes up and they do that, that's a measure,
again, of technical ability. But then somebody
could argue. Well, we've seen the revised course
and it's crap. But even if it's not crap and
it's really good, what is the outcome again in
this? It's to manage a system. It's a management
skill they're looking at here. And maybe that's
a measure of management is producing a high quality
revised course. So maybe again, they need an
external evaluator to come in and say. This is
what these managers have to reach if they're
going to revise these courses. And I haven't
checked the data source on this, but I'm assuming
it's not good. I think I'd have to check, but
I'm going to assume same problem. They don't
have an evaluator coming in to look at those
courses and say, yep, these have improved with
respect to the outcome. Therefore, the management
skill has improved, but we don't know at this
point. Another outcome, enhanced knowledge and
attitudes about TVET as a viable education and
employment pathway that is inclusive of women,
people with disabilities, and vulnerable populations.
So, first indicator for this outcome, increased
level of support among community members for
TVET education. If you look at the performance
measurement framework, a sample is taken from
the community members at large to see if the
percent supporting TVET as a viable pathway has
increased over time. This is great. It's a great
indicator. Problem is, there's no comparison
or control groups, so they cannot support their
claim that their project was responsible for
any observed increase in the percent. In addition,
there is no mention of a one -tailed, right -tailed
hypothesis test using the sampling Z distribution
of a simple single proportion. So any observed
increase in the percent over time could fail
to be statistically significant with a p -value
less than 0 .05, right? So either way, they've
got a problem. At the very minimum, they should
be doing what any second year undergraduate social
science major will tell you. They should be comparing
the percent sample of the population that they've
just done the community members sampling to see
if the percent has gone up enough to reach a
critical value where they can reject the null
and go with the alternate and say yeah the percent
has gone up and it's been statistically significant
and we're at least 95 percent confident that
there is that observed increase is statistically
significant otherwise why do the project right
If you don't do the project and the percent still
goes up, why spend almost $5 million? You got
to show it's statistically significant. It's
effective. Or some people say, not only effective,
it has impact, right? So that's a problem. Next
indicator, percent of women and people with disabilities
from sample communities who view TVET as a viable
education pathway. Now, From the PMF, a sample
is taken from women and people with disabilities
to see if the percent supporting TVET as a viable
pathway has increased over time. Again, great.
But again, no comparison groups, no control groups.
They still need to do a statistical test to see
if that percent over time has statistically significantly
increased. Otherwise, there's no point. They
got to prove that the project has made a statistically
significant difference. Right? And ideally, you
want to have a comparison group or a control
group. Now, they could be delivering this project
to every single TVAN institution in the country.
And maybe they are. That's fine. Okay, so no
point in having a control group. But they still
have to show that the percentages they've got
have statistically increased significantly. Otherwise,
they should shut the program down. Another indicator
is the percent of TVET institutions that implement
new recruitment and counseling strategies and
systems. Now, again, the outcome there is, sorry,
I went to, there's a new outcome. So let me repeat
that. There's a new outcome here for that indicator.
It is strengthened capacity of TVET institutions
to recruit and counsel students, especially youth,
women, and people with disabilities. So the first
indicator for that outcome is percent of TVET
institutions that implement new recruitment and
counseling strategies and systems. Again, great
indicator, but wrong people to see if this percent
has increased. The PMF shows that individuals
representing these TVET institutions will be
asked, did you implement new strategies to recruit
youth, women, people with disabilities? Yes or
no? This is done through a survey and interviews.
Of course, they will say yes due to self -reporting
social desirability bias. So they shouldn't be
involved in this. Again, if you look at the PMF,
it's the institutions, excuse me, serving. Of
course, they will say yes due to self -reporting.
So I have to look again at the PMF, but it's
not mentioning as the data collector, person
responsible is being. a external evaluator, which
is what you should be doing instead. Going into
the TVET institution using a predetermined checklist
that the institution actually has met the criteria
of, yes, this TVET has indeed implemented a recruitment
system. And even if they did, does the institution
have a system in place to counsel these three
groups? So external evaluator is what you need
to do instead. The second indicator for this
outcome is level of confidence among counseling
staff at TVAN institutions in their ability to
effectively counsel students, including women,
youth, and people with disabilities. Again, as
you know, you can't do that, especially when
you explicitly state ability to effectively counsel.
I mean, wow. You don't ask them how they're feeling.
You do an external evaluation. of their ability
to counsel students. There's a whole pile of
psychometric tools out there to measure. I mean,
just go to any university in Canada that has
a master's program in counseling psychology.
I'm quite sure there's a whole bunch of external
measures, observation of their counseling with
the student, rather than asking the counselor
themselves to report on their self -confidence,
right? So get an external evaluator, not a good
indicator. And the last one, same problem. Level
of confidence among recruitment staff at TVET
institutions in their ability to effectively
recruit. So that's the other skill. Counseling,
now they're looking at recruitment skills. Again,
same problem. Self -reporting confidence levels
is not a valid measure of whether recruitment
staff have the ability to effectively recruit
students. So that's not good. Next. Expected
outcome, enhanced access to economic opportunities
in the labour market for TVET graduates, especially
women and people with disabilities in high opportunity
sectors. First indicator, percent of graduates
who complete entrepreneurial training, who start
their own business. Great indicator, great measurement
through tracking. of graduates if you see their
pmf it's called a tracers study but all that
money to track these graduates just to ask hey
dude have you got a business yes or no wouldn't
you want to ask hey are you making any money
show me the money so a better indicator would
be the mean monthly income earned from that business
and track it over time to see if it has increased
also no need to have a focus group as they talk
about in the performance measurement framework,
at least not at this point, until you first use
this income indicator. Then if income has not
increased, you could hold a focus group and ask
them, what's going on, dudes? You've started
a business. It's great, but you're not making
any money from it. So what do we need to do to
identify what are the problems as to why? you're
not making any money. Is it because they didn't
get the labor market analysis right in the first
place? There's thousands of seamstresses already
out there in the market that we can't compete
with, even though I went to this institute and
learned how to sew and graduate. I can't get
any income. Maybe the United States or Canada
are dumping used clothing into the market, which
I can't compete against. Whatever, right? So
a better valid indicator of enhanced access would
not be just starting up a business, but more
money, right? And the next indicator, level of
satisfaction among graduates with their economic
opportunities. Again, they might not be satisfied.
It tells you, yeah, there's something wrong.
But what you really want to do is focus on the
income, right? And of course, we're assuming.
They've acquired the necessary skill levels.
I can't see anywhere where there's a certification
testing program in the institute to make sure
they've met a minimum standard to say that they're
a carpenter or a plumber or a mason or a seamstress.
Right? Or tailor, right? So if we track the income,
we don't need the satisfaction levels. And remember,
not other income sources, right? Got to make
sure that they're actually making money from
the training that this $5 million project gave
them in a particular technical vocation, right?
Got to be very clear about that. It's a big problem
in Africa where they'll have multiple sources
of income and then you just ask them and they
give you a summary number and you think, wait
a sec, didn't we train you how to be a welder?
How much money do you make from welding? Nothing.
Okay. And you'll notice the PMF does indicate
that the project training will align with, quote,
labor market needs. So satisfaction should be
related to income rather than the frustration
of not getting demand for their services after
graduation, right? So the next indicator is percent
of industry partners that hire a TVET graduate
in the last year by field of study. and economic
sector. Again, great indicator, but even better
would be income levels. Because they could be
hired, but paid a really bad wage, right? Or
nothing at all. They could be an unpaid intern.
We don't know what they mean by hired. So cut
to the chase, find out the incomes. And we need
to show that the income levels of the graduates
are higher than those who did not get trained
in technical vocation. Perhaps people that taught
themselves to be a carpenter. Plumber or seamstress,
right? Or tailor. Next outcome is strengthened
effectiveness of TVET agencies and institutions
to provide inclusive and environmentally responsive
TVET education to trainees, especially women
in high opportunity sectors. First indicator,
percent of students who enter a TVET institution
who graduate. Now remember, it's strengthened
effectiveness. of TVET agencies and institutions
to provide inclusive and environmentally responsive
TVET education to trainees. It's very precise,
but the first indicator has got nothing to do
with that. It's just saying whether or not they
graduate or not. We don't want to know that.
We want to know they not only graduated, but
they are familiar with inclusive and environmentally
responsive education, particularly women, right?
And also, if you read the outcome, it's not about
the student. It's about the institution being
inclusive and environmentally responsive in their
education to the students. Right. So I would
take it out. Who cares what the percent is of
students who graduate? It's completely unrelated
to the outcome. That's gone. Now, the next one
indicator for this is. level of satisfaction
among TVET graduates with the quality of their
TVET education. That's a little closer. It's
a good indicator since it asks the student if
they are satisfied that the institutions indeed
are delivering training that is, quote, inclusive
and environmentally responsive. So you have to
make sure that that satisfaction survey questionnaire
that goes to the students contains questions
specific. to that expected outcome regarding
what we just said inclusive and environmentally
responsive the only issue here is that this tracer
study is delivered again by the project m e staff
rather than an external evaluator ideally you
want an external evaluator to be hired to review
the course content develop questions for the
students that measure satisfaction levels specific
to being, quote, inclusive and environmentally
responsive. But on the whole, it's a good indicator.
So I've skipped the other indicator because I
don't think it's relevant. So just one. So here's
a new outcome. Increased participation of learners,
particularly among youth, women, and learners
in vulnerable situations in TVET education in
high -opportunity sectors. First outcome indicator
for this outcome is number of enrolled students
annually in public TVET institutions. So here
the only issue is whether this project is necessary
to achieve increased enrollment in TVET institutions
and increased percent within this enrollment
of those who are youth, women, rural, or people
with disabilities. What about another region?
within the country where this project is not
operating, and that regional government has a
policy in place with the same expected outcomes.
Now, as I mentioned earlier, if this project
covers the entire country, then this is not important.
A review of the project browser indicates a countrywide
project reaching all TVET institutions. That's
what I'm assuming. So that's a good indicator.
It's a good indicator. You want to... you want
to get those enrollments up, right? And they've
done some community announcements to try to get
that enrollment up. Now, the next one is percent
of graduates who are employed in their field
within six months of graduation. Again, percent
of graduates employed in their field is good,
but an even better indicator would be their actual
income from that employment. But I'm still accepting
this indicator as a good one. And the disaggregation
is really good. because you want to see those
percents for women, people with disabilities,
since that's a focus of the project. But again,
there is no control or comparison group. So if
those percents go up, they need to show that
they're going up statistically significantly,
right? So they need to take a sample of those
graduates at different points in time, find out
what percent of the sample actually say yeah.
I got employed within six months of graduation.
It's a simple yes. And compare that over time
using a one -tailed, right -tailed alternative
hypothesis with a P value less than 0 .05, right?
The second indicator here is level of satisfaction
within the industry of the ability of TVET graduates
to fill labor market needs. in an environmentally
responsive manner. We are assuming that the industry
people who are interviewed support environmentally
responsive manner. If the assumption is correct,
the survey used here in the PMF would be acceptable,
and this indicator complements the claim. The
graduates are indeed being environmentally responsive.
Again, an external evaluator, not the project
staff, would be better for designing and delivering.
the survey questionnaire. But I still went ahead
and said this is a good indicator, despite that.
The other indicator is, finally, level of community
support for TVET as a means to promote women's
equality. Because, you know, some people may
dispute whether women should be involved in technical
locations, right? And again, the indicator here
is a percent surveyed from the PMF. And the percent
could increase, but not enough to support effectiveness
at P less than 0 .05. So there's no control group,
no comparison group. So at the very least, they've
got to do a proper sampling, two different points
in time, and do a proper statistical test. And
any undergraduate student majoring in the social
sciences will learn the importance of inferential
statistics and showing that any increase in the
percent needs to be statistically significant
to make the claim that it has increased not just
due to chance, right? So my summary from all
of this, the 29 outcome indicators for the 11
expected outcomes, is that only four of the 29
outcome indicators are valid measures of the
project's 11 expected outcomes. So only two out
of the 11, project outcomes could Humber Polytechnic
claim that they've achieved their outcomes. So
thank you for listening. That's the end of my
review. And like I said, if you want to get the
PMF and my critique in writing, you can just
email me at evaluatecanadaid at gmail .com and
I'll be happy to send it to you. Feel free to
make any comments in the Spotify platform. where
you now can make comments on any of the episodes.
And stay tuned. My next episode is going to deal
with an international organization that has finally
decided to publish its PMF on its website, along
with the data for their percent indicators. Unfortunately,
in Canada, as you can see, we don't have any
PMFs on any of these. organizations like Humber
Polytechnic that are funded by the government
of Canada. It should be a requirement that their
PMF is on their website along with the data that
goes with it. If you go to the project browser,
they have data there, but it's absolutely useless
data because it's not related to their outcomes
or their outcome indicators. This is why I have
to go and ask for the PMF through an access to
information request. Thank you for listening.
Stay tuned for the next episode. Bye for now.