Welcome to the Improving Development Evaluation
Podcast. I'm your host, David Wand. And in this
episode, we're going to follow up on my previous
episode and look at the Global Community Engagement
Resilience Fund. And you can learn more about
GSERF, G -C -E -R -F, at gserf .org. And we're
going to look at the Western Balkans, which they
have on their website, a results... report entitled
Results from Round 1 and Round 2 Funding in the
Western Balkans 2020 to 2025. And we're going
to focus again, like my podcast has been doing
since 2023, identifying two major weaknesses
in evaluation design. The first one is failure
to do any statistical analysis of their indicators.
or doing statistical analysis that is just simply
incorrect. I'll be looking at that in this episode,
specific to the Western Balkans report that they've
put on their website, which I commend them for.
This is great to see. But more importantly, I'm
going to focus on the second area, which is the
failure to use, as you've noticed in some of
my episodes, where the outcome is very well clearly
articulated. But the outcome measure that they're
using or outcome indicator to measure that outcome
is not a valid measure of the outcome. And I'm
going to focus particularly in this episode on
their choice of indicators, outcome indicators
to measure, quote, the rehabilitation of returning
foreign terrorist fighters, looking at that indicator,
explaining to you how it is flawed, and more
importantly, practicing what I preach, and that
is providing an alternative valid measure of
rehabilitation, of returning foreign terrorist
fighters, so that I can't just stand here and
keep whining and complaining that organizations
are using the wrong outcome indicators. I've
got to go a step further, put up or shut up,
actually provide some real valid measures of
rehabilitation of returning foreign terrorist
fighters so that they can proceed in that direction.
Because that's the whole purpose of my podcast
is to get these organizations to improve or for
that matter, just even start evaluating their
projects in a way where they can actually claim
with valid data. analysis that they actually
are achieving the outcomes that they keep claiming
on their website that they are achieving but
they're not currently at least and that distinguishes
them from just a charity because we're not disputing
the fact that all these wonderful people around
the world need all these wonderful services that
we all know are being denied to them because
of a variety of reasons that's a separate issue
so you could be a charity go ahead, deliver all
the services you want and go home. Just call
yourself a charity. That's what a lot of charities
do. But when you get into this next level where
you're claiming that if you give us money, we're
not only going to be a charity, we're going to
achieve these expected outcomes, then we get
into the whole realm of intellectual dishonesty.
So put up or shut up, right? So that's what I'm
going to focus on today. And before I do... As
I've always been saying, if you live in the Global
South or if you work at an international development
organization where hopefully you are required
to go to the Global South and monitor and evaluate
these projects, then you need some useful tools,
not just psychometric tools that I'll be talking
about, but you also need three things that I
highly recommend in the Global South. Sunglasses.
Proper clothing and proper travel gear because
you're going to be in the field interviewing
people, collecting data, maybe doing some focus
groups, whatever. You need sunglasses. So you
want to check out akila .la. That's A -K -I -L
-A dot L -A. They also do prescription lenses,
not just regular sunglasses, but prescription
sunglasses, all made from... 50 % recycled material.
You get a 10 % discount if you use the link that
I'll provide in the episode notes. And if you
do that and purchase, I get a 15 % commission
when you purchase any product using that promo
code. And so you have your sunglasses to protect
you from the sun in the global south, but you
need the right clothes to be comfortable in that
hot and humid climate. And so you want to go.
for your clothes to hypernaturalstyle .com. That's
hypernaturalstyle .com where you can get some
men's tees and button -ups. If you purchase from
that link in my episode notes, you get 20 % off
any product when you use my link in the episode
notes and I get a 20 % commission off. from the
total purchase you make when you use my link.
And you might be wondering, so what is so cool
about these hypernatural products? They have
a patent pending hypercooled jade technology
that when it's hot, lowers your body temperature
by three to five degrees. They're also breathable,
thanks to Supina cotton, and they are sold in
premium retailers like Nordstrom. And we're rated
the best. men's polo shirts in 2024 in Men's
Journal, Forbes, and Esquire. And the last product
I've recommended, which is new, that I've discovered,
which is good, is Torain, T -O -R -A -I -N .org.
That's Torain .org. They sell backpacks, totes,
wallets, and bags. And what's good about them
in the global south is they're durable, they're
lightweight, they're water resistant, in case
you're going during the rainy season perfect
for travel and outdoor use if you are in the
global south particularly in the field monitoring
and evaluating a project all terrain products
are crafted from upcycled feed and rice bags
sourced globally each bag's interior is lined
with recycled plastic bottles handcrafted by
artisans supporting fair trade By purchasing
a Touraine bag, customers contribute to global
sustainability. You get 15 % off any product
if you use my link that is in the episode notes.
The link is mydeals .page forward slash 1HJL.
When you do that, I get a 15 % commission of
the total cart purchase. So just to refresh your
memory. The Global Community Engagement Resilience
Fund on their website is responsible, even though
it's a fund, they're still responsible for the
proper evaluation design of the projects that
they fund, or as they say, grant. So at the grant
level, this is quote from their website, at the
grant level, each project is guided by a theory
of change and a results framework with outcome.
and output indicators requiring grantees to collect
baseline data, report quarterly, and conduct
final assessments. So it's clear that GSERF is
responsible for making sure that the design of
the evaluation is proper. That means valid outcome
indicators. That means proper statistical analysis
to support their continued claims that they're
achieving their outcomes. On their website, they
clearly state four outcomes we've talked about
before, but they also have done a new expected
outcome that they claim they're achieving, and
that is the rehabilitation of returning foreign
terrorist fighters. So we're going to take a
closer look at their results from round one and
round two funding in the Western Balkans. Report
2020 to 2025. Again, They say it's an abridged
version of the original paper, but I can't find
the original paper. So to be fair to them, maybe
the entire paper includes all of the outcome
indicators. But what I've read so far is there's
only one outcome indicator from this report on
actual measurement of rehabilitation of returning
foreign terrorist fighters. So we're only going
to look at that one indicator. So we're going
to start with just reading. from page one, paragraph
two, or sorry, page two of that report, paragraph
two. And I quote, GSURF has worked in the region
since 2017. It started by supporting preventing
violent extremism work in some foreign terrorist
fighter communities of origin. And then later,
reintegration and rehabilitation of returnees
in Kosovo. In 2020, with the learnings of its
experience in Kosovo, G -Surf approved its regional
strategy focused on rehabilitation and reintegration
covering Albania, Kosovo, and North Macedonia,
later expanding to Bosnia and Herzegovina. So
we've got one, two, three, four countries in
the report. I see them. Albania, Bosnia and Herzegovina,
Kosovo, and North. where they're involved in
funding organizations that are delivering, among
other things, I presume, psychological services.
Those are your outputs to achieve the expected
outcome of, quote, rehabilitation of returning
foreign terrorist fighters. So there's a background
for you. So if you go to page three of this report,
figure three, it says, quote, progress on regional
strategy indicators that represent key dimensions
of its rehabilitation and reintegration strategy.
And this is one of the problems you get into
when the people at GSERF who are responsible
for performance and impact have got the wrong
university degrees. They've all got degrees,
undergraduate, even undergraduate, bachelor's,
master's, and PhD in the same international affairs
subject area. So you obviously know right away
that they're going to be obsessed with national
level aggregation and even international combining
across countries indicators. But that's not what
you want here. What you want to see is for an
individual project, which they, being international
affairs degree people, are obsessed, will say,
ooh, that's too granular. That's too granular.
But that's how you're going to properly evaluate
whether these, quote, returning foreign terrorist
fighters have actually been, quote, rehabilitated.
You can't combine this indicator on reintegration
across several grants and then across several
countries. Because even if you do, and even assuming
your indicator is valid, where are you going
to locate which grant is doing well? Which one
isn't? You have to go back. Why do evaluation
if you can't find out which grant is screwing
up, which one's being effective? So that's a
major problem. But let's get into the specifics.
First of all, the weakness in the statistical
analysis. So they have one indicator on page
four that's related to, quote, reintegration
of returning foreign terrorist fighters. foreign
terrorist fighters and their families who report
participation in social activities with members
of receiving community so they start with a baseline
percent that actually report participation who
actually report participation at 70 percent At
the end of the program on page four, they claim
it's gone up to 93%. So they're claiming that
there is a increase from 70 to 93%. The problem
is they're combining it across several grants
in several countries. So we don't know which
grant in which country has actually gone up on
this indicator and by how much. or gone down.
The other problem is they're using an average
of this percent. They should use the median because
the median, as you've learned from your second
year undergraduate stats course, the median is
less sensitive to central tendency. Sorry, less
sensitive to outlier, large values or small values,
right? It's better to use the median and then
maybe compare the median percent at baseline
and see if it's gone up to... statistically significant
up to 93%. So that's the first problem they have
with their statistical analysis. The second problem
is they've combined, not only averages is wrong,
they've combined all the grants across all the
projects within a country and across all the
countries that I listed, which is what? How many
countries we got? One, two, three, four. Bosnia
and Herzegovina, Kosovo, Albania, North Macedonia.
The other problem is I've mentioned in the other
episode is this percent. We don't know if it's
a sample percent or a population percent. It
could be a population percent because when you
read in the report, they only talk about on page
five, 127 returnees. So they could easily without
expense measure all 127, which gets to my more
important point. If there's only 127, instead
of... using a biased, poor indicator of the percent
who, quote, report participation in social activities
and their families, you're going to have a self
-reporting bias. It doesn't objectively measure
whether or not, quote, they've been reintegrated.
It doesn't get to the core. So that's the major
problem. The indicator is not a valid measure,
and it's biased also. due to self -reporting
bias, in claiming that they're actually been
successful at reintegration over time, right?
So what do you do instead? What is a valid indicator
of reintegration of returning foreign terrorist
fighters? Assuming that they're getting, and
they talk about it in the report, but I don't
want to focus. on these outputs, but I presume
they're going to be delivering a variety of different
psychological services. It is good that they've
recognized the outcome indicator is the same
across all countries, even though there's cross
-cultural differences, but they've just chosen
the wrong indicator because it's not a valid
measure. It's not a standardized, objective,
global indicator of measuring rehabilitation.
reintegration of returning foreign terrorist
fighters it's not but guess what there are a
few valid psychometric tools out there and this
is why that background education background in
international affairs unfortunately gives them
a bias where they're not interested in looking
at psychology which is what this whole funding
is about psychological services to get these
former returning foreign terrorist fighters to
rehabilitate right complex so you need to get
a proper psychometric tool that looks at it so
you might be wondering what is there out there
so this is what i did i just entered into co
-pilot the following question what is a valid
psychometric tool to measure a reduction in an
individual's predicted threat level over time
where this individual was a former terrorist
fighter. Up comes from co -pilot, quote, the
most widely accepted valid psychometric and risk
assessment tool specifically designed to measure
changes in an individual's terrorist threat level
over time is the, get ready for it, violent extremism
risk assessment. V -E -R -A, Vera, version two
revised. Now I thought, this is co -pilot. Sometimes
you get wrong answers. So then I entered that
into just a Google search and up came a website.
www .vera -2r .nl backslash instrument. And lo
and behold, it's been developed by the Dutch.
But let me just give you a little bit more detail.
Quote, unlike standard criminal risk tools, the
VERA2R operates on a structured professional
judgment framework. This methodology allows psychologists,
intelligence professionals, and correctional
staff to reevaluate a former fighter at regular
intervals to observe whether dynamic risk factors
are decreasing and protective factors are increasing.
While several tools exist, two primary instruments
are internationally recognized for monitoring
a former fighter's rehabilitation and trajectory
over time. It is an objective, evidence -based
tool optimized for individuals who have been
incarcerated or detained for ideologically motivated
violence. By the way, if you're listening to
this on Spotify, feel free to make comments.
If my critique is flawed in any way, if I'm using
the wrong psychometric tools, let everybody know
what we should use because the taxpayer is paying
for all of this, right? Measuring change. The
tool specifically tracks 34 distinct indicators
across five domains because many of these factors
are dynamic, changeable. A practitioner can run
the protocol every six to 12 months to chart
an individual's progress or regression. So this
is what GSERF should be using. The beauty of
GSERF being a fund, they now have a tool here.
All they have to do is find an organization that
will be subjected to the training and use of
the tool. Then they can just report back to GSERF
on the indicator, right? So there's another one,
ERG22 +, ERG, Extremism Risk Guidelines, developed
in the United Kingdom. The extremism risk guidance
22 plus is used heavily by forensic psychologists
and probation officers to evaluate convicted
terrorists. And you can Google that and you'll
find there's a publication coming up on researchgate
.net. The application over time as a former fighter
undergoes rehabilitation programming. The ERG
22 plus maps whether their intent to commit has
deflated. even if they still hold a lingering
portion of their original ideological engagement.
Just to give you a little bit more, psychometric
tools in this domain do not spit out a numerical
score like an IQ test. Doing so is considered
highly dangerous by security experts due to the
low base rate of terrorist events. Instead, they
measure threat reduction by looking for specific
behavioral changes. Two more. Depending on the
specific context of the former fighter, two other
frameworks are occasionally paired with the VERA
-2R. One, the TRAP -18, Terrorist Radicalization
Assessment Protocol, TRAP, a tool consisting
of eight proximal warning behaviors and ten distal
characteristics. It is highly useful if security
agencies need to determine the imminence of a
threat, or if a former fighter is actively planning
a relapse into operational activities. You can
actually buy the TRAP -18 online. Another psychometric
tool, the RAN. That's the European Radicalization
Awareness Network. It's developed by the European
Radicalization Awareness Network. This tool is
optimized for foreign terrorist fighters who
have returned from... conflict zones. Sounds
pretty relevant to what GSURF talks about, returning
foreign terrorist fighters. They should check
this one out. This tool is optimized, blah, blah,
blah, to measure their specific reintegration
risks. So that even fits with the second part
that GSURF is claiming they're achieving, which
is reintegration after rehabilitation. Now, there
is an indicator on reintegration, but I don't
want to talk about it right now. And there's
even a paper that reviews all the psychometric
tools. But you can find that out online. So what
about buying it? You cannot buy the Vera 2R manual
or its psychometric protocols. So what you have
to do is it's officially offered by the Netherlands
Institute of Forensic Psychiatry and Psychology.
And what they will do is they'll provide the
certified training. curriculum development and
user registries across Europe and works with
licensed global trainers internationally. There's
even regions outside of Europe that also are
periodically licensed to run authorized cohorts.
So if you contact them, you can learn more about
the training. So again, GSERV, not being an NGO
that delivers the services directly, they can
get an organization. in their request for proposals
that's willing to do that. And they could just
fund them. And they'll go off and train these
people using this valid outcome indicator on
the reintegration of returning foreign terrorist
fighters. And then all they have to do is come
back to GSURF and report on this indicator as
part of the evaluation framework that GSURF,
that is, they're responsible for developing.
So included in the purchase is the course curriculum,
the official manual upon successful graduation,
passing the qualification parameters. You're
handed the physical VERA 2R professional manual.
And then you're also, this practitioner is added
to an elite monitored international register,
which is great because if you can get local citizens
in these countries. Kosovo, et cetera, to become
practitioners, you could even argue that's one
of the other outcome indicators that GSERF keeps
claiming they're achieving, which is vocational
skills training. So they've also achieved that
and also additional income, increased incomes.
So that's a great example where if they use this
psychometric tool, they can achieve two outcomes
at once. They can achieve an increase in vocational
skills, followed by an increase in subsequent
income. And hopefully if they're using that indicator,
as I recommend they do, and also using these
valid measures of rehabilitation of returning
foreign terrorist fighters. So that gives you
an example of how their choice of outcome indicators
are not valid, and therefore they can't claim.
like they are claiming that they're being successful
at rehabilitation when they aren't, for the reasons
I provided. They're not using a valid measure.
I have provided, what is it, four valid measures,
psychometrically, of rehabilitation of returning
foreign terrorist fighters. To be clear here,
the indicator that they have provided in this
report that I've reviewed, that as invalid, is
actually a measure of reintegration because they're
asking the self -reporting bias of the returning
foreign terrorist fighters to report on their
participation. along with their families, which
also is self -reporting bias, as to whether they
have participated in social activities with members
of the receiving community. That clearly suggests
this is an indicator of reintegration, which
further stresses the point that in the report
that GSURF has produced, there's not a single
indicator they have shown in that report that
measures validly. rehabilitation. And so the
four measures that I've offered, they should
be looking at those. That's very important because
unless I've missed something, those indicators,
there's not a single one that deals with measuring
validly rehabilitation. Now it might be in the
larger report. We don't know, but you'd think
they would have it in at least the shortened
report that they've posted on their website because
they're claiming that they have been successful
in rehabilitating these fighters and obviously
that's a precursor you first have to rehabilitate
them before you reintegrate them right so that
even stresses the point even further where are
your valid indicators of rehabilitation because
they're not there so g -serve has four that i've
suggested in this episode that they should draw
on so what i'm going to do now is since g -serve
has received money from the government of canada
in the past and they are listed on the Global
Affairs Canada project browser website. I will
be, as usual, sending an email to the Secretary
of State for International Development, along
with copies of that email. They'll be copied
on it. All the opposition party critics, shout
-out critics on international development, recommending
that they use this indicator for any organization,
including GSURF, that they're funding to deliver
services. that the Canadian taxpayer is paying
for to achieve this expected outcome of rehabilitation,
of returning foreign terrorist fighters. Also,
recommending that they follow the lead of GSURF
in actually putting the performance measurement
frameworks, that's what they're calling Canada,
on the website. of the organization that's received
the money from the Government of Canada. And
even better, that they put them on the Government
of Canada website, which they are currently not
doing. So, thank you for listening. Bye for now.