WEBVTT 00:00:00.000 --> 00:00:02.259 Welcome to the Improving Development Evaluation 00:00:02.259 --> 00:00:05.700 Podcast. I'm your host, David Wand. And in this 00:00:05.700 --> 00:00:08.519 episode, we're going to follow up on my previous 00:00:08.519 --> 00:00:12.160 episode and look at the Global Community Engagement 00:00:12.160 --> 00:00:15.820 Resilience Fund. And you can learn more about 00:00:15.820 --> 00:00:20.359 GSERF, G -C -E -R -F, at gserf .org. And we're 00:00:20.359 --> 00:00:24.719 going to look at the Western Balkans, which they 00:00:24.719 --> 00:00:30.329 have on their website, a results... report entitled 00:00:30.329 --> 00:00:34.369 Results from Round 1 and Round 2 Funding in the 00:00:34.369 --> 00:00:39.009 Western Balkans 2020 to 2025. And we're going 00:00:39.009 --> 00:00:42.229 to focus again, like my podcast has been doing 00:00:42.229 --> 00:00:47.250 since 2023, identifying two major weaknesses 00:00:47.250 --> 00:00:51.009 in evaluation design. The first one is failure 00:00:51.009 --> 00:00:54.570 to do any statistical analysis of their indicators. 00:00:55.560 --> 00:00:58.579 or doing statistical analysis that is just simply 00:00:58.579 --> 00:01:01.259 incorrect. I'll be looking at that in this episode, 00:01:01.460 --> 00:01:05.000 specific to the Western Balkans report that they've 00:01:05.000 --> 00:01:07.060 put on their website, which I commend them for. 00:01:07.560 --> 00:01:10.959 This is great to see. But more importantly, I'm 00:01:10.959 --> 00:01:15.299 going to focus on the second area, which is the 00:01:15.299 --> 00:01:17.480 failure to use, as you've noticed in some of 00:01:17.480 --> 00:01:21.780 my episodes, where the outcome is very well clearly 00:01:21.780 --> 00:01:25.689 articulated. But the outcome measure that they're 00:01:25.689 --> 00:01:28.530 using or outcome indicator to measure that outcome 00:01:28.530 --> 00:01:32.150 is not a valid measure of the outcome. And I'm 00:01:32.150 --> 00:01:35.629 going to focus particularly in this episode on 00:01:35.629 --> 00:01:39.129 their choice of indicators, outcome indicators 00:01:39.129 --> 00:01:45.090 to measure, quote, the rehabilitation of returning 00:01:45.090 --> 00:01:48.510 foreign terrorist fighters, looking at that indicator, 00:01:48.769 --> 00:01:51.950 explaining to you how it is flawed, and more 00:01:51.950 --> 00:01:54.739 importantly, practicing what I preach, and that 00:01:54.739 --> 00:01:59.859 is providing an alternative valid measure of 00:01:59.859 --> 00:02:04.700 rehabilitation, of returning foreign terrorist 00:02:04.700 --> 00:02:07.760 fighters, so that I can't just stand here and 00:02:07.760 --> 00:02:11.500 keep whining and complaining that organizations 00:02:11.500 --> 00:02:14.379 are using the wrong outcome indicators. I've 00:02:14.379 --> 00:02:16.919 got to go a step further, put up or shut up, 00:02:17.020 --> 00:02:22.180 actually provide some real valid measures of 00:02:23.449 --> 00:02:27.810 rehabilitation of returning foreign terrorist 00:02:27.810 --> 00:02:31.449 fighters so that they can proceed in that direction. 00:02:31.590 --> 00:02:34.909 Because that's the whole purpose of my podcast 00:02:34.909 --> 00:02:40.250 is to get these organizations to improve or for 00:02:40.250 --> 00:02:43.789 that matter, just even start evaluating their 00:02:43.789 --> 00:02:46.449 projects in a way where they can actually claim 00:02:46.449 --> 00:02:53.979 with valid data. analysis that they actually 00:02:53.979 --> 00:02:57.180 are achieving the outcomes that they keep claiming 00:02:57.180 --> 00:02:59.539 on their website that they are achieving but 00:02:59.539 --> 00:03:04.319 they're not currently at least and that distinguishes 00:03:04.319 --> 00:03:07.379 them from just a charity because we're not disputing 00:03:07.379 --> 00:03:09.719 the fact that all these wonderful people around 00:03:09.719 --> 00:03:13.460 the world need all these wonderful services that 00:03:13.460 --> 00:03:15.699 we all know are being denied to them because 00:03:15.699 --> 00:03:18.800 of a variety of reasons that's a separate issue 00:03:18.800 --> 00:03:22.919 so you could be a charity go ahead, deliver all 00:03:22.919 --> 00:03:25.159 the services you want and go home. Just call 00:03:25.159 --> 00:03:27.560 yourself a charity. That's what a lot of charities 00:03:27.560 --> 00:03:30.360 do. But when you get into this next level where 00:03:30.360 --> 00:03:33.219 you're claiming that if you give us money, we're 00:03:33.219 --> 00:03:35.300 not only going to be a charity, we're going to 00:03:35.300 --> 00:03:38.080 achieve these expected outcomes, then we get 00:03:38.080 --> 00:03:41.080 into the whole realm of intellectual dishonesty. 00:03:41.280 --> 00:03:45.020 So put up or shut up, right? So that's what I'm 00:03:45.020 --> 00:03:49.889 going to focus on today. And before I do... As 00:03:49.889 --> 00:03:52.810 I've always been saying, if you live in the Global 00:03:52.810 --> 00:03:55.750 South or if you work at an international development 00:03:55.750 --> 00:03:59.030 organization where hopefully you are required 00:03:59.030 --> 00:04:03.310 to go to the Global South and monitor and evaluate 00:04:03.310 --> 00:04:08.430 these projects, then you need some useful tools, 00:04:08.590 --> 00:04:11.909 not just psychometric tools that I'll be talking 00:04:11.909 --> 00:04:14.909 about, but you also need three things that I 00:04:14.909 --> 00:04:17.350 highly recommend in the Global South. Sunglasses. 00:04:18.189 --> 00:04:21.769 Proper clothing and proper travel gear because 00:04:21.769 --> 00:04:24.610 you're going to be in the field interviewing 00:04:24.610 --> 00:04:29.310 people, collecting data, maybe doing some focus 00:04:29.310 --> 00:04:33.730 groups, whatever. You need sunglasses. So you 00:04:33.730 --> 00:04:36.990 want to check out akila .la. That's A -K -I -L 00:04:36.990 --> 00:04:40.790 -A dot L -A. They also do prescription lenses, 00:04:40.889 --> 00:04:43.689 not just regular sunglasses, but prescription 00:04:43.689 --> 00:04:47.769 sunglasses, all made from... 50 % recycled material. 00:04:48.449 --> 00:04:53.149 You get a 10 % discount if you use the link that 00:04:53.149 --> 00:04:55.910 I'll provide in the episode notes. And if you 00:04:55.910 --> 00:04:58.610 do that and purchase, I get a 15 % commission 00:04:58.610 --> 00:05:02.990 when you purchase any product using that promo 00:05:02.990 --> 00:05:06.329 code. And so you have your sunglasses to protect 00:05:06.329 --> 00:05:08.769 you from the sun in the global south, but you 00:05:08.769 --> 00:05:10.930 need the right clothes to be comfortable in that 00:05:10.930 --> 00:05:15.550 hot and humid climate. And so you want to go. 00:05:15.959 --> 00:05:20.600 for your clothes to hypernaturalstyle .com. That's 00:05:20.600 --> 00:05:24.459 hypernaturalstyle .com where you can get some 00:05:24.459 --> 00:05:28.879 men's tees and button -ups. If you purchase from 00:05:28.879 --> 00:05:36.879 that link in my episode notes, you get 20 % off 00:05:36.879 --> 00:05:40.519 any product when you use my link in the episode 00:05:40.519 --> 00:05:45.470 notes and I get a 20 % commission off. from the 00:05:45.470 --> 00:05:49.350 total purchase you make when you use my link. 00:05:49.850 --> 00:05:52.509 And you might be wondering, so what is so cool 00:05:52.509 --> 00:05:55.670 about these hypernatural products? They have 00:05:55.670 --> 00:05:58.170 a patent pending hypercooled jade technology 00:05:58.170 --> 00:06:01.269 that when it's hot, lowers your body temperature 00:06:01.269 --> 00:06:04.829 by three to five degrees. They're also breathable, 00:06:04.829 --> 00:06:07.970 thanks to Supina cotton, and they are sold in 00:06:07.970 --> 00:06:11.649 premium retailers like Nordstrom. And we're rated 00:06:11.649 --> 00:06:16.019 the best. men's polo shirts in 2024 in Men's 00:06:16.019 --> 00:06:20.379 Journal, Forbes, and Esquire. And the last product 00:06:20.379 --> 00:06:24.500 I've recommended, which is new, that I've discovered, 00:06:24.600 --> 00:06:29.540 which is good, is Torain, T -O -R -A -I -N .org. 00:06:29.980 --> 00:06:34.220 That's Torain .org. They sell backpacks, totes, 00:06:34.220 --> 00:06:36.620 wallets, and bags. And what's good about them 00:06:36.620 --> 00:06:39.199 in the global south is they're durable, they're 00:06:39.199 --> 00:06:41.899 lightweight, they're water resistant, in case 00:06:41.899 --> 00:06:44.480 you're going during the rainy season perfect 00:06:44.480 --> 00:06:46.740 for travel and outdoor use if you are in the 00:06:46.740 --> 00:06:50.180 global south particularly in the field monitoring 00:06:50.180 --> 00:06:53.779 and evaluating a project all terrain products 00:06:53.779 --> 00:06:58.899 are crafted from upcycled feed and rice bags 00:06:58.899 --> 00:07:03.480 sourced globally each bag's interior is lined 00:07:03.480 --> 00:07:07.459 with recycled plastic bottles handcrafted by 00:07:07.459 --> 00:07:11.139 artisans supporting fair trade By purchasing 00:07:11.139 --> 00:07:14.720 a Touraine bag, customers contribute to global 00:07:14.720 --> 00:07:19.120 sustainability. You get 15 % off any product 00:07:19.120 --> 00:07:21.879 if you use my link that is in the episode notes. 00:07:22.040 --> 00:07:27.759 The link is mydeals .page forward slash 1HJL. 00:07:28.459 --> 00:07:32.060 When you do that, I get a 15 % commission of 00:07:32.060 --> 00:07:36.920 the total cart purchase. So just to refresh your 00:07:36.920 --> 00:07:39.879 memory. The Global Community Engagement Resilience 00:07:39.879 --> 00:07:44.319 Fund on their website is responsible, even though 00:07:44.319 --> 00:07:47.759 it's a fund, they're still responsible for the 00:07:47.759 --> 00:07:51.000 proper evaluation design of the projects that 00:07:51.000 --> 00:07:55.040 they fund, or as they say, grant. So at the grant 00:07:55.040 --> 00:07:57.480 level, this is quote from their website, at the 00:07:57.480 --> 00:08:00.579 grant level, each project is guided by a theory 00:08:00.579 --> 00:08:03.420 of change and a results framework with outcome. 00:08:04.060 --> 00:08:07.360 and output indicators requiring grantees to collect 00:08:07.360 --> 00:08:10.819 baseline data, report quarterly, and conduct 00:08:10.819 --> 00:08:14.220 final assessments. So it's clear that GSERF is 00:08:14.220 --> 00:08:17.959 responsible for making sure that the design of 00:08:17.959 --> 00:08:21.860 the evaluation is proper. That means valid outcome 00:08:21.860 --> 00:08:25.319 indicators. That means proper statistical analysis 00:08:25.319 --> 00:08:28.779 to support their continued claims that they're 00:08:28.779 --> 00:08:32.679 achieving their outcomes. On their website, they 00:08:32.679 --> 00:08:34.899 clearly state four outcomes we've talked about 00:08:34.899 --> 00:08:39.320 before, but they also have done a new expected 00:08:39.320 --> 00:08:41.820 outcome that they claim they're achieving, and 00:08:41.820 --> 00:08:46.759 that is the rehabilitation of returning foreign 00:08:46.759 --> 00:08:51.259 terrorist fighters. So we're going to take a 00:08:51.259 --> 00:08:54.820 closer look at their results from round one and 00:08:54.820 --> 00:08:58.240 round two funding in the Western Balkans. Report 00:08:58.240 --> 00:09:02.299 2020 to 2025. Again, They say it's an abridged 00:09:02.299 --> 00:09:04.879 version of the original paper, but I can't find 00:09:04.879 --> 00:09:07.580 the original paper. So to be fair to them, maybe 00:09:07.580 --> 00:09:10.480 the entire paper includes all of the outcome 00:09:10.480 --> 00:09:14.399 indicators. But what I've read so far is there's 00:09:14.399 --> 00:09:17.159 only one outcome indicator from this report on 00:09:17.159 --> 00:09:23.379 actual measurement of rehabilitation of returning 00:09:23.379 --> 00:09:27.159 foreign terrorist fighters. So we're only going 00:09:27.159 --> 00:09:29.019 to look at that one indicator. So we're going 00:09:29.019 --> 00:09:33.909 to start with just reading. from page one, paragraph 00:09:33.909 --> 00:09:38.950 two, or sorry, page two of that report, paragraph 00:09:38.950 --> 00:09:41.769 two. And I quote, GSURF has worked in the region 00:09:41.769 --> 00:09:47.129 since 2017. It started by supporting preventing 00:09:47.129 --> 00:09:50.490 violent extremism work in some foreign terrorist 00:09:50.490 --> 00:09:53.950 fighter communities of origin. And then later, 00:09:54.049 --> 00:09:58.769 reintegration and rehabilitation of returnees 00:09:58.769 --> 00:10:03.070 in Kosovo. In 2020, with the learnings of its 00:10:03.070 --> 00:10:06.809 experience in Kosovo, G -Surf approved its regional 00:10:06.809 --> 00:10:11.149 strategy focused on rehabilitation and reintegration 00:10:11.149 --> 00:10:15.049 covering Albania, Kosovo, and North Macedonia, 00:10:15.230 --> 00:10:19.929 later expanding to Bosnia and Herzegovina. So 00:10:19.929 --> 00:10:23.990 we've got one, two, three, four countries in 00:10:23.990 --> 00:10:27.909 the report. I see them. Albania, Bosnia and Herzegovina, 00:10:28.070 --> 00:10:32.470 Kosovo, and North. where they're involved in 00:10:32.470 --> 00:10:36.190 funding organizations that are delivering, among 00:10:36.190 --> 00:10:39.350 other things, I presume, psychological services. 00:10:39.490 --> 00:10:42.309 Those are your outputs to achieve the expected 00:10:42.309 --> 00:10:47.269 outcome of, quote, rehabilitation of returning 00:10:47.269 --> 00:10:51.950 foreign terrorist fighters. So there's a background 00:10:51.950 --> 00:10:56.509 for you. So if you go to page three of this report, 00:10:56.669 --> 00:11:01.059 figure three, it says, quote, progress on regional 00:11:01.059 --> 00:11:04.679 strategy indicators that represent key dimensions 00:11:04.679 --> 00:11:08.700 of its rehabilitation and reintegration strategy. 00:11:09.100 --> 00:11:11.519 And this is one of the problems you get into 00:11:11.519 --> 00:11:14.519 when the people at GSERF who are responsible 00:11:14.519 --> 00:11:17.879 for performance and impact have got the wrong 00:11:17.879 --> 00:11:20.159 university degrees. They've all got degrees, 00:11:20.460 --> 00:11:23.559 undergraduate, even undergraduate, bachelor's, 00:11:23.559 --> 00:11:27.539 master's, and PhD in the same international affairs 00:11:27.539 --> 00:11:32.210 subject area. So you obviously know right away 00:11:32.210 --> 00:11:35.190 that they're going to be obsessed with national 00:11:35.190 --> 00:11:39.370 level aggregation and even international combining 00:11:39.370 --> 00:11:43.730 across countries indicators. But that's not what 00:11:43.730 --> 00:11:46.289 you want here. What you want to see is for an 00:11:46.289 --> 00:11:49.389 individual project, which they, being international 00:11:49.389 --> 00:11:52.769 affairs degree people, are obsessed, will say, 00:11:53.009 --> 00:11:56.269 ooh, that's too granular. That's too granular. 00:11:56.370 --> 00:11:58.529 But that's how you're going to properly evaluate 00:11:58.529 --> 00:12:02.730 whether these, quote, returning foreign terrorist 00:12:02.730 --> 00:12:06.509 fighters have actually been, quote, rehabilitated. 00:12:06.529 --> 00:12:11.769 You can't combine this indicator on reintegration 00:12:11.769 --> 00:12:15.429 across several grants and then across several 00:12:15.429 --> 00:12:19.929 countries. Because even if you do, and even assuming 00:12:19.929 --> 00:12:22.330 your indicator is valid, where are you going 00:12:22.330 --> 00:12:25.009 to locate which grant is doing well? Which one 00:12:25.009 --> 00:12:27.690 isn't? You have to go back. Why do evaluation 00:12:27.690 --> 00:12:30.570 if you can't find out which grant is screwing 00:12:30.570 --> 00:12:34.750 up, which one's being effective? So that's a 00:12:34.750 --> 00:12:37.570 major problem. But let's get into the specifics. 00:12:39.149 --> 00:12:41.269 First of all, the weakness in the statistical 00:12:41.269 --> 00:12:43.889 analysis. So they have one indicator on page 00:12:43.889 --> 00:12:48.009 four that's related to, quote, reintegration 00:12:48.009 --> 00:12:54.919 of returning foreign terrorist fighters. foreign 00:12:54.919 --> 00:12:58.879 terrorist fighters and their families who report 00:12:58.879 --> 00:13:04.000 participation in social activities with members 00:13:04.000 --> 00:13:11.179 of receiving community so they start with a baseline 00:13:11.179 --> 00:13:15.940 percent that actually report participation who 00:13:15.940 --> 00:13:19.909 actually report participation at 70 percent At 00:13:19.909 --> 00:13:22.129 the end of the program on page four, they claim 00:13:22.129 --> 00:13:27.789 it's gone up to 93%. So they're claiming that 00:13:27.789 --> 00:13:32.870 there is a increase from 70 to 93%. The problem 00:13:32.870 --> 00:13:36.169 is they're combining it across several grants 00:13:36.169 --> 00:13:39.769 in several countries. So we don't know which 00:13:39.769 --> 00:13:44.450 grant in which country has actually gone up on 00:13:44.450 --> 00:13:47.500 this indicator and by how much. or gone down. 00:13:47.659 --> 00:13:49.980 The other problem is they're using an average 00:13:49.980 --> 00:13:53.360 of this percent. They should use the median because 00:13:53.360 --> 00:13:55.659 the median, as you've learned from your second 00:13:55.659 --> 00:13:59.200 year undergraduate stats course, the median is 00:13:59.200 --> 00:14:02.340 less sensitive to central tendency. Sorry, less 00:14:02.340 --> 00:14:06.379 sensitive to outlier, large values or small values, 00:14:06.480 --> 00:14:09.320 right? It's better to use the median and then 00:14:09.320 --> 00:14:11.440 maybe compare the median percent at baseline 00:14:11.440 --> 00:14:16.750 and see if it's gone up to... statistically significant 00:14:16.750 --> 00:14:19.850 up to 93%. So that's the first problem they have 00:14:19.850 --> 00:14:23.029 with their statistical analysis. The second problem 00:14:23.029 --> 00:14:27.350 is they've combined, not only averages is wrong, 00:14:27.549 --> 00:14:30.690 they've combined all the grants across all the 00:14:30.690 --> 00:14:33.970 projects within a country and across all the 00:14:33.970 --> 00:14:37.509 countries that I listed, which is what? How many 00:14:37.509 --> 00:14:39.950 countries we got? One, two, three, four. Bosnia 00:14:39.950 --> 00:14:43.710 and Herzegovina, Kosovo, Albania, North Macedonia. 00:14:44.110 --> 00:14:45.970 The other problem is I've mentioned in the other 00:14:45.970 --> 00:14:49.350 episode is this percent. We don't know if it's 00:14:49.350 --> 00:14:52.350 a sample percent or a population percent. It 00:14:52.350 --> 00:14:54.990 could be a population percent because when you 00:14:54.990 --> 00:14:58.309 read in the report, they only talk about on page 00:14:58.309 --> 00:15:03.710 five, 127 returnees. So they could easily without 00:15:03.710 --> 00:15:08.610 expense measure all 127, which gets to my more 00:15:08.610 --> 00:15:12.190 important point. If there's only 127, instead 00:15:12.190 --> 00:15:17.000 of... using a biased, poor indicator of the percent 00:15:17.000 --> 00:15:20.840 who, quote, report participation in social activities 00:15:20.840 --> 00:15:23.960 and their families, you're going to have a self 00:15:23.960 --> 00:15:27.259 -reporting bias. It doesn't objectively measure 00:15:27.259 --> 00:15:30.340 whether or not, quote, they've been reintegrated. 00:15:31.000 --> 00:15:35.139 It doesn't get to the core. So that's the major 00:15:35.139 --> 00:15:38.440 problem. The indicator is not a valid measure, 00:15:38.480 --> 00:15:42.029 and it's biased also. due to self -reporting 00:15:42.029 --> 00:15:45.970 bias, in claiming that they're actually been 00:15:45.970 --> 00:15:50.370 successful at reintegration over time, right? 00:15:51.470 --> 00:15:55.710 So what do you do instead? What is a valid indicator 00:15:55.710 --> 00:16:00.350 of reintegration of returning foreign terrorist 00:16:00.350 --> 00:16:04.090 fighters? Assuming that they're getting, and 00:16:04.090 --> 00:16:06.450 they talk about it in the report, but I don't 00:16:06.450 --> 00:16:10.360 want to focus. on these outputs, but I presume 00:16:10.360 --> 00:16:13.000 they're going to be delivering a variety of different 00:16:13.000 --> 00:16:15.720 psychological services. It is good that they've 00:16:15.720 --> 00:16:18.860 recognized the outcome indicator is the same 00:16:18.860 --> 00:16:22.860 across all countries, even though there's cross 00:16:22.860 --> 00:16:25.639 -cultural differences, but they've just chosen 00:16:25.639 --> 00:16:27.899 the wrong indicator because it's not a valid 00:16:27.899 --> 00:16:30.360 measure. It's not a standardized, objective, 00:16:30.639 --> 00:16:35.559 global indicator of measuring rehabilitation. 00:16:37.240 --> 00:16:40.179 reintegration of returning foreign terrorist 00:16:40.179 --> 00:16:44.539 fighters it's not but guess what there are a 00:16:44.539 --> 00:16:48.860 few valid psychometric tools out there and this 00:16:48.860 --> 00:16:52.320 is why that background education background in 00:16:52.320 --> 00:16:55.399 international affairs unfortunately gives them 00:16:55.399 --> 00:16:57.639 a bias where they're not interested in looking 00:16:57.639 --> 00:17:01.559 at psychology which is what this whole funding 00:17:01.559 --> 00:17:05.559 is about psychological services to get these 00:17:05.559 --> 00:17:09.619 former returning foreign terrorist fighters to 00:17:09.619 --> 00:17:15.140 rehabilitate right complex so you need to get 00:17:15.140 --> 00:17:19.759 a proper psychometric tool that looks at it so 00:17:19.759 --> 00:17:21.460 you might be wondering what is there out there 00:17:21.460 --> 00:17:25.759 so this is what i did i just entered into co 00:17:25.759 --> 00:17:29.880 -pilot the following question what is a valid 00:17:29.880 --> 00:17:33.940 psychometric tool to measure a reduction in an 00:17:33.940 --> 00:17:37.329 individual's predicted threat level over time 00:17:37.329 --> 00:17:42.309 where this individual was a former terrorist 00:17:42.309 --> 00:17:47.589 fighter. Up comes from co -pilot, quote, the 00:17:47.589 --> 00:17:51.130 most widely accepted valid psychometric and risk 00:17:51.130 --> 00:17:54.569 assessment tool specifically designed to measure 00:17:54.569 --> 00:17:57.650 changes in an individual's terrorist threat level 00:17:57.650 --> 00:18:02.549 over time is the, get ready for it, violent extremism 00:18:02.549 --> 00:18:06.829 risk assessment. V -E -R -A, Vera, version two 00:18:06.829 --> 00:18:10.549 revised. Now I thought, this is co -pilot. Sometimes 00:18:10.549 --> 00:18:13.049 you get wrong answers. So then I entered that 00:18:13.049 --> 00:18:17.109 into just a Google search and up came a website. 00:18:17.970 --> 00:18:25.670 www .vera -2r .nl backslash instrument. And lo 00:18:25.670 --> 00:18:27.549 and behold, it's been developed by the Dutch. 00:18:29.069 --> 00:18:31.369 But let me just give you a little bit more detail. 00:18:31.549 --> 00:18:34.789 Quote, unlike standard criminal risk tools, the 00:18:34.789 --> 00:18:38.450 VERA2R operates on a structured professional 00:18:38.450 --> 00:18:42.529 judgment framework. This methodology allows psychologists, 00:18:43.069 --> 00:18:45.589 intelligence professionals, and correctional 00:18:45.589 --> 00:18:49.750 staff to reevaluate a former fighter at regular 00:18:49.750 --> 00:18:53.710 intervals to observe whether dynamic risk factors 00:18:53.710 --> 00:18:57.450 are decreasing and protective factors are increasing. 00:18:59.849 --> 00:19:03.130 While several tools exist, two primary instruments 00:19:03.130 --> 00:19:06.589 are internationally recognized for monitoring 00:19:06.589 --> 00:19:10.509 a former fighter's rehabilitation and trajectory 00:19:10.509 --> 00:19:14.230 over time. It is an objective, evidence -based 00:19:14.230 --> 00:19:17.569 tool optimized for individuals who have been 00:19:17.569 --> 00:19:21.470 incarcerated or detained for ideologically motivated 00:19:21.470 --> 00:19:24.170 violence. By the way, if you're listening to 00:19:24.170 --> 00:19:27.230 this on Spotify, feel free to make comments. 00:19:27.829 --> 00:19:30.809 If my critique is flawed in any way, if I'm using 00:19:30.809 --> 00:19:34.089 the wrong psychometric tools, let everybody know 00:19:34.089 --> 00:19:36.849 what we should use because the taxpayer is paying 00:19:36.849 --> 00:19:40.809 for all of this, right? Measuring change. The 00:19:40.809 --> 00:19:43.569 tool specifically tracks 34 distinct indicators 00:19:43.569 --> 00:19:47.029 across five domains because many of these factors 00:19:47.029 --> 00:19:50.809 are dynamic, changeable. A practitioner can run 00:19:50.809 --> 00:19:53.829 the protocol every six to 12 months to chart 00:19:53.829 --> 00:19:57.200 an individual's progress or regression. So this 00:19:57.200 --> 00:19:59.660 is what GSERF should be using. The beauty of 00:19:59.660 --> 00:20:02.519 GSERF being a fund, they now have a tool here. 00:20:02.599 --> 00:20:05.680 All they have to do is find an organization that 00:20:05.680 --> 00:20:08.579 will be subjected to the training and use of 00:20:08.579 --> 00:20:11.599 the tool. Then they can just report back to GSERF 00:20:11.599 --> 00:20:15.480 on the indicator, right? So there's another one, 00:20:16.279 --> 00:20:23.200 ERG22 +, ERG, Extremism Risk Guidelines, developed 00:20:23.200 --> 00:20:26.319 in the United Kingdom. The extremism risk guidance 00:20:26.319 --> 00:20:30.819 22 plus is used heavily by forensic psychologists 00:20:30.819 --> 00:20:34.039 and probation officers to evaluate convicted 00:20:34.039 --> 00:20:37.279 terrorists. And you can Google that and you'll 00:20:37.279 --> 00:20:40.519 find there's a publication coming up on researchgate 00:20:40.519 --> 00:20:44.640 .net. The application over time as a former fighter 00:20:44.640 --> 00:20:47.980 undergoes rehabilitation programming. The ERG 00:20:47.980 --> 00:20:52.500 22 plus maps whether their intent to commit has 00:20:52.500 --> 00:20:55.920 deflated. even if they still hold a lingering 00:20:55.920 --> 00:20:59.640 portion of their original ideological engagement. 00:21:02.000 --> 00:21:04.559 Just to give you a little bit more, psychometric 00:21:04.559 --> 00:21:07.599 tools in this domain do not spit out a numerical 00:21:07.599 --> 00:21:11.220 score like an IQ test. Doing so is considered 00:21:11.220 --> 00:21:14.259 highly dangerous by security experts due to the 00:21:14.259 --> 00:21:18.779 low base rate of terrorist events. Instead, they 00:21:18.779 --> 00:21:21.599 measure threat reduction by looking for specific 00:21:21.599 --> 00:21:27.079 behavioral changes. Two more. Depending on the 00:21:27.079 --> 00:21:30.259 specific context of the former fighter, two other 00:21:30.259 --> 00:21:33.640 frameworks are occasionally paired with the VERA 00:21:33.640 --> 00:21:38.700 -2R. One, the TRAP -18, Terrorist Radicalization 00:21:38.700 --> 00:21:42.880 Assessment Protocol, TRAP, a tool consisting 00:21:42.880 --> 00:21:47.220 of eight proximal warning behaviors and ten distal 00:21:47.220 --> 00:21:51.000 characteristics. It is highly useful if security 00:21:51.000 --> 00:21:54.000 agencies need to determine the imminence of a 00:21:54.000 --> 00:21:57.299 threat, or if a former fighter is actively planning 00:21:57.299 --> 00:22:01.519 a relapse into operational activities. You can 00:22:01.519 --> 00:22:06.500 actually buy the TRAP -18 online. Another psychometric 00:22:06.500 --> 00:22:11.819 tool, the RAN. That's the European Radicalization 00:22:11.819 --> 00:22:14.700 Awareness Network. It's developed by the European 00:22:14.700 --> 00:22:19.119 Radicalization Awareness Network. This tool is 00:22:19.119 --> 00:22:21.440 optimized for foreign terrorist fighters who 00:22:21.440 --> 00:22:24.150 have returned from... conflict zones. Sounds 00:22:24.150 --> 00:22:27.690 pretty relevant to what GSURF talks about, returning 00:22:27.690 --> 00:22:31.009 foreign terrorist fighters. They should check 00:22:31.009 --> 00:22:35.309 this one out. This tool is optimized, blah, blah, 00:22:35.309 --> 00:22:37.670 blah, to measure their specific reintegration 00:22:37.670 --> 00:22:40.309 risks. So that even fits with the second part 00:22:40.309 --> 00:22:42.509 that GSURF is claiming they're achieving, which 00:22:42.509 --> 00:22:46.690 is reintegration after rehabilitation. Now, there 00:22:46.690 --> 00:22:49.349 is an indicator on reintegration, but I don't 00:22:49.349 --> 00:22:51.400 want to talk about it right now. And there's 00:22:51.400 --> 00:22:53.900 even a paper that reviews all the psychometric 00:22:53.900 --> 00:22:58.599 tools. But you can find that out online. So what 00:22:58.599 --> 00:23:02.059 about buying it? You cannot buy the Vera 2R manual 00:23:02.059 --> 00:23:05.880 or its psychometric protocols. So what you have 00:23:05.880 --> 00:23:08.980 to do is it's officially offered by the Netherlands 00:23:08.980 --> 00:23:12.660 Institute of Forensic Psychiatry and Psychology. 00:23:13.380 --> 00:23:18.980 And what they will do is they'll provide the 00:23:18.980 --> 00:23:21.690 certified training. curriculum development and 00:23:21.690 --> 00:23:24.289 user registries across Europe and works with 00:23:24.289 --> 00:23:28.029 licensed global trainers internationally. There's 00:23:28.029 --> 00:23:32.809 even regions outside of Europe that also are 00:23:32.809 --> 00:23:36.549 periodically licensed to run authorized cohorts. 00:23:37.089 --> 00:23:40.910 So if you contact them, you can learn more about 00:23:40.910 --> 00:23:44.269 the training. So again, GSERV, not being an NGO 00:23:44.269 --> 00:23:47.269 that delivers the services directly, they can 00:23:47.269 --> 00:23:50.059 get an organization. in their request for proposals 00:23:50.059 --> 00:23:53.140 that's willing to do that. And they could just 00:23:53.140 --> 00:23:55.960 fund them. And they'll go off and train these 00:23:55.960 --> 00:24:00.480 people using this valid outcome indicator on 00:24:00.480 --> 00:24:04.059 the reintegration of returning foreign terrorist 00:24:04.059 --> 00:24:06.359 fighters. And then all they have to do is come 00:24:06.359 --> 00:24:11.180 back to GSURF and report on this indicator as 00:24:11.180 --> 00:24:14.259 part of the evaluation framework that GSURF, 00:24:14.279 --> 00:24:16.980 that is, they're responsible for developing. 00:24:18.920 --> 00:24:21.740 So included in the purchase is the course curriculum, 00:24:22.000 --> 00:24:25.160 the official manual upon successful graduation, 00:24:25.660 --> 00:24:28.400 passing the qualification parameters. You're 00:24:28.400 --> 00:24:31.220 handed the physical VERA 2R professional manual. 00:24:31.400 --> 00:24:34.240 And then you're also, this practitioner is added 00:24:34.240 --> 00:24:36.680 to an elite monitored international register, 00:24:36.900 --> 00:24:40.779 which is great because if you can get local citizens 00:24:40.779 --> 00:24:44.069 in these countries. Kosovo, et cetera, to become 00:24:44.069 --> 00:24:46.849 practitioners, you could even argue that's one 00:24:46.849 --> 00:24:49.829 of the other outcome indicators that GSERF keeps 00:24:49.829 --> 00:24:52.269 claiming they're achieving, which is vocational 00:24:52.269 --> 00:24:54.789 skills training. So they've also achieved that 00:24:54.789 --> 00:24:58.130 and also additional income, increased incomes. 00:24:58.430 --> 00:25:01.869 So that's a great example where if they use this 00:25:01.869 --> 00:25:05.369 psychometric tool, they can achieve two outcomes 00:25:05.369 --> 00:25:08.789 at once. They can achieve an increase in vocational 00:25:08.789 --> 00:25:11.269 skills, followed by an increase in subsequent 00:25:11.269 --> 00:25:14.670 income. And hopefully if they're using that indicator, 00:25:14.869 --> 00:25:17.529 as I recommend they do, and also using these 00:25:17.529 --> 00:25:24.430 valid measures of rehabilitation of returning 00:25:24.430 --> 00:25:28.890 foreign terrorist fighters. So that gives you 00:25:28.890 --> 00:25:33.910 an example of how their choice of outcome indicators 00:25:33.910 --> 00:25:36.589 are not valid, and therefore they can't claim. 00:25:37.069 --> 00:25:39.470 like they are claiming that they're being successful 00:25:39.470 --> 00:25:44.609 at rehabilitation when they aren't, for the reasons 00:25:44.609 --> 00:25:46.750 I provided. They're not using a valid measure. 00:25:46.849 --> 00:25:51.109 I have provided, what is it, four valid measures, 00:25:51.150 --> 00:25:56.589 psychometrically, of rehabilitation of returning 00:25:56.589 --> 00:26:01.029 foreign terrorist fighters. To be clear here, 00:26:01.210 --> 00:26:03.650 the indicator that they have provided in this 00:26:03.650 --> 00:26:07.089 report that I've reviewed, that as invalid, is 00:26:07.089 --> 00:26:09.690 actually a measure of reintegration because they're 00:26:09.690 --> 00:26:13.150 asking the self -reporting bias of the returning 00:26:13.150 --> 00:26:16.130 foreign terrorist fighters to report on their 00:26:16.130 --> 00:26:18.910 participation. along with their families, which 00:26:18.910 --> 00:26:21.970 also is self -reporting bias, as to whether they 00:26:21.970 --> 00:26:25.029 have participated in social activities with members 00:26:25.029 --> 00:26:28.150 of the receiving community. That clearly suggests 00:26:28.150 --> 00:26:31.150 this is an indicator of reintegration, which 00:26:31.150 --> 00:26:33.650 further stresses the point that in the report 00:26:33.650 --> 00:26:36.349 that GSURF has produced, there's not a single 00:26:36.349 --> 00:26:41.690 indicator they have shown in that report that 00:26:41.690 --> 00:26:48.599 measures validly. rehabilitation. And so the 00:26:48.599 --> 00:26:50.900 four measures that I've offered, they should 00:26:50.900 --> 00:26:53.240 be looking at those. That's very important because 00:26:53.240 --> 00:26:55.839 unless I've missed something, those indicators, 00:26:56.140 --> 00:26:58.819 there's not a single one that deals with measuring 00:26:58.819 --> 00:27:02.740 validly rehabilitation. Now it might be in the 00:27:02.740 --> 00:27:05.420 larger report. We don't know, but you'd think 00:27:05.420 --> 00:27:07.700 they would have it in at least the shortened 00:27:07.700 --> 00:27:10.200 report that they've posted on their website because 00:27:10.200 --> 00:27:12.559 they're claiming that they have been successful 00:27:12.559 --> 00:27:16.690 in rehabilitating these fighters and obviously 00:27:16.690 --> 00:27:19.309 that's a precursor you first have to rehabilitate 00:27:19.309 --> 00:27:22.009 them before you reintegrate them right so that 00:27:22.009 --> 00:27:24.809 even stresses the point even further where are 00:27:24.809 --> 00:27:28.170 your valid indicators of rehabilitation because 00:27:28.170 --> 00:27:31.809 they're not there so g -serve has four that i've 00:27:31.809 --> 00:27:34.210 suggested in this episode that they should draw 00:27:34.210 --> 00:27:37.730 on so what i'm going to do now is since g -serve 00:27:37.730 --> 00:27:40.109 has received money from the government of canada 00:27:40.109 --> 00:27:43.309 in the past and they are listed on the Global 00:27:43.309 --> 00:27:47.309 Affairs Canada project browser website. I will 00:27:47.309 --> 00:27:50.690 be, as usual, sending an email to the Secretary 00:27:50.690 --> 00:27:53.269 of State for International Development, along 00:27:53.269 --> 00:27:57.009 with copies of that email. They'll be copied 00:27:57.009 --> 00:28:01.109 on it. All the opposition party critics, shout 00:28:01.109 --> 00:28:03.789 -out critics on international development, recommending 00:28:03.789 --> 00:28:07.430 that they use this indicator for any organization, 00:28:07.710 --> 00:28:11.170 including GSURF, that they're funding to deliver 00:28:11.170 --> 00:28:13.900 services. that the Canadian taxpayer is paying 00:28:13.900 --> 00:28:19.099 for to achieve this expected outcome of rehabilitation, 00:28:19.259 --> 00:28:26.880 of returning foreign terrorist fighters. Also, 00:28:26.920 --> 00:28:30.579 recommending that they follow the lead of GSURF 00:28:30.579 --> 00:28:33.819 in actually putting the performance measurement 00:28:33.819 --> 00:28:35.440 frameworks, that's what they're calling Canada, 00:28:35.640 --> 00:28:39.430 on the website. of the organization that's received 00:28:39.430 --> 00:28:42.049 the money from the Government of Canada. And 00:28:42.049 --> 00:28:44.549 even better, that they put them on the Government 00:28:44.549 --> 00:28:47.750 of Canada website, which they are currently not 00:28:47.750 --> 00:28:52.890 doing. So, thank you for listening. Bye for now.