WEBVTT 00:00:00.000 --> 00:00:02.620 Welcome to the Improving Development Evaluation 00:00:02.620 --> 00:00:06.519 Podcast. I'm your host, David Wand, and in this 00:00:06.519 --> 00:00:08.980 episode, we're going to revisit the Global Community 00:00:08.980 --> 00:00:12.599 Engagement Resilience Fund. You can learn more 00:00:12.599 --> 00:00:15.199 about the Global Community Engagement Resilience 00:00:15.199 --> 00:00:19.019 Fund, GSURF, at gsurf .org. You might be wondering 00:00:19.019 --> 00:00:22.660 why I am returning to this particular international 00:00:22.660 --> 00:00:25.719 development organization. because I featured 00:00:25.719 --> 00:00:29.039 them on two previous episodes. The reason is 00:00:29.039 --> 00:00:32.939 I've discovered they've actually improved development 00:00:32.939 --> 00:00:36.000 evaluation in some ways, and in some ways they 00:00:36.000 --> 00:00:39.119 haven't, but it's a step in the right direction. 00:00:40.399 --> 00:00:43.259 And you may have figured out through my LinkedIn 00:00:43.259 --> 00:00:46.640 profile that I used to be at GSurf back in April 00:00:46.640 --> 00:00:50.320 of 2020. So that's more than six years ago. Back 00:00:50.320 --> 00:00:53.799 then, they were not in the mood. to do what I've 00:00:53.799 --> 00:00:58.259 been trying to get organizations to do when it 00:00:58.259 --> 00:01:00.979 comes to improving development evaluation. That 00:01:00.979 --> 00:01:03.939 is putting on their website their performance 00:01:03.939 --> 00:01:06.480 measurement framework, or as the Americans call 00:01:06.480 --> 00:01:09.579 it, the activity monitoring evaluation learning. 00:01:10.040 --> 00:01:13.420 plan or as other people call them results frameworks 00:01:13.420 --> 00:01:16.079 these are documents that list the outcomes they 00:01:16.079 --> 00:01:19.019 expect to achieve along with the outcome indicators 00:01:19.019 --> 00:01:23.120 they expect to use to measure hopefully validly 00:01:23.120 --> 00:01:25.120 the outcome statements that they're claiming 00:01:25.120 --> 00:01:30.340 they want to achieve also the data that's associated 00:01:30.340 --> 00:01:33.400 with those outcome indicators and finally the 00:01:33.400 --> 00:01:36.680 statistical analysis that shows that if the outcome 00:01:36.680 --> 00:01:40.310 indicator is indeed a valid measure of the outcome 00:01:40.310 --> 00:01:43.469 and if for example the percentage on that outcome 00:01:43.469 --> 00:01:46.870 indicator has indeed increased over time from 00:01:46.870 --> 00:01:51.450 baseline to midline that that increase in percent 00:01:51.450 --> 00:01:55.890 is actually due to statistical significance if 00:01:55.890 --> 00:01:59.170 they're using samples rather than due to chance 00:01:59.170 --> 00:02:03.090 that's what we're trying to get proper evaluation 00:02:03.090 --> 00:02:06.709 of the project so that the claims that g -surf 00:02:06.709 --> 00:02:10.500 keeps making on its website and also on its podcast, 00:02:10.919 --> 00:02:14.840 actually have been achieved because of G -Surf 00:02:14.840 --> 00:02:18.819 and not due to chance. So far, they have failed 00:02:18.819 --> 00:02:22.219 to do that. So just in retrospect, you may recall 00:02:22.219 --> 00:02:25.500 on one episode I showed they received $5 million 00:02:25.500 --> 00:02:28.759 from the United States government. I received 00:02:28.759 --> 00:02:31.719 two letters from the United States State Department 00:02:31.719 --> 00:02:34.439 and the United States Agency for International 00:02:34.439 --> 00:02:37.680 Development saying, We have no activity mail 00:02:37.680 --> 00:02:40.400 plan. We have no outcome indicators for that 00:02:40.400 --> 00:02:44.360 $5 million that has been spent. Bye -bye. So 00:02:44.360 --> 00:02:46.960 that's a problem. Number two, they received $1 00:02:46.960 --> 00:02:51.379 .1 million from the Canadian government. Again, 00:02:51.639 --> 00:02:55.840 I asked for the results framework. It took me 00:02:55.840 --> 00:02:59.020 about a year. I finally got a letter. what's 00:02:59.020 --> 00:03:02.400 more like a report from GSERF. And in that report, 00:03:02.580 --> 00:03:05.560 there was absolutely no mentioning of outcome 00:03:05.560 --> 00:03:09.500 indicators showing any evidence that they had 00:03:09.500 --> 00:03:12.360 achieved any of the four outcomes that they keep 00:03:12.360 --> 00:03:14.099 claiming they're achieving on their website. 00:03:14.379 --> 00:03:17.539 Those four outcomes are one, increased social 00:03:17.539 --> 00:03:20.780 cohesion, two, increased ability to mobilize, 00:03:21.020 --> 00:03:23.639 organize, represent their own interests, three, 00:03:23.759 --> 00:03:27.120 increased vocational skills, followed by an increase. 00:03:27.689 --> 00:03:29.889 in livelihood, such as an increase in income. 00:03:30.069 --> 00:03:33.349 And four, increased confidence or increased critical 00:03:33.349 --> 00:03:36.310 thinking skills or increased life skills or increased 00:03:36.310 --> 00:03:39.569 self -worth or increased resilience, what they 00:03:39.569 --> 00:03:44.250 like to call sense of purpose. But finally, this 00:03:44.250 --> 00:03:47.330 episode, we are going to celebrate the fact that 00:03:47.330 --> 00:03:51.310 GSurf has moved on. And despite their resistance 00:03:51.310 --> 00:03:54.210 over six years ago, when I was with them, they 00:03:54.210 --> 00:03:57.939 were refusing to even mention. a results framework 00:03:57.939 --> 00:04:01.840 being on their website with indicators and they 00:04:01.840 --> 00:04:05.919 were also dismissing trying to measure all of 00:04:05.919 --> 00:04:08.939 the social psychological constructs that they 00:04:08.939 --> 00:04:12.180 are claiming they're achieving like vocational 00:04:12.180 --> 00:04:16.339 skills increases social cohesion increases attitude 00:04:16.339 --> 00:04:18.620 increases in the right direction or perception 00:04:18.620 --> 00:04:22.319 increases they were not interested at all and 00:04:22.319 --> 00:04:24.689 one of the reasons they were not interested at 00:04:24.689 --> 00:04:27.449 all, was because most of them there at GSERF 00:04:27.449 --> 00:04:31.350 have the wrong degrees. They don't have an undergraduate 00:04:31.350 --> 00:04:34.209 degree in psychology, which is all you need to 00:04:34.209 --> 00:04:38.470 do these evaluation of these social psychology 00:04:38.470 --> 00:04:41.829 and clinical psychology constructs. And all you 00:04:41.829 --> 00:04:44.050 need to do is complete that second year undergraduate 00:04:44.050 --> 00:04:47.129 university course in inferential statistics, 00:04:47.329 --> 00:04:50.829 and you'd be off and running. But most of them, 00:04:50.829 --> 00:04:53.620 unfortunately, at GSERF have PhDs and masters 00:04:53.620 --> 00:04:58.480 in international relations or international affairs. 00:04:58.759 --> 00:05:02.319 And I've met students who have master's degrees 00:05:02.319 --> 00:05:05.060 in international affairs and relations, even 00:05:05.060 --> 00:05:08.220 undergraduate degrees in international relations 00:05:08.220 --> 00:05:10.980 or affairs. And they never have to take a course 00:05:10.980 --> 00:05:14.839 in inferential statistics. Why? Because they're 00:05:14.839 --> 00:05:17.899 often looking at macro level indicators like 00:05:17.899 --> 00:05:21.689 trade statistics or whatever. So there's a mismatch. 00:05:21.930 --> 00:05:25.189 That is one of the many reasons as to why GSURF 00:05:25.189 --> 00:05:30.069 was resistant to looking at these basic skills 00:05:30.069 --> 00:05:33.750 that you need to properly evaluate the services 00:05:33.750 --> 00:05:36.269 that they're delivering, where they're all focusing 00:05:36.269 --> 00:05:39.829 on changing individuals and groups of individuals 00:05:39.829 --> 00:05:43.970 in the four outcomes that they clearly state 00:05:43.970 --> 00:05:46.569 on their website. But now they have finally, 00:05:46.730 --> 00:05:51.850 on their website, released and published some 00:05:51.850 --> 00:05:55.050 results and impact reports so I'm going to look 00:05:55.050 --> 00:05:59.449 at one of them from Ghana and go through it and 00:05:59.449 --> 00:06:03.050 give you an idea of how far they've come at least 00:06:03.050 --> 00:06:06.069 they've published it on the website which is 00:06:06.069 --> 00:06:09.290 a pioneering effort compared to a lot of NGOs 00:06:09.290 --> 00:06:12.569 and number two they've actually published the 00:06:12.569 --> 00:06:15.769 data on their outcome indicators so we're going 00:06:15.769 --> 00:06:17.629 to look at whether or not the outcome indicators 00:06:18.379 --> 00:06:21.379 are indeed valid measures of the outcome? And 00:06:21.379 --> 00:06:25.000 number two, even if they are, the data that they 00:06:25.000 --> 00:06:27.819 provide, have they gone to the proper level of 00:06:27.819 --> 00:06:30.800 statistical analysis? That is, have they gone 00:06:30.800 --> 00:06:33.220 back to their notes in second year university 00:06:33.220 --> 00:06:36.000 where they took that required inferential statistics 00:06:36.000 --> 00:06:39.500 course and figured out this is what I should 00:06:39.500 --> 00:06:42.459 do to make the claim that the percentage increase 00:06:42.459 --> 00:06:46.220 that I'm claiming has actually gone up due to 00:06:46.399 --> 00:06:50.060 to statistical significance rather than due to 00:06:50.060 --> 00:06:54.100 chance, that is understanding the basic second 00:06:54.100 --> 00:06:58.199 year concept of sampling distributions, then 00:06:58.199 --> 00:07:02.699 we're in a good phase. So before I do that, if 00:07:02.699 --> 00:07:07.220 you are living in the global south or you, God 00:07:07.220 --> 00:07:10.639 forbid, have to go to the global south to Allah 00:07:10.639 --> 00:07:15.500 forbid, monitor and evaluate. an actual international 00:07:15.500 --> 00:07:19.540 development project in the field, you need a 00:07:19.540 --> 00:07:22.980 few things. So I'm not going to be talking about 00:07:22.980 --> 00:07:26.079 desiccated liver or granola. I'm going to talk 00:07:26.079 --> 00:07:28.660 about products that actually are related to going 00:07:28.660 --> 00:07:31.560 to the global south or living in the global south, 00:07:31.660 --> 00:07:35.259 where most of the time it's hot and humid, or 00:07:35.259 --> 00:07:37.980 if it's not, it's raining. You get the idea. 00:07:38.120 --> 00:07:44.540 So sunglasses, check out Aquila. A -K -I -L -A 00:07:44.540 --> 00:07:48.779 dot L -A, where you can get sunglasses, including 00:07:48.779 --> 00:07:52.720 prescription lenses, all made from 50 % recycled 00:07:52.720 --> 00:07:55.920 material. You get a 10 % discount if you use 00:07:55.920 --> 00:07:59.920 my code, WANDCOOLT -SHIRTS, or if you use the 00:07:59.920 --> 00:08:03.680 actual link in the episode notes, which is mydeals 00:08:03.680 --> 00:08:09.819 .page backslash 1HJL. You'll see it in the episode 00:08:09.819 --> 00:08:13.670 notes. And I get a 15 % commission when you purchase 00:08:13.670 --> 00:08:19.949 any product using that My Deals link. So you 00:08:19.949 --> 00:08:22.189 have your sunglasses to protect you from the 00:08:22.189 --> 00:08:24.970 sun in the global south, but you need the right 00:08:24.970 --> 00:08:27.649 clothes to be comfortable in that hot and humid 00:08:27.649 --> 00:08:33.129 climate. So just to remind you, AKILA .LA. That's 00:08:33.129 --> 00:08:37.330 the website. But better just to click on the 00:08:37.330 --> 00:08:40.960 My Deals. Link in the episode notes. The second 00:08:40.960 --> 00:08:45.440 product I've got is men's polos, tees, and button 00:08:45.440 --> 00:08:49.559 ups at hypernaturalstyle .com. You get 20 % off 00:08:49.559 --> 00:08:53.940 any product when you use my, my deals link or 00:08:53.940 --> 00:08:57.539 the promo code wand, cool t -shirts. And I get 00:08:57.539 --> 00:09:00.279 a 20 % commission of the total purchase you make 00:09:00.279 --> 00:09:06.509 when you use that link. And we. Want to know, 00:09:06.610 --> 00:09:09.529 so what is so cool about Hypernatural products? 00:09:09.769 --> 00:09:13.169 They have a patent pending Hypercool Jade technology 00:09:13.169 --> 00:09:16.309 that when it's hot, lowers your body temperature 00:09:16.309 --> 00:09:19.730 by three to five degrees. They're also breathable 00:09:19.730 --> 00:09:23.029 thanks to Supina cotton and they are sold in 00:09:23.029 --> 00:09:26.470 premium retailers like Nordstrom and were voted 00:09:26.470 --> 00:09:30.509 the best men's polo shirt in 2024 in Men's Journal, 00:09:30.690 --> 00:09:34.549 Forbes and Esquire. And I've got a new product. 00:09:37.039 --> 00:09:40.539 which is if you go to terrain .org, that's T 00:09:40.539 --> 00:09:46.100 -O -R -R -A -I -N .org, they have backpacks, 00:09:46.100 --> 00:09:49.460 totes, wallets, and bags. But what's really cool 00:09:49.460 --> 00:09:53.059 about them is, again, they're developed for the 00:09:53.059 --> 00:09:55.399 global south. They're durable, lightweight, water 00:09:55.399 --> 00:09:58.200 -resistant, perfect for travel and outdoor use. 00:09:58.340 --> 00:10:01.080 If you are in the global south, in the field, 00:10:01.200 --> 00:10:05.409 monitoring and evaluating a project. All Terrain 00:10:05.409 --> 00:10:09.429 products are crafted from upcycled feed and rice 00:10:09.429 --> 00:10:14.090 bags sourced globally. Each bag's interior is 00:10:14.090 --> 00:10:17.549 lined with recycled plastic bottles, handcrafted 00:10:17.549 --> 00:10:22.649 by artisans supporting fair trade. And by purchasing 00:10:22.649 --> 00:10:26.129 a Terrain bag, customers contribute to global 00:10:26.129 --> 00:10:29.490 sustainability, which is what international development 00:10:29.490 --> 00:10:34.110 is all about. You get a 15 % off any product 00:10:34.110 --> 00:10:36.950 if you use my link that is in the episode notes. 00:10:37.049 --> 00:10:43.610 Again, mydeals .page backslash 1HJL. I get a 00:10:43.610 --> 00:10:47.750 15 % commission of the total cart purchase. That's 00:10:47.750 --> 00:10:52.389 Terrain. Look at this Ghana report that's called, 00:10:52.529 --> 00:10:56.669 if you go to the gsurf .org website, results 00:10:56.669 --> 00:11:00.120 paper. Ghana emergency grant. Now to be fair 00:11:00.120 --> 00:11:03.279 if you go to that grant it'll say it's an abridged 00:11:03.279 --> 00:11:06.519 version of a larger report. Unfortunately I couldn't 00:11:06.519 --> 00:11:10.019 find that larger report so maybe some of my comments 00:11:10.019 --> 00:11:12.679 that I'm going to raise right now are in that 00:11:12.679 --> 00:11:17.399 larger report. I do not know. I'm assuming they 00:11:17.399 --> 00:11:19.120 are not in there because you would certainly 00:11:19.120 --> 00:11:22.019 think for what they post on their website if 00:11:22.019 --> 00:11:25.100 they knew their statistics they would indicate 00:11:25.100 --> 00:11:28.419 what I'm going to describe. in this abridged 00:11:28.419 --> 00:11:32.360 shorter version report. But I'm so happy that 00:11:32.360 --> 00:11:35.700 at least they've moved from trying to hide their 00:11:35.700 --> 00:11:38.460 performance measurement framework to a point 00:11:38.460 --> 00:11:41.360 where they're putting parts of it in this report 00:11:41.360 --> 00:11:45.480 on their website. So there's four sort of areas 00:11:45.480 --> 00:11:47.600 where there's a weakness in this Ghana report. 00:11:48.159 --> 00:11:50.559 First of all, some of their outcome indicators, 00:11:50.840 --> 00:11:54.240 just like in other episodes with other NGOs that 00:11:54.240 --> 00:11:57.279 I've talked about, there's a poor design. where 00:11:57.279 --> 00:12:00.139 the indicator really doesn't validly measure 00:12:00.139 --> 00:12:03.899 the expected outcome. Number two, they don't 00:12:03.899 --> 00:12:08.480 mention in this report just the title of the 00:12:08.480 --> 00:12:10.799 psychometric tool that they're using to measure 00:12:10.799 --> 00:12:14.019 the outcome indicator. And I'm going to suggest 00:12:14.019 --> 00:12:17.799 some possible psychometric tools that they could 00:12:17.799 --> 00:12:21.230 use, reminding them. that if they had an undergraduate 00:12:21.230 --> 00:12:24.769 degree in psychology with that second year required 00:12:24.769 --> 00:12:27.590 course in inferential statistics, this would 00:12:27.590 --> 00:12:31.070 not be an issue. And number three, there are 00:12:31.070 --> 00:12:33.509 some missing outcome indicators on skills training, 00:12:33.730 --> 00:12:36.190 but again, they might be in the larger report, 00:12:36.450 --> 00:12:40.429 to be fair to them. Number four, and this is 00:12:40.429 --> 00:12:44.090 the new part, which I commend GSERF for at least 00:12:44.090 --> 00:12:48.230 putting the data on their website so we can at 00:12:48.230 --> 00:12:51.480 least have a... discussion and that is their 00:12:51.480 --> 00:12:55.120 failure to show that the percent increases that 00:12:55.120 --> 00:12:58.100 they've shown on certain indicators is actually 00:12:58.100 --> 00:13:02.480 quote statistically significant rather than due 00:13:02.480 --> 00:13:05.799 to chance and that's a basic concept that you 00:13:05.799 --> 00:13:09.419 learn in second year undergraduate social sciences 00:13:09.419 --> 00:13:12.080 or business because you have to take that stats 00:13:12.080 --> 00:13:15.039 course so if you're majoring in psychology sociology 00:13:15.039 --> 00:13:19.850 political science or economics That's why you 00:13:19.850 --> 00:13:22.309 take that course, because later on in the literature, 00:13:22.490 --> 00:13:25.649 you're going to see p -value. And when I say 00:13:25.649 --> 00:13:28.210 p -value, I'm not talking about the quantity 00:13:28.210 --> 00:13:33.009 of urine in a bottle, right? So let's take a 00:13:33.009 --> 00:13:38.570 look. Poor design of outcome indicators. So one 00:13:38.570 --> 00:13:43.090 of the things they look at in their report is 00:13:43.090 --> 00:13:46.870 they talk about improved livelihood and employment. 00:13:47.009 --> 00:13:50.480 That's the expected outcome. for women. They 00:13:50.480 --> 00:13:56.519 got 200 women they list in the project. And they 00:13:56.519 --> 00:13:58.500 have the indicator for that as being the percent 00:13:58.500 --> 00:14:01.659 of women from host and refugee communities with, 00:14:01.779 --> 00:14:04.759 here's the problem, quote, sufficient productive 00:14:04.759 --> 00:14:08.639 assets with access to economic opportunities. 00:14:10.019 --> 00:14:12.580 That's a problem. They've got two variables in 00:14:12.580 --> 00:14:15.720 there, sufficient productive asset and access 00:14:15.720 --> 00:14:18.590 to economic opportunities. Anybody who knows 00:14:18.590 --> 00:14:21.169 how to properly design an indicator, you don't 00:14:21.169 --> 00:14:23.649 throw in two variables within one indicator. 00:14:23.909 --> 00:14:26.649 That's the first problem. Second problem is, 00:14:26.769 --> 00:14:30.169 as they've already stated on their website, and 00:14:30.169 --> 00:14:32.909 I've quoted it earlier, they are responsible 00:14:32.909 --> 00:14:38.210 for developing a theory of change. In this report, 00:14:38.350 --> 00:14:41.409 they indicate clearly that they are going to 00:14:41.409 --> 00:14:44.750 be training women in tailoring, parboiling rice, 00:14:45.029 --> 00:14:48.029 and shea butter processing. Training them on 00:14:48.029 --> 00:14:51.049 that. That's the theory of change. If we train 00:14:51.049 --> 00:14:53.549 them in that, we've already figured out they're 00:14:53.549 --> 00:14:57.649 going to earn more income from those vocational 00:14:57.649 --> 00:14:59.909 skills they've acquired. They've also figured 00:14:59.909 --> 00:15:03.730 out, hopefully by their theory of change, that 00:15:03.730 --> 00:15:07.649 there is adequate demand for tailoring services, 00:15:08.029 --> 00:15:11.009 parboiled rice, and shea butter. They've already 00:15:11.009 --> 00:15:14.370 figured that out. I lived in Ghana for three 00:15:14.370 --> 00:15:17.779 years. You have to be aware they're experiencing 00:15:17.779 --> 00:15:21.299 importations in rice from Thailand and the United 00:15:21.299 --> 00:15:24.440 States. Maybe there isn't a market demand for 00:15:24.440 --> 00:15:26.679 their parboiled rice. Maybe there is because 00:15:26.679 --> 00:15:29.539 they're taking the imported rice, just boiling 00:15:29.539 --> 00:15:31.980 it and then reselling it. That's fine. But they've 00:15:31.980 --> 00:15:35.299 already figured out from their theory of change, 00:15:35.399 --> 00:15:37.340 this is the training we're going to do and it's 00:15:37.340 --> 00:15:39.679 going to lead to an increase in vocational skill 00:15:39.679 --> 00:15:44.899 followed by an increase in income. Done. indicator 00:15:44.899 --> 00:15:48.700 is too vague and too complex all they really 00:15:48.700 --> 00:15:52.639 need to do is look at the mean income of this 00:15:52.639 --> 00:15:56.379 group of 200 women before when the project started 00:15:56.379 --> 00:15:59.139 versus later on in the project they don't need 00:15:59.139 --> 00:16:01.059 this percent they should actually be tracking 00:16:01.059 --> 00:16:04.500 the income and also more importantly making sure 00:16:04.500 --> 00:16:08.860 that that income they're tracking is from tailoring 00:16:08.860 --> 00:16:11.620 par boiling rice selling it and selling shea 00:16:11.620 --> 00:16:14.960 butter right So that's a poor indicator. They 00:16:14.960 --> 00:16:17.639 need to improve that because it's not a valid 00:16:17.639 --> 00:16:20.399 measure. The second thing related to that is 00:16:20.399 --> 00:16:22.600 they're missing, and again, it might be in the 00:16:22.600 --> 00:16:26.759 larger report, they're missing no indicators, 00:16:26.919 --> 00:16:30.279 outcome indicators, showing an increase in vocational 00:16:30.279 --> 00:16:33.419 skill due to the training. You need to test these 00:16:33.419 --> 00:16:36.139 women after they get the training to see if they 00:16:36.139 --> 00:16:39.120 can actually tailor properly, to see if they 00:16:39.120 --> 00:16:41.720 actually know how to process shea butter properly, 00:16:41.960 --> 00:16:45.669 right? There's no measures of that short -term 00:16:45.669 --> 00:16:48.450 expected outcome first, because first they have 00:16:48.450 --> 00:16:51.590 to acquire the vocational skill. Pretty straightforward, 00:16:51.929 --> 00:16:54.649 pretty easy to show, especially when you have 00:16:54.649 --> 00:16:59.149 only 200 women that you're working with. So there's 00:16:59.149 --> 00:17:01.970 a failure there on the outcome indicator for 00:17:01.970 --> 00:17:04.789 skills training. For the youth, it's a little 00:17:04.789 --> 00:17:07.329 better, but for the women, it's missing. The 00:17:07.329 --> 00:17:10.839 third issue is... And it's a very simple one. 00:17:10.880 --> 00:17:13.460 It's a basic one. And that is their failure to 00:17:13.460 --> 00:17:16.859 specify the psychometric tool that they're using 00:17:16.859 --> 00:17:22.720 to measure increased trust. For example, they 00:17:22.720 --> 00:17:24.799 have a perception change as an expected outcome 00:17:24.799 --> 00:17:27.880 for the youth. One of the indicators is percent 00:17:27.880 --> 00:17:30.740 of youth who trust people from other communities, 00:17:30.819 --> 00:17:35.700 ethnic backgrounds. So they don't indicate it. 00:17:35.740 --> 00:17:38.180 They may have a psychometric tool that they've 00:17:38.180 --> 00:17:41.200 designed. that the evaluation consultant was 00:17:41.200 --> 00:17:44.039 hired to design, or that they designed even before 00:17:44.039 --> 00:17:46.440 the project started. All they have to do is indicate 00:17:46.440 --> 00:17:49.579 what that is. And what I've done is just for 00:17:49.579 --> 00:17:55.220 fun, I used Copilot and found four, well, I'll 00:17:55.220 --> 00:17:58.440 talk about three different standardized valid 00:17:58.440 --> 00:18:01.579 measures of generalized trust. The first one, 00:18:01.680 --> 00:18:04.500 generalized trust scale developed by Toshio. 00:18:04.920 --> 00:18:08.480 Yamagishi, highly reliable six -item psychometric 00:18:08.480 --> 00:18:11.359 questionnaire. It measures an individual's expectation 00:18:11.359 --> 00:18:14.339 of human benevolence and whether they view trusting 00:18:14.339 --> 00:18:18.500 others as a high risk. It is widely considered 00:18:18.500 --> 00:18:21.019 the gold standard for cross -cultural. That's 00:18:21.019 --> 00:18:23.640 important because G -Surf is operating in different 00:18:23.640 --> 00:18:26.500 countries. Cross -cultural psychological research. 00:18:26.880 --> 00:18:29.480 Another one, it's only one question, comes from 00:18:29.480 --> 00:18:32.519 the World Values Survey, trust question. It's 00:18:32.519 --> 00:18:35.269 a single item. forced choice question used across 00:18:35.269 --> 00:18:40.710 global sociological studies. A third one is Rotor's 00:18:40.710 --> 00:18:43.789 Interpersonal Trust Scale, a classic comprehensive 00:18:43.789 --> 00:18:48.289 25 -item tool designed by Julian Rotor. It measures 00:18:48.289 --> 00:18:52.009 a person's generalized expectancy that the verbal 00:18:52.009 --> 00:18:54.569 or written promises of others can be relied upon. 00:18:54.869 --> 00:18:58.450 Now, another option for them is to hire a psychologist 00:18:58.450 --> 00:19:04.390 to develop a proprietary private just for g -surf 00:19:04.390 --> 00:19:08.190 measurement tool on trust because they may have 00:19:08.190 --> 00:19:10.769 a particular interest and i had recommended in 00:19:10.769 --> 00:19:13.089 a previous episode that i dealt with g -surf 00:19:13.089 --> 00:19:16.029 they could also use for example if they want 00:19:16.029 --> 00:19:18.710 to measure levels of trust over time between 00:19:18.710 --> 00:19:21.549 madrasa school students religious school students 00:19:21.549 --> 00:19:25.309 and non -religious public school students where 00:19:25.309 --> 00:19:28.430 they've held a soccer match and they want to 00:19:29.960 --> 00:19:33.299 see if before the soccer match compared to during 00:19:33.299 --> 00:19:36.480 and after the levels of social capital or trust 00:19:36.480 --> 00:19:39.019 between those two groups has increased you can 00:19:39.019 --> 00:19:41.880 do it with the university of california irvine 00:19:41.880 --> 00:19:45.299 net uc net software package all they have to 00:19:45.299 --> 00:19:48.859 do is mention in their performance results framework 00:19:48.859 --> 00:19:53.240 like a report like this what tool did they actually 00:19:53.240 --> 00:19:57.480 use so those are the issues there the final one 00:19:59.109 --> 00:20:01.490 is failure to show that the percent increase 00:20:01.490 --> 00:20:05.710 is statistically significant based on sampling 00:20:05.710 --> 00:20:10.509 or that the entire population was measured instead. 00:20:10.769 --> 00:20:12.789 It's not clear if you look at the Ghana report. 00:20:13.069 --> 00:20:19.890 They do use the S word. They do mention on page 00:20:19.890 --> 00:20:24.849 six, and I quote, the percentage of youth rejecting 00:20:24.849 --> 00:20:29.609 violence rose from 69 % to 74%, though these 00:20:29.609 --> 00:20:33.670 changes were not statistically significant. So 00:20:33.670 --> 00:20:36.829 that implies that they, out of the 600 youth 00:20:36.829 --> 00:20:41.769 that they mentioned, that they trained on critical 00:20:41.769 --> 00:20:46.170 thinking and digital literacy training, that 00:20:46.170 --> 00:20:51.750 the percent went up from 69 % to 74%. Okay, fair 00:20:51.750 --> 00:20:54.420 enough. It's not statistically significant, but 00:20:54.420 --> 00:20:56.920 what would be ideal is that they actually showed 00:20:56.920 --> 00:21:00.839 the sample sizes from the 600. I mean, did they 00:21:00.839 --> 00:21:05.019 do it twice, two samples? And then you would 00:21:05.019 --> 00:21:07.420 understand they have to use a sampling distribution, 00:21:07.680 --> 00:21:11.400 one tail to the right, 95 % confidence. It takes 00:21:11.400 --> 00:21:14.420 on a Z normal distribution, the sampling distribution 00:21:14.420 --> 00:21:17.509 for two. population proportions, if you take 00:21:17.509 --> 00:21:20.930 samples once at baseline and once at midline, 00:21:21.049 --> 00:21:23.210 this is something they should basically be doing 00:21:23.210 --> 00:21:26.109 and using in all of their reports. If they're 00:21:26.109 --> 00:21:30.769 going to go to the point of using the words statistically 00:21:30.769 --> 00:21:33.329 significant, then they have to go back to their 00:21:33.329 --> 00:21:36.390 second year undergraduate notes and say, ah, 00:21:36.549 --> 00:21:39.849 this is the test I use. And they can also do 00:21:39.849 --> 00:21:42.950 it for the mean income for the women at baseline 00:21:42.950 --> 00:21:44.970 and midline, right? There's a different test 00:21:44.970 --> 00:21:47.349 for that. You probably remember it. It takes 00:21:47.349 --> 00:21:50.410 on a T distribution. If I understand correctly, 00:21:50.690 --> 00:21:53.170 yep. I'm just doing it from memory, but you can 00:21:53.170 --> 00:21:57.329 look it up. One tail to the right. You have to 00:21:57.329 --> 00:21:59.670 figure out your degrees of freedom. So they get 00:21:59.670 --> 00:22:01.910 the idea. So that's what they need to do next. 00:22:02.690 --> 00:22:05.650 Maybe they can go back to the reports and make 00:22:05.650 --> 00:22:08.269 it clear that they are drawing samples. The other 00:22:08.269 --> 00:22:10.789 issue that is a little confusing from the report 00:22:10.789 --> 00:22:13.910 is they're hinting here that they are doing sample 00:22:13.910 --> 00:22:16.910 proportions. But if you look at the populations, 00:22:17.390 --> 00:22:20.089 they're not too large to begin with. It's 200 00:22:20.089 --> 00:22:24.190 women that got the training on tailoring, parboiling 00:22:24.190 --> 00:22:27.710 rice, shea butter processing. And it's 600 youth 00:22:27.710 --> 00:22:31.890 individuals that got the training. And it's not 00:22:31.890 --> 00:22:35.390 clear if they measured the entire populations 00:22:35.390 --> 00:22:38.950 or samples. I think it's samples. And if it is, 00:22:39.049 --> 00:22:42.250 they need to show the sample sizes and do the 00:22:42.250 --> 00:22:45.019 statistical analysis. One tail. to the right, 00:22:45.119 --> 00:22:49.599 right? Get the critical value for T or Z, depending 00:22:49.599 --> 00:22:53.819 on the variable you're looking at, either a mean 00:22:53.819 --> 00:22:58.839 or a proportion. And that would go a long way 00:22:58.839 --> 00:23:02.940 to showing that GSURF is getting slowly, after 00:23:02.940 --> 00:23:06.079 six years, to the point where they're producing 00:23:06.079 --> 00:23:08.920 data to support their claims that they've been 00:23:08.920 --> 00:23:13.349 making since 2020 with no evidence. So now they're 00:23:13.349 --> 00:23:15.250 getting to the point where they can actually 00:23:15.250 --> 00:23:19.130 make claims that the percentage increase that 00:23:19.130 --> 00:23:21.950 they observed and measured was not due to chance 00:23:21.950 --> 00:23:23.990 because of their knowledge of sampling distribution 00:23:23.990 --> 00:23:27.390 and instead was, quote, statistically significant. 00:23:28.210 --> 00:23:32.410 So that's good. And I thank them for moving in 00:23:32.410 --> 00:23:35.609 that direction. It's great to see. So I think 00:23:35.609 --> 00:23:37.569 what I'll do is I'll look at the other reports 00:23:37.569 --> 00:23:42.670 and do episodes. on this analysis and also their 00:23:42.670 --> 00:23:47.609 episodes they've also started a podcast and I 00:23:47.609 --> 00:23:50.670 made a comment to this effect on their first 00:23:50.670 --> 00:23:52.910 episode where again they made claims that the 00:23:52.910 --> 00:23:56.009 percentage had increased and it wasn't clear 00:23:56.009 --> 00:23:58.789 if it was an output or an outcome even if it's 00:23:58.789 --> 00:24:01.309 an output where they're trying to reach more 00:24:01.309 --> 00:24:05.269 scale it up as they say you need to show that 00:24:05.269 --> 00:24:08.569 it's statistically significant But more importantly, 00:24:08.630 --> 00:24:10.849 I'm focused on the outcomes because that's what 00:24:10.849 --> 00:24:15.470 they like to claim on their website. So thank 00:24:15.470 --> 00:24:18.670 you for listening. And I'll be back with another 00:24:18.670 --> 00:24:21.650 episode on GSurf on another one of their reports. 00:24:21.890 --> 00:24:25.250 I'm particularly interested in their rehabilitation 00:24:25.250 --> 00:24:29.390 and reintegration, which they talk about on their 00:24:29.390 --> 00:24:33.609 first episode. And those are basic, again, psychological 00:24:33.609 --> 00:24:37.910 constructs that have been around for years. where 00:24:37.910 --> 00:24:41.150 there's all sorts of standardized measures of 00:24:41.150 --> 00:24:44.329 how do we define and measure reintegration? How 00:24:44.329 --> 00:24:47.430 do we define and measure rehabilitation? And 00:24:47.430 --> 00:24:51.150 it'd be interesting to see if in those reports, 00:24:51.309 --> 00:24:55.390 they actually make an effort to do some statistical 00:24:55.390 --> 00:24:58.289 analysis rather than just saying the percent 00:24:58.289 --> 00:25:02.390 went up. Thank you for your time. Bye for now.