1 00:00:00,000 --> 00:00:05,840 All right, buckle up everybody because today we are diving into some serious AI intrigue. 2 00:00:05,840 --> 00:00:06,960 Oh yeah. 3 00:00:06,960 --> 00:00:10,960 We are tackling a paper that asks a question. 4 00:00:11,760 --> 00:00:16,720 I bet a lot of you out there have been wondering, can AI be full of it? 5 00:00:16,720 --> 00:00:20,880 You're not holding back at all. But yeah, I mean that's basically what this study from the 6 00:00:20,880 --> 00:00:23,600 University of Cambridge is looking into. They're really trying to. 7 00:00:23,600 --> 00:00:26,160 They want to see if we can actually measure bullshit. 8 00:00:26,160 --> 00:00:29,360 In the language models that power things like chat GPT. 9 00:00:29,360 --> 00:00:30,160 Exactly. 10 00:00:30,160 --> 00:00:30,880 Okay, hold on. 11 00:00:30,880 --> 00:00:31,280 Yeah. 12 00:00:31,280 --> 00:00:32,160 Measuring bullshit. 13 00:00:32,160 --> 00:00:33,360 It sounds kind of funny. 14 00:00:33,360 --> 00:00:33,680 I know. 15 00:00:33,680 --> 00:00:34,960 Like is that a real thing? 16 00:00:34,960 --> 00:00:36,880 It's more real than you might think. 17 00:00:36,880 --> 00:00:37,600 Okay. 18 00:00:37,600 --> 00:00:41,840 These researchers are using a very specific definition of bullshit. 19 00:00:41,840 --> 00:00:42,320 Okay. 20 00:00:42,320 --> 00:00:45,280 From a philosopher named Harry Frankfurt. 21 00:00:45,280 --> 00:00:45,920 Okay. 22 00:00:45,920 --> 00:00:48,960 And basically it's language that doesn't care about the truth. 23 00:00:49,600 --> 00:00:51,760 It just cares about sounding good. 24 00:00:51,760 --> 00:00:52,320 Okay. 25 00:00:52,320 --> 00:00:57,440 And that's super important because AI is getting really good at sounding like us. 26 00:00:57,440 --> 00:00:57,760 Right. 27 00:00:57,760 --> 00:00:59,920 Even if it doesn't actually understand what it's saying. 28 00:00:59,920 --> 00:01:01,600 So it's like that friend we all have. 29 00:01:02,880 --> 00:01:04,240 Who can talk a good game? 30 00:01:04,240 --> 00:01:04,800 Totally. 31 00:01:05,520 --> 00:01:07,840 But when you actually listen to what they're saying. 32 00:01:07,840 --> 00:01:08,320 Right. 33 00:01:08,320 --> 00:01:10,720 It's all fluff and no substance. 34 00:01:10,720 --> 00:01:11,120 Got it. 35 00:01:11,680 --> 00:01:12,320 Okay. 36 00:01:12,320 --> 00:01:12,640 So this paper. 37 00:01:12,640 --> 00:01:17,600 It's all about trying to create a bullshit detector for AI. 38 00:01:17,600 --> 00:01:18,080 Wow. 39 00:01:18,080 --> 00:01:20,240 Specifically for chat GPT. 40 00:01:20,240 --> 00:01:21,680 A bullshit detector. 41 00:01:21,680 --> 00:01:22,000 Yeah. 42 00:01:22,000 --> 00:01:23,120 That's kind of wild. 43 00:01:23,120 --> 00:01:23,760 It is. 44 00:01:23,760 --> 00:01:26,160 How do you even begin to measure something like that? 45 00:01:26,160 --> 00:01:26,800 Right. 46 00:01:26,800 --> 00:01:28,880 I mean, bullshit is so subjective. 47 00:01:28,880 --> 00:01:29,360 Yeah. 48 00:01:29,360 --> 00:01:29,600 Right. 49 00:01:29,600 --> 00:01:30,560 It really is. 50 00:01:30,560 --> 00:01:34,800 What one person thinks is BS, another person might think is totally legit. 51 00:01:34,800 --> 00:01:37,040 That is what makes this study so interesting. 52 00:01:37,040 --> 00:01:43,040 They had to figure out a way to objectively measure something that feels really subjective. 53 00:01:43,040 --> 00:01:44,800 So they did something pretty clever. 54 00:01:44,800 --> 00:01:45,120 Okay. 55 00:01:45,120 --> 00:01:48,720 They tricked chat GPT into writing fake scientific articles. 56 00:01:48,720 --> 00:01:49,360 Wait a minute. 57 00:01:49,360 --> 00:01:49,600 Yeah. 58 00:01:49,600 --> 00:01:51,840 They made chat GPT write fake science. 59 00:01:51,840 --> 00:01:52,800 Huh? 60 00:01:52,800 --> 00:01:54,000 How do they do that? 61 00:01:54,000 --> 00:01:58,960 They basically fed it all the instructions for writing a paper for the general nature. 62 00:01:59,600 --> 00:02:00,400 Oh, wow. 63 00:02:00,400 --> 00:02:02,720 You know, like the super prestigious science publication? 64 00:02:02,720 --> 00:02:03,440 Very prestigious. 65 00:02:04,000 --> 00:02:04,320 Yeah. 66 00:02:04,320 --> 00:02:07,920 And chat GPT just spat out these incredibly convincing articles. 67 00:02:07,920 --> 00:02:08,160 Right. 68 00:02:08,160 --> 00:02:10,400 Complete with made up data and everything. 69 00:02:10,400 --> 00:02:11,120 No way. 70 00:02:11,120 --> 00:02:11,600 Yeah. 71 00:02:11,600 --> 00:02:17,520 So chat GPT is actually capable of writing convincing scientific BS. 72 00:02:18,240 --> 00:02:19,040 It seems so. 73 00:02:19,040 --> 00:02:20,000 That's a little scary. 74 00:02:20,000 --> 00:02:22,000 It definitely raises some eyebrows. 75 00:02:22,000 --> 00:02:22,240 Yeah. 76 00:02:22,240 --> 00:02:23,680 But here's the really cool part. 77 00:02:23,680 --> 00:02:23,920 Okay. 78 00:02:24,960 --> 00:02:27,600 They then use those fake articles. 79 00:02:28,160 --> 00:02:28,560 Okay. 80 00:02:28,560 --> 00:02:32,400 Along with a bunch of real nature articles to train their bullshit detector. 81 00:02:32,960 --> 00:02:36,400 So they created a control group of pure BS. 82 00:02:36,400 --> 00:02:36,880 Yes. 83 00:02:36,880 --> 00:02:38,640 And then compared it to the real deal. 84 00:02:38,640 --> 00:02:39,440 Exactly. 85 00:02:39,440 --> 00:02:39,600 Okay. 86 00:02:39,600 --> 00:02:40,960 I'm starting to see where this is going. 87 00:02:40,960 --> 00:02:41,200 Yeah. 88 00:02:41,200 --> 00:02:44,080 But how do they actually build this BS detector? 89 00:02:44,080 --> 00:02:46,320 It's algorithmic, but not magic. 90 00:02:46,320 --> 00:02:46,480 Okay. 91 00:02:46,480 --> 00:02:47,520 It's not magic. 92 00:02:47,520 --> 00:02:49,600 They use two different machine learning models. 93 00:02:49,600 --> 00:02:50,080 Okay. 94 00:02:50,080 --> 00:02:52,080 The first one is called XGBoost. 95 00:02:52,080 --> 00:02:52,560 Okay. 96 00:02:52,560 --> 00:02:56,320 And it focuses on how often certain words show up in the text. 97 00:02:56,320 --> 00:02:56,640 Okay. 98 00:02:56,640 --> 00:03:01,200 Like let's say you see the word quantum a bunch of times in a paper that's supposed to be about gardening. 99 00:03:01,200 --> 00:03:02,240 Yeah, that would be weird. 100 00:03:02,240 --> 00:03:03,360 That's probably a red flag. 101 00:03:03,360 --> 00:03:03,600 Right. 102 00:03:03,600 --> 00:03:04,480 A big red flag. 103 00:03:04,480 --> 00:03:05,360 Yeah, exactly. 104 00:03:05,360 --> 00:03:05,840 Yep. 105 00:03:05,840 --> 00:03:08,240 And the second model they use is called Roberta. 106 00:03:08,240 --> 00:03:08,720 Okay. 107 00:03:08,720 --> 00:03:11,040 And this one's a bit more sophisticated. 108 00:03:11,040 --> 00:03:11,600 Okay. 109 00:03:11,600 --> 00:03:14,240 It doesn't just look at the words themselves. 110 00:03:14,240 --> 00:03:14,720 Uh-huh. 111 00:03:14,720 --> 00:03:16,640 It analyzes the whole context. 112 00:03:16,640 --> 00:03:17,040 It's okay. 113 00:03:17,040 --> 00:03:18,800 Of how words are being used together. 114 00:03:18,800 --> 00:03:21,120 So it's kind of like a linguistic detective. 115 00:03:21,120 --> 00:03:21,280 Yeah. 116 00:03:21,280 --> 00:03:22,320 Trying to understand. 117 00:03:22,320 --> 00:03:23,360 That's a good way to put it. 118 00:03:23,360 --> 00:03:24,560 The deeper meaning. 119 00:03:24,560 --> 00:03:25,120 Yeah. 120 00:03:25,120 --> 00:03:26,960 Or lack thereof behind the words. 121 00:03:26,960 --> 00:03:27,680 Exactly. 122 00:03:27,680 --> 00:03:35,920 It's looking for those subtle patterns in the language that give away whether something is genuine or just a bunch of fancy sounding nonsense. 123 00:03:35,920 --> 00:03:38,000 I like that bunch of fancy sounding nonsense. 124 00:03:38,000 --> 00:03:38,400 Yeah. 125 00:03:38,400 --> 00:03:41,280 And the results they got with these models are pretty mind blowing. 126 00:03:41,280 --> 00:03:41,520 Okay. 127 00:03:41,520 --> 00:03:42,800 I'm on the edge of my seat. 128 00:03:42,800 --> 00:03:43,520 Oh, bad. 129 00:03:43,520 --> 00:03:44,640 What did they find? 130 00:03:44,640 --> 00:03:51,280 Both models were incredibly accurate at telling the real nature articles from the chat GPT fakes. 131 00:03:51,280 --> 00:03:52,240 Really? 132 00:03:52,240 --> 00:03:54,480 We're talking 100% accuracy. 133 00:03:54,480 --> 00:03:55,360 Get out of here. 134 00:03:55,360 --> 00:03:55,920 I'm serious. 135 00:03:55,920 --> 00:03:56,400 Yeah. 136 00:03:56,400 --> 00:03:56,880 100%. 137 00:03:56,880 --> 00:03:57,520 Yeah. 138 00:03:57,520 --> 00:04:02,960 The XGBoost model was 99.84% confident in its judgments. 139 00:04:02,960 --> 00:04:03,520 Wow. 140 00:04:03,520 --> 00:04:06,960 And the Roberta model was even higher at 99.97%. 141 00:04:06,960 --> 00:04:08,240 Okay. 142 00:04:08,240 --> 00:04:08,960 100%. 143 00:04:08,960 --> 00:04:09,680 Yeah. 144 00:04:09,680 --> 00:04:11,280 That's remarkable. 145 00:04:11,280 --> 00:04:15,920 It's like they created this BS detector that's practically foolproof. 146 00:04:15,920 --> 00:04:16,400 Yeah. 147 00:04:16,400 --> 00:04:25,760 But wait, if they can spot this AI generated BS so easily, shouldn't they just be able to, you know, fix chat GPT? 148 00:04:25,760 --> 00:04:26,800 That's a great question. 149 00:04:26,800 --> 00:04:29,360 Make it stop producing BS in the first place. 150 00:04:29,360 --> 00:04:31,280 It's more complicated than you might think. 151 00:04:31,280 --> 00:04:31,840 Yeah. 152 00:04:31,840 --> 00:04:32,240 Yeah. 153 00:04:32,240 --> 00:04:40,240 To really understand why chat GPT is so good at BSing, we need to look at the difference between what's called a large language model. 154 00:04:40,240 --> 00:04:40,640 Okay. 155 00:04:40,640 --> 00:04:41,360 Or LLM. 156 00:04:41,360 --> 00:04:42,160 LLM. 157 00:04:42,160 --> 00:04:43,040 And a chat bot. 158 00:04:43,040 --> 00:04:43,360 Right. 159 00:04:43,360 --> 00:04:44,640 Built on top of that model. 160 00:04:44,640 --> 00:04:44,880 Okay. 161 00:04:44,880 --> 00:04:45,840 I think I'm following you. 162 00:04:45,840 --> 00:04:46,160 Yeah. 163 00:04:46,160 --> 00:04:48,720 LLM's are like the underlying technology, right? 164 00:04:48,720 --> 00:04:48,960 That'd be cool. 165 00:04:48,960 --> 00:04:50,560 Kind of like the engine of the whole thing. 166 00:04:50,560 --> 00:04:51,040 Yeah. 167 00:04:51,040 --> 00:04:53,200 But what makes a chat bot different? 168 00:04:53,200 --> 00:04:56,720 So an LLM is like this massive library of text. 169 00:04:56,720 --> 00:04:57,520 Okay. 170 00:04:57,520 --> 00:05:03,200 It can predict what the next word in a sequence is going to be based on all the data that it's been trained on. 171 00:05:03,200 --> 00:05:03,680 Right. 172 00:05:03,680 --> 00:05:05,760 But it doesn't actually understand the meaning. 173 00:05:05,760 --> 00:05:06,160 Okay. 174 00:05:06,160 --> 00:05:08,160 Or the truth of what it's saying. 175 00:05:08,160 --> 00:05:08,800 Uh-huh. 176 00:05:08,800 --> 00:05:11,360 It's just really, really good at pattern recognition. 177 00:05:11,360 --> 00:05:13,360 So it's all about the patterns, not the meaning. 178 00:05:13,360 --> 00:05:13,920 Exactly. 179 00:05:13,920 --> 00:05:15,840 It's like those autocomplete features on our phones. 180 00:05:15,840 --> 00:05:16,400 Yeah. 181 00:05:16,400 --> 00:05:20,480 They can predict the next word you're going to type, but they don't actually know what you're trying to say. 182 00:05:20,480 --> 00:05:21,520 Perfect analogy. 183 00:05:21,520 --> 00:05:22,000 Okay. 184 00:05:22,000 --> 00:05:27,680 Now a chat bot takes that LLM and it adds something called a dialogue management system. 185 00:05:27,680 --> 00:05:28,160 Okay. 186 00:05:28,160 --> 00:05:29,600 Or DMS on top of it. 187 00:05:29,600 --> 00:05:30,240 Okay, got it. 188 00:05:30,240 --> 00:05:34,880 And the DMS is what makes it feel like you're having a conversation. 189 00:05:34,880 --> 00:05:35,280 Okay. 190 00:05:35,280 --> 00:05:40,560 It provides instructions and prompts and it even uses reinforcement learning. 191 00:05:40,560 --> 00:05:40,640 Wow. 192 00:05:40,640 --> 00:05:43,680 To make the responses sound more human-like. 193 00:05:43,680 --> 00:05:47,280 So the DMS is kind of like the chat bot's personality. 194 00:05:47,280 --> 00:05:48,800 Yeah, you could say that. 195 00:05:48,800 --> 00:05:53,840 It's like shaping how the LLM's output is being presented to the user. 196 00:05:53,840 --> 00:05:54,400 Uh-huh. 197 00:05:54,400 --> 00:05:59,760 It's like the difference between getting like a raw data dump and a carefully crafted story. 198 00:05:59,760 --> 00:06:00,480 You nailed it. 199 00:06:00,480 --> 00:06:00,880 Okay. 200 00:06:00,880 --> 00:06:08,960 And the researchers in this paper, they argue that it's actually this DMS with its focus on mimicking human conversation. 201 00:06:08,960 --> 00:06:09,520 Yeah. 202 00:06:09,520 --> 00:06:12,480 That encourages chat GPT to produce BS. 203 00:06:12,480 --> 00:06:13,120 Really? 204 00:06:13,120 --> 00:06:13,600 Yeah. 205 00:06:13,600 --> 00:06:14,240 Why is that? 206 00:06:14,240 --> 00:06:15,360 Well, it's all about incentives. 207 00:06:16,080 --> 00:06:20,320 The DMS is designed to make the chat bot sound engaging and believable. 208 00:06:20,960 --> 00:06:23,120 Even if that means sacrificing accuracy. 209 00:06:23,120 --> 00:06:23,520 Okay. 210 00:06:23,520 --> 00:06:28,720 Remember, the underlying LLM doesn't actually know what's true or false. 211 00:06:28,720 --> 00:06:28,960 Right. 212 00:06:28,960 --> 00:06:36,240 So it can easily get led astray by the DMS's desire to create a really convincing conversation. 213 00:06:36,240 --> 00:06:37,600 So it's like a politician. 214 00:06:37,600 --> 00:06:38,160 Oh, yeah. 215 00:06:38,160 --> 00:06:40,080 Who's really good at giving speeches. 216 00:06:40,080 --> 00:06:40,720 Uh-huh. 217 00:06:40,720 --> 00:06:44,320 But doesn't actually have any substance behind the words. 218 00:06:44,320 --> 00:06:44,560 Yeah. 219 00:06:44,560 --> 00:06:46,480 They're just saying what people want to hear. 220 00:06:46,480 --> 00:06:47,040 Exactly. 221 00:06:47,040 --> 00:06:48,480 Even if it's a little BS. 222 00:06:48,480 --> 00:06:52,560 Or think of it like a really smooth talking salesperson. 223 00:06:52,560 --> 00:06:52,960 Okay. 224 00:06:52,960 --> 00:06:53,200 Yeah. 225 00:06:53,200 --> 00:06:57,520 Who's more interested in closing the deal than telling you the whole truth about the product. 226 00:06:57,520 --> 00:06:58,000 I see. 227 00:06:58,000 --> 00:06:58,480 I see. 228 00:06:58,480 --> 00:06:58,720 Yeah. 229 00:06:58,720 --> 00:06:58,960 Okay. 230 00:06:58,960 --> 00:06:59,520 That makes sense. 231 00:06:59,520 --> 00:06:59,840 Yeah. 232 00:06:59,840 --> 00:07:02,880 So is that why people keep saying that chat GPT hallucinates? 233 00:07:02,880 --> 00:07:04,240 That's a big part of it. 234 00:07:04,240 --> 00:07:08,800 Because it's basically making stuff up to fill in the gaps in its knowledge. 235 00:07:08,800 --> 00:07:13,120 The term hallucination is really interesting because it implies that the chat bot is like 236 00:07:13,120 --> 00:07:16,240 actually perceiving something that isn't real. 237 00:07:16,240 --> 00:07:17,760 But that's not really what's happening. 238 00:07:17,760 --> 00:07:18,240 Okay. 239 00:07:18,240 --> 00:07:24,640 It's more like it's manipulating language to create this really convincing facade. 240 00:07:24,640 --> 00:07:24,960 Okay. 241 00:07:24,960 --> 00:07:26,880 Even if there's nothing real behind it. 242 00:07:26,880 --> 00:07:31,040 So how do we avoid falling for this AI generated BS? 243 00:07:31,040 --> 00:07:32,480 That's the million dollar question. 244 00:07:32,480 --> 00:07:33,360 I know. 245 00:07:33,360 --> 00:07:35,680 I mean these chat bots are getting so sophisticated. 246 00:07:35,680 --> 00:07:35,760 Right. 247 00:07:35,760 --> 00:07:37,920 It's hard to tell what's real and what's not. 248 00:07:37,920 --> 00:07:38,720 It really is. 249 00:07:38,720 --> 00:07:41,280 And that's one of the big takeaways from this paper. 250 00:07:41,280 --> 00:07:41,680 Okay. 251 00:07:41,680 --> 00:07:46,480 We need to be critical consumers of all this AI generated content. 252 00:07:47,120 --> 00:07:51,600 Just because something sounds really convincing doesn't mean it's true. 253 00:07:51,600 --> 00:07:52,240 Yeah. 254 00:07:52,240 --> 00:07:56,400 We need to develop our own BS detectors so to speak. 255 00:07:56,400 --> 00:07:59,760 So we can't just blindly trust what AI tells us. 256 00:07:59,760 --> 00:08:00,320 Not at all. 257 00:08:00,320 --> 00:08:05,520 We need to be asking questions and checking sources and being aware of the potential for 258 00:08:05,520 --> 00:08:06,880 bias and manipulation. 259 00:08:06,880 --> 00:08:07,360 Exactly. 260 00:08:07,360 --> 00:08:09,360 Just like we do with any other source of information. 261 00:08:09,360 --> 00:08:09,920 Absolutely. 262 00:08:09,920 --> 00:08:11,120 Hit the nail on the head. 263 00:08:11,120 --> 00:08:12,640 This isn't just about AI. 264 00:08:12,640 --> 00:08:12,880 Yeah. 265 00:08:12,880 --> 00:08:16,160 It's about how we consume information in general. 266 00:08:16,160 --> 00:08:16,640 I like that. 267 00:08:16,640 --> 00:08:19,520 We need to develop our own internal BS detectors. 268 00:08:20,160 --> 00:08:20,400 Yeah. 269 00:08:20,400 --> 00:08:21,600 That's a great way to put it. 270 00:08:21,600 --> 00:08:21,840 Yeah. 271 00:08:21,840 --> 00:08:25,040 So what can we do to kind of sharpen our BS detection skills? 272 00:08:25,040 --> 00:08:29,680 Well, first of all, be aware of the limitations of AI. 273 00:08:29,680 --> 00:08:30,000 Okay. 274 00:08:30,000 --> 00:08:32,000 Remember these language models? 275 00:08:32,000 --> 00:08:34,080 They're trained on massive amounts of data. 276 00:08:34,080 --> 00:08:34,640 Right. 277 00:08:34,640 --> 00:08:37,280 But they don't actually understand the world the way we do. 278 00:08:37,280 --> 00:08:39,520 They're just really good at recognizing patterns. 279 00:08:39,520 --> 00:08:40,080 Exactly. 280 00:08:40,080 --> 00:08:40,560 Yeah. 281 00:08:40,560 --> 00:08:44,640 So don't be afraid to question what you see and what you hear. 282 00:08:44,640 --> 00:08:45,280 Right. 283 00:08:45,280 --> 00:08:48,800 If something seems too good to be true or it just doesn't sit right with you, 284 00:08:49,520 --> 00:08:54,240 dig a little deeper, check the sources, look for alternative perspectives, 285 00:08:54,240 --> 00:08:57,520 and don't be afraid to challenge the information that you're given. 286 00:08:57,520 --> 00:08:59,440 That's good advice for life in general. 287 00:08:59,440 --> 00:09:00,480 I agree with you there. 288 00:09:00,480 --> 00:09:02,000 Not just when we're dealing with AI. 289 00:09:02,000 --> 00:09:02,640 Absolutely. 290 00:09:02,640 --> 00:09:03,280 Okay, cool. 291 00:09:03,280 --> 00:09:08,400 And I think this research also has implications for how we develop and design these AI systems 292 00:09:08,400 --> 00:09:09,040 in the future. 293 00:09:09,040 --> 00:09:09,520 Yeah. 294 00:09:09,520 --> 00:09:13,360 If we want AI to be a tool for truth and understanding, 295 00:09:13,360 --> 00:09:17,360 we have to be very careful about the incentives that we build into these systems. 296 00:09:17,360 --> 00:09:21,680 So we need to make sure we're not rewarding AI for being a good bullshitter, basically. 297 00:09:21,680 --> 00:09:22,160 Precisely. 298 00:09:22,160 --> 00:09:28,000 We need to prioritize accuracy and transparency over fluency and persuasiveness. 299 00:09:28,000 --> 00:09:28,720 Got it. 300 00:09:28,720 --> 00:09:33,440 Otherwise, we risk creating a world where BS reigns supreme. 301 00:09:33,440 --> 00:09:35,440 Yeah, that's a scary thought. 302 00:09:35,440 --> 00:09:36,240 It is a little bit. 303 00:09:36,240 --> 00:09:36,800 But you know what? 304 00:09:37,360 --> 00:09:39,280 This paper didn't just stop there. 305 00:09:39,280 --> 00:09:40,000 No, they didn't. 306 00:09:40,000 --> 00:09:42,400 They didn't just build a BS detector and call it a day. 307 00:09:42,400 --> 00:09:43,920 They took it out for a spin. 308 00:09:43,920 --> 00:09:44,480 They did. 309 00:09:44,480 --> 00:09:45,920 In the real world, so to speak. 310 00:09:45,920 --> 00:09:46,880 Oh, this is getting good. 311 00:09:46,880 --> 00:09:47,840 I know, right? 312 00:09:47,840 --> 00:09:49,520 Where do they take this BS detector? 313 00:09:49,520 --> 00:09:51,920 They started with something that's never been in short supply. 314 00:09:52,560 --> 00:09:52,880 What's that? 315 00:09:52,880 --> 00:09:54,000 Political language. 316 00:09:54,000 --> 00:09:54,880 Okay, now we're talking. 317 00:09:54,880 --> 00:09:56,160 Oh, I knew you'd like that. 318 00:09:56,160 --> 00:09:59,680 Politics and BS, name a more iconic duo. 319 00:10:00,080 --> 00:10:01,280 I can't. 320 00:10:01,280 --> 00:10:04,320 What did they do analyze political speeches for BS? 321 00:10:04,880 --> 00:10:10,960 They looked at political party manifestos from the UK, going all the way back to 1945. 322 00:10:11,920 --> 00:10:17,280 They wanted to see if these manifestos would trigger their BS detector. 323 00:10:17,280 --> 00:10:19,440 And they got some pretty interesting results. 324 00:10:19,440 --> 00:10:20,720 Okay, spill the tea. 325 00:10:20,720 --> 00:10:27,280 Okay, so to make it a fair comparison, they also analyzed a bunch of transcripts of everyday 326 00:10:27,280 --> 00:10:31,920 conversations, like people chatting with their friends or having lessons in school. 327 00:10:32,480 --> 00:10:37,360 Basically the kind of language we use when we're not trying to be persuasive or manipulative. 328 00:10:37,920 --> 00:10:42,560 So they pitted political manifestos against everyday chit chat? 329 00:10:42,560 --> 00:10:43,440 Pretty much. 330 00:10:43,440 --> 00:10:43,920 I love it. 331 00:10:43,920 --> 00:10:44,320 Yeah. 332 00:10:44,320 --> 00:10:44,880 What happened? 333 00:10:44,880 --> 00:10:47,360 The BS detector definitely picked up a signal. 334 00:10:47,920 --> 00:10:48,400 Really? 335 00:10:48,400 --> 00:10:54,960 The average BS score for the political manifestos was way higher than for the everyday conversations. 336 00:10:54,960 --> 00:10:56,560 Wow, so what you're saying is... 337 00:10:56,560 --> 00:10:57,280 It seems like it. 338 00:10:57,840 --> 00:11:00,080 Politicians really are full of BS. 339 00:11:00,080 --> 00:11:01,360 It does appear that way. 340 00:11:01,360 --> 00:11:02,400 Shocking, I know. 341 00:11:03,280 --> 00:11:05,120 But seriously, what does this tell us? 342 00:11:05,120 --> 00:11:08,480 Well, it suggests that there's something about political language. 343 00:11:09,200 --> 00:11:15,040 At least the language used in these manifestos that shares characteristics with the kind of BS 344 00:11:15,040 --> 00:11:17,120 that chat GPT produces. 345 00:11:17,120 --> 00:11:22,960 It's not necessarily that every politician is intentionally lying or trying to deceive, 346 00:11:23,680 --> 00:11:28,640 but there's definitely a tendency to use language that's more about persuasion than truth. 347 00:11:29,200 --> 00:11:30,240 Okay, that makes sense. 348 00:11:30,240 --> 00:11:30,800 Yeah. 349 00:11:30,800 --> 00:11:32,400 But wait, there's more, right? 350 00:11:32,400 --> 00:11:32,960 There is. 351 00:11:33,520 --> 00:11:38,800 You said they took this BS detector to other places besides the political arena. 352 00:11:38,800 --> 00:11:41,280 They did, and this is where it gets really interesting. 353 00:11:41,280 --> 00:11:41,760 Okay. 354 00:11:41,760 --> 00:11:45,600 At least for anyone who's ever felt like they were stuck in a meaningless job. 355 00:11:45,600 --> 00:11:46,240 Oh. 356 00:11:46,240 --> 00:11:50,480 They decided to look at the language of what's called bullshit jobs. 357 00:11:50,480 --> 00:11:51,360 Bullshit jobs. 358 00:11:51,360 --> 00:11:51,760 Yeah. 359 00:11:51,760 --> 00:11:53,440 Is that like a technical term? 360 00:11:53,440 --> 00:11:54,000 It is. 361 00:11:54,000 --> 00:11:56,480 It actually comes from the work of an anthropologist. 362 00:11:56,480 --> 00:11:57,120 Oh, okay. 363 00:11:57,120 --> 00:11:59,920 David Graber, who wrote a whole book about this. 364 00:11:59,920 --> 00:12:00,320 Really? 365 00:12:00,320 --> 00:12:07,200 Yeah, he defines bullshit jobs as jobs that are so pointless, unnecessary, or even harmful. 366 00:12:07,200 --> 00:12:07,840 Oh, wow. 367 00:12:07,840 --> 00:12:11,360 That even the people doing them can't justify their existence. 368 00:12:11,360 --> 00:12:13,600 Oh, I think we've all had one of those at some point. 369 00:12:13,600 --> 00:12:15,040 I think you're probably right. 370 00:12:15,040 --> 00:12:16,640 What kind of jobs are we talking about here? 371 00:12:16,640 --> 00:12:23,280 Well, he breaks it down into different categories, like flunkies, goons, duck tapers, box tickers, 372 00:12:23,280 --> 00:12:24,480 and taskmasters. 373 00:12:24,480 --> 00:12:24,880 Okay. 374 00:12:24,880 --> 00:12:28,480 Basically, jobs that feel like they're just creating work for the sake of work 375 00:12:28,480 --> 00:12:31,440 without actually contributing anything meaningful to society. 376 00:12:31,440 --> 00:12:31,840 Okay. 377 00:12:31,840 --> 00:12:33,600 That's a pretty wide range of jobs. 378 00:12:33,600 --> 00:12:34,160 It is. 379 00:12:34,160 --> 00:12:37,360 But how do they analyze the language of these bullshit jobs? 380 00:12:37,360 --> 00:12:37,520 Okay. 381 00:12:37,520 --> 00:12:42,720 So they collected a bunch of text samples from online sources, things like job descriptions, 382 00:12:42,720 --> 00:12:45,440 company websites, and even social media posts. 383 00:12:45,440 --> 00:12:46,000 Wow. 384 00:12:46,000 --> 00:12:52,240 And then they compared those samples to text from jobs that are generally considered to have 385 00:12:52,240 --> 00:12:56,480 clear social value, like teachers, doctors, nurses, that kind of thing. 386 00:12:56,480 --> 00:12:59,040 So it's bullshit jobs versus essential jobs. 387 00:12:59,040 --> 00:13:01,120 Yeah, it's like a linguistic cage match. 388 00:13:01,120 --> 00:13:01,760 I love it. 389 00:13:01,760 --> 00:13:02,640 What was the outcome? 390 00:13:02,640 --> 00:13:03,280 What happened? 391 00:13:03,280 --> 00:13:05,680 You guessed it, the BS detector went off again. 392 00:13:05,680 --> 00:13:06,320 No way. 393 00:13:06,320 --> 00:13:12,400 The text from the bullshit jobs had a significantly higher BS score than the text from the essential 394 00:13:12,400 --> 00:13:12,880 jobs. 395 00:13:12,880 --> 00:13:18,720 So even the language we use to describe our jobs can reveal whether or not they're full of BS. 396 00:13:18,720 --> 00:13:19,840 It seems that way. 397 00:13:19,840 --> 00:13:20,880 That's kind of mind blowing. 398 00:13:20,880 --> 00:13:21,760 It is pretty wild. 399 00:13:21,760 --> 00:13:23,200 But this is all getting a bit heavy. 400 00:13:23,200 --> 00:13:23,600 Yeah. 401 00:13:23,600 --> 00:13:25,600 Can we bring it back to the AI stuff for a sec? 402 00:13:25,600 --> 00:13:25,680 Can we? 403 00:13:25,680 --> 00:13:32,080 What does this all mean for the future of chatbots and language models? 404 00:13:33,840 --> 00:13:34,080 All right. 405 00:13:34,080 --> 00:13:40,640 So we've gone from these fake scientific papers to political manifestos to the world of work. 406 00:13:40,640 --> 00:13:42,240 Look, that's quite the journey. 407 00:13:42,240 --> 00:13:47,680 And it seems like this BS detector is picking up a signal just about everywhere. 408 00:13:47,680 --> 00:13:48,560 It really is. 409 00:13:48,560 --> 00:13:53,280 But what does this all mean for the everyday people using these AI tools? 410 00:13:53,280 --> 00:13:53,760 Right. 411 00:13:53,760 --> 00:13:54,720 That's the big question. 412 00:13:54,720 --> 00:13:55,200 Yeah. 413 00:13:55,200 --> 00:13:58,320 And I think this research is a real wake up call for all of us. 414 00:13:58,320 --> 00:13:58,880 OK. 415 00:13:58,880 --> 00:14:02,880 It shows us that AI can be incredibly good at mimicking human language. 416 00:14:02,880 --> 00:14:03,360 Uh-huh. 417 00:14:03,360 --> 00:14:06,000 But that doesn't necessarily mean it understands what it's saying. 418 00:14:06,000 --> 00:14:07,840 So it's all style over substance. 419 00:14:07,840 --> 00:14:08,960 Kind of, yeah. 420 00:14:08,960 --> 00:14:11,520 Like it can talk the cock, but it can't walk the walk. 421 00:14:11,520 --> 00:14:12,400 Exactly. 422 00:14:12,400 --> 00:14:16,480 And that's why it's so important for us to be critical thinkers when we're using AI, 423 00:14:16,480 --> 00:14:18,000 you know, when we're interacting with it. 424 00:14:18,000 --> 00:14:18,400 Right. 425 00:14:18,400 --> 00:14:24,720 Just because something sounds really convincing or even authoritative doesn't mean it's true. 426 00:14:24,720 --> 00:14:27,840 So we can't just take what these chatbots tell us at face value. 427 00:14:27,840 --> 00:14:28,640 Yeah, no, no. 428 00:14:28,640 --> 00:14:29,600 We have to be skeptical. 429 00:14:29,600 --> 00:14:30,080 Yeah. 430 00:14:30,080 --> 00:14:31,280 Do our own research. 431 00:14:31,280 --> 00:14:31,680 Yeah. 432 00:14:31,680 --> 00:14:37,360 And just be aware of the potential for bias, just like we do with any other source of information. 433 00:14:37,360 --> 00:14:39,120 Exactly. You hit the nail on the head. 434 00:14:39,120 --> 00:14:43,280 OK. So how can we sharpen our BS detection skills? 435 00:14:43,280 --> 00:14:46,400 Well, first of all, just be aware of the limitations of AI. 436 00:14:46,400 --> 00:14:46,800 OK. 437 00:14:46,800 --> 00:14:51,440 Remember, these language models, they're trained on these massive amounts of data, 438 00:14:51,440 --> 00:14:54,160 but they don't actually understand the world the way that we do. 439 00:14:54,160 --> 00:14:55,680 Right. They're just recognizing patterns. 440 00:14:55,680 --> 00:15:00,400 Exactly. So don't be afraid to question what you see, what you hear. 441 00:15:00,400 --> 00:15:00,880 Yeah. 442 00:15:00,880 --> 00:15:02,880 If something seems too good to be true. 443 00:15:02,880 --> 00:15:03,600 Uh-huh. 444 00:15:03,600 --> 00:15:05,680 Or it just doesn't sit right with you. 445 00:15:05,680 --> 00:15:06,000 OK. 446 00:15:06,000 --> 00:15:07,520 Dig a little deeper. 447 00:15:07,520 --> 00:15:07,680 Yeah. 448 00:15:07,680 --> 00:15:09,040 Check those sources. 449 00:15:09,040 --> 00:15:09,440 OK. 450 00:15:09,440 --> 00:15:11,760 Look for alternative perspectives. 451 00:15:11,760 --> 00:15:14,720 And don't be afraid to challenge the information that you're given. 452 00:15:14,720 --> 00:15:17,440 That's good advice for life in general. 453 00:15:18,080 --> 00:15:19,600 You know what I agree with you there? 454 00:15:19,600 --> 00:15:21,360 Not just when we're dealing with AI. 455 00:15:21,360 --> 00:15:22,240 Absolutely. 456 00:15:22,240 --> 00:15:27,120 And I think this research also has some implications for how we design and develop 457 00:15:27,120 --> 00:15:29,120 AI systems going forward. 458 00:15:29,120 --> 00:15:29,520 OK. 459 00:15:29,520 --> 00:15:33,760 If we want AI to be a tool for truth and understanding. 460 00:15:33,760 --> 00:15:34,000 Right. 461 00:15:34,000 --> 00:15:37,920 We have to be very careful about the incentives that we build into these systems. 462 00:15:37,920 --> 00:15:42,000 Right. So we need to make sure that we're not rewarding AI for being a good bullshitter. 463 00:15:42,000 --> 00:15:45,840 Exactly. We need to prioritize accuracy and transparency. 464 00:15:45,840 --> 00:15:46,240 Yeah. 465 00:15:46,240 --> 00:15:48,720 Over fluency and persuasiveness. 466 00:15:48,720 --> 00:15:49,520 OK. I got it. 467 00:15:49,520 --> 00:15:54,640 Otherwise, we really risk creating a world where BS reigns supreme. 468 00:15:54,640 --> 00:15:56,720 Yeah. That is a scary thought. 469 00:15:56,720 --> 00:15:58,400 It is a little bit unsettling. 470 00:15:58,400 --> 00:16:01,680 Well, I think this deep drive has given us some really valuable tools for 471 00:16:01,680 --> 00:16:04,720 navigating this increasingly complex world of AI. 472 00:16:04,720 --> 00:16:05,600 I think so too. 473 00:16:05,600 --> 00:16:10,720 It's all about being informed and being critical and maybe having a little fun with it along the way. 474 00:16:10,720 --> 00:16:16,080 Exactly. And you know, if AI can be so good at mimicking human BS. 475 00:16:16,080 --> 00:16:16,560 Yeah. 476 00:16:16,560 --> 00:16:18,640 Maybe that says something about us too. 477 00:16:18,640 --> 00:16:20,400 That's a great point. Something to think about. 478 00:16:20,400 --> 00:16:21,520 Something to ponder. 479 00:16:21,520 --> 00:16:22,880 As we go about our day. 480 00:16:22,880 --> 00:16:27,680 Well, thanks for joining us for this deep dive into the world of AI and BS. 481 00:16:27,680 --> 00:16:28,480 My pleasure. 482 00:16:28,480 --> 00:16:32,480 We'll catch you next time.