Speaker 1
00:00:00,000 --> 00:02:33,680
You're listening to a Bellingcat Discord server stage talk titled The Secretive World of AI Pornography. This week we were joined by Bellingcat researcher Kolina Koltaj. She is an expert in how socio-technical systems influence the decision-making of social groups with a PhD from the University of Texas School of Information. She has also previously worked at the Center for an Informed Public at the University of Washington. Now at Bellingcat, her focus is into online ideologies and her latest research has been into the phenomenon of AI girlfriends and AI sexual content. In the Discord stage talk, Kolina spoke about why she started looking into this area and why she feels so passionately about doing more research, slash speaking for regulation on this topic. A small content warning. In the discussion we reference child sexual abuse material and the death of a child, as well as revenge porn. You can find links to Kolina's research in the description of this podcast. The talk was hosted by me, Charlotte Maher, on Thursday the 1st of February in the Bellingcat Discord server. 

Welcome to this week's Bellingcat Discord stage talk. Non-consensual sexual deep fakes have been in the headlines this week. As faked, explicit images of Taylor Swift went viral. For many, this type of exploitative content was a new concept, but for Bellingcat researcher Kolina Koltai, who joins us today, it is a trend she's been following for a while. Kolina joins us today to discuss the AI porn industry and the platform set up to profit Any Dream. In late November, Kolina published research on a US artificial intelligence company that was collecting money for a service that can create non-consensual pornographic deep fakes using financial services company Stripe, which bans processing payments for adult material.
After joining the Discord community, may I note, fostered by the company for creation of these images, she discovered references to child sexual material being created and distributed, and non-consensual generated images of celebrities and real women being shared. This kind of generated material has sadly recently led to the death of a young girl in London, which we will also speak to during the course of this talk. As Kolina talks, please put all of your questions in the chat. I'll ask them as we move into the Q&A section. So please put them in the chat and I'll be moderating that. Thanks so much for coming, Kolina. The stage is yours.

Speaker 2
00:02:33,680 --> 00:02:43,760
Thanks so much for having me here, Charley. I am excited, but it feels like sometimes a weird word to say when you're talking about a icky topic that you're excited to talk about the topic.

Speaker 2
00:02:43,760 --> 00:03:24,720
But this is obviously a really important topic that's happening today and that I am thrilled that it is getting so much more attention because it's something that's been awful,
awful stuff happening, occurring at sometimes different corners of the internet. And we like to think they're in these like dark, shady quarters where you have to go on the dark web. But no, you can go in their websites. You could access easily where you can create this content and find this content. So let me start from the beginning before I kind of just derail myself because there is so much to cover and talk about that I feel like I can just go a million miles a minute talking about it. But I hope everyone is having a good morning, good evening or good afternoon,

Speaker 2
00:03:24,720 --> 00:03:45,840
depending on where you are. And I want to talk a little bit first about some of the initial research we did and we published on in November, which we were looking at a company that we found to be based here in the US where I am, but actually have like incorporation in the UK.
And obviously with a lot of stuff that you see on the internet, there's some really clear borders.

Speaker 2
00:03:46,960 --> 00:04:13,280
At first, we kind of started looking into this topic of AI girlfriends. Actually, on like a little bit of a whim. I was scrolling Twitter and someone said they found a popular influencer and that she had not disclosed she was actually using AI images. But I was like, oh, this is kind of interesting. I wonder how many people on Instagram are using AI images of themselves or other women pretending to be someone that they're not. And I started digging into this topic. And Charley's

00:04:13,280 --> 00:04:18,720
there too, because I actually talked to Charley a little bit about this topic last year. I started

47
00:04:18,720 --> 00:04:24,240
getting questions like, man, how are all these people creating these images? There are, you know,

48
00:04:24,240 --> 00:04:29,440
dozens of hundreds, sometimes thousands if you think about it, people who are using not just

49
00:04:29,440 --> 00:04:36,800
a sort of shop, but like using AI technology to create new images, sometimes edited versions of

50
00:04:36,800 --> 00:04:43,440
themselves or completely new women. And the seemingly the barrier to entry and creating this

51
00:04:43,440 --> 00:04:48,720
content was drastically dropping at a speed that we had anticipated, I think, it comes to like

52
00:04:48,720 --> 00:04:55,600
visual media. So I started looking into the tech component. And very quickly, a company's name kind

53
00:04:55,600 --> 00:05:00,960
of started consistently popping up, which was one called AnyDream. I went out and checked out their

54
00:05:00,960 --> 00:05:08,560
site and found that they had a Discord community. And also what they still have, and it's still up

55
00:05:08,560 --> 00:05:12,080
unfortunately, is they have something called a Discover section on their website in which

56
00:05:12,640 --> 00:05:19,600
you can go there and view all the images people are creating and publicly share. So people are

57
00:05:19,600 --> 00:05:25,280
electing to share these images publicly on AnyDream's website, including their prompts. So

58
00:05:25,280 --> 00:05:30,000
I think the goal of the developers, people who own this, would be like, oh, you can see the images

59
00:05:30,000 --> 00:05:34,800
and what kind of like words and terms people are using to generate the images, but then use that

60
00:05:34,800 --> 00:05:38,000
to build off it. Or you see this image, you're like, oh, I'm going to take this image, but then

61
00:05:38,000 --> 00:05:43,040
do a little bit of something different with it. So this can work really great in a community

62
00:05:44,000 --> 00:05:51,440
that is not necessarily steeped in non-consensual sexual imagery. If you're like, oh, you know,

63
00:05:51,440 --> 00:05:56,160
I want to make images around like making these animals or something like that. But when we come

64
00:05:56,160 --> 00:06:02,160
to the idea that you're making images of women who are not just made up women, not completely AI

65
00:06:02,160 --> 00:06:07,920
women, but women that are actually based within real people. And one of the particularly unique

66
00:06:07,920 --> 00:06:14,960
features of AnyDream is some other components that you can upload the face of anyone. So like,

67
00:06:14,960 --> 00:06:22,080
you can upload like our founders, Ali Hinkins face, or you can upload the Taylor Smith face.

68
00:06:22,080 --> 00:06:30,160
You can upload the face of your neighbors or anyone that you know, a celebrity, real, whoever,

69
00:06:30,160 --> 00:06:36,880
and then generates images of them and sexual images of them, generate photography, and it

70
00:06:36,880 --> 00:06:41,120
worked really well. And that's the thing. Like there is, as with any sort of AI image technology,

71
00:06:41,120 --> 00:06:46,080
sometimes a little bit of a play on it, but actually worked pretty well. So immediately,

72
00:06:46,640 --> 00:06:50,320
I saw this like, do we got to focus on this company? And while they were trying really hard,

73
00:06:50,320 --> 00:06:54,480
I think, to be secretive, I think people, they, people get conky. They don't realize

74
00:06:55,200 --> 00:06:59,520
how much information they're sharing about themselves so that people care about this,

75
00:06:59,520 --> 00:07:05,600
right? There hasn't been very much scrutiny on the companies and the communities that are

76
00:07:05,600 --> 00:07:11,120
focused on this. Like I'm creating it. I have joined plenty of Discord communities that are

77
00:07:11,120 --> 00:07:16,000
generating this content for different companies, and that's just one of them. And people are really

78
00:07:16,000 --> 00:07:21,760
bold. They try using a pseudonym, but with a little bit about some OZIP skills, you can usually

79
00:07:21,760 --> 00:07:26,880
figure out who's behind certain pseudonym because people like to use it consistently. So one other

80
00:07:26,880 --> 00:07:31,840
company, a much smaller one that I was looking into, and I'm actively monitoring their Discords,

81
00:07:31,840 --> 00:07:37,760
their Discords, they, you know, this guy's, what are their devs slash their owner uses the same

82
00:07:37,760 --> 00:07:43,600
username across all his profiles, which made it very easy that defined like his GitHub, his name,

83
00:07:43,600 --> 00:07:47,680
is like, you know, if you use the same profile image and username across it, it becomes very easy.

84
00:07:48,400 --> 00:07:54,800
But it's, I'm derailing. So what we decided to look at is not just obviously look at this community

85
00:07:54,800 --> 00:08:01,760
in which we found, you know, not just like the sharing of like AI generated non-custential imagery,

86
00:08:01,760 --> 00:08:05,520
but like I think the act of like sort of encouragement of it. They had channels completely

87
00:08:05,520 --> 00:08:12,160
dedicated to sharing those images and seemingly really no pushback when it came to showing images

88
00:08:12,160 --> 00:08:18,400
of celebrities, of real people. You would have users very clearly say, hey, I made nudes of my

89
00:08:18,400 --> 00:08:25,120
wife's frag, here they are. I've made nudes of this person. But there is, from the community,

90
00:08:25,120 --> 00:08:31,040
they did not seem to really improve CSAM, which is good, but there is so little work. Like whenever

91
00:08:31,040 --> 00:08:37,120
someone would question the devs or the owners about saying, hey, you know, hey, keep finding

92
00:08:37,120 --> 00:08:41,520
CSAM. Hey, there's really disgusting stuff here. Hey, do something about it. And he really was doing

93
00:08:41,520 --> 00:08:45,680
the bare minimum and kind of was like, oh, I'm just going to leave it for someone else's problem. I'm

94
00:08:45,680 --> 00:08:54,720
not focused on that. So it is, it was really, it still is about that small company. After we published

95
00:08:54,720 --> 00:09:00,560
the story, because we found that they were hiding their payments. So one thing that they were doing

96
00:09:01,200 --> 00:09:06,800
to disguise their payments from Stripe was they would use a redirect. So if you click to say, oh,

97
00:09:06,800 --> 00:09:12,960
I'm going to pay another sort of site, like a loading window would come up very briefly. And

98
00:09:12,960 --> 00:09:17,760
if you will look at the address bar, it would change the name of that website. And then it would

99
00:09:17,760 --> 00:09:22,240
say like Stripe's payments and the sales screen. And you would see that it wasn't being processed

100
00:09:22,240 --> 00:09:27,440
by imagery. It was being crossed payment from some other site. And not only does this give protection

101
00:09:27,440 --> 00:09:33,760
to people who are purchasing it, because it doesn't show up like you're buying tokens or coins to use

102
00:09:33,760 --> 00:09:40,800
a AI generation service. But it also hides some Stripe who had no idea. Because right after,

103
00:09:40,800 --> 00:09:46,960
when we started contacting Stripe for comments about it, immediately, like within the hour after

104
00:09:46,960 --> 00:09:51,840
we contacted Stripe, their payment system got shut down. So Stripe took it very, very seriously,

105
00:09:51,840 --> 00:09:57,120
which is great on Stripe. And later after the fact, one of my friends contacted me about it.

106
00:09:57,120 --> 00:10:01,840
And she was like, hey, I didn't know at the time, but like her husband, I was working at Stripe,

107
00:10:01,840 --> 00:10:06,080
or still is working at Stripe. And she was like, I said, I could have like got that shut down for

108
00:10:06,080 --> 00:10:12,240
you just let me know. They take it very seriously. So good on Stripe for taking it seriously. But

109
00:10:12,240 --> 00:10:16,800
of course, there's no way for them to know that like going in and investigating that these are

110
00:10:18,160 --> 00:10:23,120
using a redirect to match this payment, which can be a really common way to match payments from

111
00:10:23,120 --> 00:10:26,320
these kind of providers. We just set up another website and it seems like payments are going to

112
00:10:26,320 --> 00:10:31,760
that website. They also were very bold to be like, hey, you can just PayPal me. Here's my like personal

113
00:10:31,760 --> 00:10:38,640
PayPal address. And it took us a while, but we did eventually get PayPal to also agree to like shut

114
00:10:38,640 --> 00:10:44,240
down those accounts. There is definitely a variance in response time from some of these financial

115
00:10:44,240 --> 00:10:49,840
companies for financial payment processors, even though this very clearly breaks the rules.

116
00:10:50,640 --> 00:10:56,320
And also, for example, there are the websites being hosted through Squarespace. Be fantastic,

117
00:10:56,320 --> 00:11:02,080
Squarespace. Hey, this violates your policy on adult content. There is very clearly adult content

118
00:11:02,080 --> 00:11:07,040
being hosted on Squarespace. Websites that love Squarespace haven't done anything. And so

119
00:11:07,760 --> 00:11:12,240
when we talk about it, think about the public pressure that's happening right now.

120
00:11:12,240 --> 00:11:17,760
You know, there's talk about new legislation. There's public outcry. And I'm like, this is

121
00:11:17,760 --> 00:11:24,080
great because me sitting here at my keyboard and telling y'all is just part of that process. It is

122
00:11:24,080 --> 00:11:30,160
getting upset at people who are in positions to actually do something. Be reporting on it.

123
00:11:30,160 --> 00:11:34,800
But they on belly cat writing stories is how we're going to actually, I think, institute some change

124
00:11:34,800 --> 00:11:41,120
when it comes to putting pressure on systems like the payment processor, because we could address

125
00:11:41,120 --> 00:11:48,560
the issue of payment. You know, it really incentivize people. Now it's always possible to go to crypto,

126
00:11:48,560 --> 00:11:52,320
but a lot of people still don't want to necessarily want to do the crypto. It's much easier to just say

127
00:11:52,320 --> 00:11:57,840
take a credit card, use PayPal, use one of these mainstream payment providers. The other thing is

128
00:11:57,840 --> 00:12:02,000
we can push them off to other parts of the internet. So it's not just like an easily found

129
00:12:02,000 --> 00:12:08,880
thing on, you know, social media platforms on through Google search. You know, these services,

130
00:12:09,600 --> 00:12:14,800
and a dream and other ones are being used to generate hundreds, thousands and sort of like

131
00:12:14,800 --> 00:12:23,360
almost incomprehensible amounts of non-consensual delimitry. And most arguably, I doubt that the,

132
00:12:24,320 --> 00:12:29,280
I can guarantee you I'd be willing to bet money that it's overwhelmingly women who are being

133
00:12:29,280 --> 00:12:34,800
targets. And I think that's really, unfortunately, not everyone who is above the age of 18 or targets.

134
00:12:34,800 --> 00:12:39,120
And I think that is the really tricky part, because even when I go to some of the telegram

135
00:12:39,120 --> 00:12:45,920
groups that are directed to it, these groups are huge. Some of them, like the smaller ones,

136
00:12:46,480 --> 00:12:51,200
as for services that are paying for this on the small end, have like 10, 15,000 people in it.

137
00:12:51,200 --> 00:12:56,800
On the larger end, the largest group I've seen on Telegram has 800,000 people on the singular channel

138
00:12:56,800 --> 00:13:01,120
dedicated to the creation of this to pay for it. 800,000 people, that's more Twitter followers,

139
00:13:01,120 --> 00:13:06,160
Twitter followers than, but I guess like one of the largest telegram groups I think I've ever seen.

140
00:13:06,160 --> 00:13:14,320
It is huge. And so this is a problem, I think that, you know, people will say like, oh, we've had

141
00:13:14,320 --> 00:13:20,080
Photoshop for a really long time. How does this change anything? Just sometimes people will go

142
00:13:20,080 --> 00:13:24,720
and talk about like personal. Yes, we've had Photoshop. But think about the e-photoshop

143
00:13:24,720 --> 00:13:28,640
expensive, you can of course, they don't hire copyers, other sort of knockoffs out there,

144
00:13:28,640 --> 00:13:34,960
but the barrier to entry are creating a convincing bit of non-consensual intimate

145
00:13:34,960 --> 00:13:43,920
media was much higher five years ago. You know, in the past like year, you know, 12, 18 months,

146
00:13:43,920 --> 00:13:50,880
things have changed dramatically. And that barrier to entry, the barrier and the ease of it, right?

147
00:13:50,880 --> 00:13:56,720
You don't even need to be that sophisticated and creating AI imagery, right? You know, you can go

148
00:13:56,720 --> 00:14:05,600
and use something like mid-journey or any of the other platforms, sample diffusion, and sort of

149
00:14:05,600 --> 00:14:09,600
just play around with it and create it. You can even go into Bing, you know, even my father, who's

150
00:14:09,600 --> 00:14:14,640
80, could probably figure out how to generate a basic image. And so really think about it, it

151
00:14:14,640 --> 00:14:21,520
becomes incredibly easy to create these kinds of content and then people figure out how to get

152
00:14:21,520 --> 00:14:27,440
around it. People have not figured out the great way to go around modifying or, you know, really

153
00:14:27,440 --> 00:14:32,560
about how to moderate these spaces well, because, you know, you know that you can write a name like

154
00:14:32,560 --> 00:14:37,760
Taylor Swift, and if you want to ban the word Taylor Swift, you can put a singer that looks like

155
00:14:37,760 --> 00:14:42,080
Taylor Swift, and sometimes that will work. Or you misspell Taylor Swift slightly and the system

156
00:14:42,080 --> 00:14:45,920
goes, oh, I think you're trying to misspell Taylor Swift and it still works. Or you put in the word

157
00:14:45,920 --> 00:14:50,080
like singer and between it and break it up into your bit, right? Or if they're really lazy, you

158
00:14:50,080 --> 00:14:56,480
can do lowercase so that it works. Like it is, there's so, people are a, they haven't thought

159
00:14:56,480 --> 00:15:00,160
enough about moderation, two are lazy, and three people are really smart to figure around

160
00:15:00,160 --> 00:15:03,760
moderation. Some of my other previous work was looked at how communities got around moderating

161
00:15:03,760 --> 00:15:10,720
systems, and humans are really, really smart at figuring it out. So it's something that we need to

162
00:15:10,720 --> 00:15:14,720
be really proactive. We can't be just reactive. Like right now we're reacting to this image being

163
00:15:14,720 --> 00:15:18,880
generated like that's out there, but I can tell you that those images about Taylor Swift,

164
00:15:18,880 --> 00:15:24,400
I mean, not those specific ones, but AI non-consensual intimate imagery of Taylor Swift

165
00:15:24,400 --> 00:15:31,600
and thousands of other women are being generated daily and have been for months, years. I think

166
00:15:31,600 --> 00:15:39,280
it is what I think is also a little disheartening that it takes like a giant global pop superstar

167
00:15:39,280 --> 00:15:45,440
to do it, but I am glad in a sense because if something actually happens, it's going to be

168
00:15:45,440 --> 00:15:49,440
really frustrating if nothing does happen. And I think we need something on a global scale.

169
00:15:49,440 --> 00:15:55,040
We talked about legislation as who is responsible. It's not just supposed to be just on one country.

170
00:15:55,760 --> 00:15:59,760
I think there does need to be pressure on obviously the services that are doing this,

171
00:15:59,760 --> 00:16:03,200
payment processors who are allowing it to kind of continue on to accept payment,

172
00:16:04,080 --> 00:16:09,920
people who are sharing, disseminating that content. There needs to be restrictions in a similar way

173
00:16:09,920 --> 00:16:17,120
we think about CSAM, which is child sex abuse material. We are globally like most countries

174
00:16:17,120 --> 00:16:23,040
have really strong rules and regulations around it. We are still behind on AI CSAM. We are definitely

175
00:16:23,040 --> 00:16:30,320
behind on deep fake content and non-consensual deep fake content because our legislation is slow.

176
00:16:30,320 --> 00:16:36,560
Things need to happen. So legislation is slow everywhere. We know technology on pace and I feel

177
00:16:36,560 --> 00:16:43,120
like I'm going on and on about this. Let's see what else about this? I think there are as far as

178
00:16:45,120 --> 00:16:50,640
where these spaces are, it is literally I think it doesn't take that long to find it. There are

179
00:16:50,640 --> 00:16:55,760
forums dedicated to it. You can find discords dedicated to it. You can find it on Twitter.

180
00:16:57,920 --> 00:17:03,440
I was just searching, there's a recent report from Graphica I think. I might have to pull up the link.

181
00:17:03,440 --> 00:17:09,520
The Graphica did a recent short report on referral links and sort of the rise of that space.

182
00:17:09,520 --> 00:17:17,440
I think they saw like it just in 2023, so last year, something like a 2600,000% increase in

183
00:17:17,440 --> 00:17:24,480
referral links for this kind of content just on Reddit and Twitter, which is like skyrocketing high.

184
00:17:25,840 --> 00:17:29,840
I think people are also really just unabashed with the people who are creating this. It's like

185
00:17:29,840 --> 00:17:35,600
people don't care. I kept following the anydream community space after they moved to Telegram

186
00:17:35,600 --> 00:17:42,000
and I kind of lurked in their Telegram. They felt like they were being discriminated,

187
00:17:42,000 --> 00:17:47,120
like literally they said we're being discriminated against because we got kicked off Stripe and we

188
00:17:47,120 --> 00:17:51,840
got kicked off Discord. They don't understand us. We're not doing anything wrong. We just have to

189
00:17:51,840 --> 00:17:57,040
stick to Cripo for now because people don't understand what we're doing. And remember,

190
00:17:57,040 --> 00:18:00,080
this is the community of people who are generating this content. And some people are like,

191
00:18:00,080 --> 00:18:03,680
oh, I don't understand what my Discord account, like I wasn't the one creating the celebrity

192
00:18:03,680 --> 00:18:10,080
content, but you're in there. Like some of the mods were upset and it's like the community people

193
00:18:10,080 --> 00:18:17,680
and they are really unabashed. And these are people from sort of all walks of life. There are

194
00:18:17,680 --> 00:18:23,200
people all over the globe. I don't think that many people who are really actively in that space feel

195
00:18:23,200 --> 00:18:29,440
guilty about it because that's what they're saying in those spaces. So I think it is a,

196
00:18:29,440 --> 00:18:38,640
I don't know how many, how pervasive this ideology is becoming, but it is overwhelmingly women who

197
00:18:38,640 --> 00:18:46,640
are victims of this and overwhelmingly the men who are in those spaces don't see it as like an issue.

198
00:18:46,640 --> 00:18:54,000
I will say there is sort of a room and space. There are say women who use AI technology

199
00:18:54,000 --> 00:18:59,120
because they themselves are consenting to it. They're making AI images of themselves and that's a

200
00:18:59,120 --> 00:19:06,720
very different thing, right? There are, that's the consensual commit for overwhelmingly this kind of

201
00:19:06,720 --> 00:19:12,000
content and this type of technology, it is non-consensual. Women are not consenting to it

202
00:19:12,000 --> 00:19:18,800
and overwhelmingly targets and oftentimes the onerous is on the woman. So like for the

203
00:19:18,800 --> 00:19:24,720
Eddie Dre example, when we were questioning them about like the kind of content and the non-consensual

204
00:19:24,720 --> 00:19:29,040
imagery of like the women who are real women and people were using their names and I was able to

205
00:19:29,040 --> 00:19:35,120
Google them and find them incredibly easily on the site. And they're like, it is then up to that

206
00:19:35,120 --> 00:19:42,480
woman to A, find out that images of her being created and then contact some brand and developer,

207
00:19:42,480 --> 00:19:46,800
some brand guys running a website being like, hey, there's all these images of me that I didn't agree

208
00:19:46,800 --> 00:19:51,840
to. And then hopefully those images get taken down versus like this guy who's making, still making

209
00:19:51,840 --> 00:20:00,400
those images, posting the images of this woman on Twitter, on Reddit, on everywhere. And it's on her,

210
00:20:00,400 --> 00:20:08,160
it's on the woman to ask places to stop. And that's unfair. I would hate like, imagine how hard

211
00:20:08,160 --> 00:20:11,440
that would be to go on like the internet and suddenly see fake nudes being made of you. Like

212
00:20:11,440 --> 00:20:17,280
you read the stories, like there's this case in Spain that made huge news in September last year

213
00:20:17,280 --> 00:20:22,640
because they were using one of these AI services and it was high school boys who were targeting

214
00:20:22,640 --> 00:20:29,440
their female classmates. So like even though the boys themselves ranged from ages 12, 14,

215
00:20:29,440 --> 00:20:36,000
the girls in question were aged range from 11 to 17. So you're all minors involved. And so,

216
00:20:36,000 --> 00:20:41,120
you know, I think the, assuming these boys, you can't just say boys will be boys because they

217
00:20:41,120 --> 00:20:45,040
thought, so they really thought there was nothing wrong with creating a group in which they made

218
00:20:45,040 --> 00:20:50,800
these images of their classmates, share those images. And then even some of the kids tried to

219
00:20:50,800 --> 00:20:56,720
extort their high school classmates saying like, I have these fake images of you, send me money,

220
00:20:56,720 --> 00:21:01,200
send me other fake nudes of you or send me other news of you, otherwise I better share these around.

221
00:21:01,200 --> 00:21:09,360
And huge outcry. Women are minors, girls are at the core of this, that company still up and going,

222
00:21:09,360 --> 00:21:18,640
no consequence. And I think there's, things need to be done. I don't know, maybe I'll leave it there.

223
00:21:18,640 --> 00:21:22,320
That's probably a terrible story to end it on, but like I think that's made such huge news and

224
00:21:22,320 --> 00:21:28,640
absolutely nothing happened except, you know, a bunch of girls being victims. And, you know,

225
00:21:28,640 --> 00:21:33,040
we can talk a little bit about ideology, ideology, what makes a teenage woman to do that, but I think

226
00:21:33,040 --> 00:21:37,280
we're making it okay. There's no consequence besides the slap on the wrist. There's no

227
00:21:37,280 --> 00:21:40,880
consequence of the companies that are happening. Something needs to change.

Speaker 1
00:21:40,880 --> 00:22:48,640
Thanks, Kolina. Thanks for explaining that. Maybe one of the mods could put your piece of research in the chat. I think that would be really useful, if you're out there. Thank you so much. You can feel your passion for this topic coming through the screen. And I think it's a topic that all of us should be highly passionate about. We've had some really interesting questions while you've been talking, coming into the chat, including discussions on what we can do about this problem.
But I first wanted to start with the Simian's question, because I think it relates to a lot of what you were saying. They were mentioning that in film class, they were taught that before releasing any media of any people who start in projects, they have to get them to sign media consent and release forms. How come AI generated pornographic content isn't in the same area?
Is it possible nowadays to kind of own your own image in this kind of social media world? And is this kind of consent possible in terms of this kind of AI generated pornography?

Speaker 2
00:22:49,680 --> 00:22:54,320
We see people share images of other people all the time without their consent.

240
00:22:55,040 --> 00:22:59,920
Gosh, I remember even when this is like Percy's work, but even on my Instagram, I remember I shared

241
00:22:59,920 --> 00:23:04,320
a photo of me from a Halloween costume. So my face is covered, but I remember I posted this on

242
00:23:04,320 --> 00:23:09,040
Instagram. And someone who followed me took my photo and reposted it as if it was her,

243
00:23:09,040 --> 00:23:13,280
without attributions. It's like that was her Halloween costume. And I was like, what?

244
00:23:14,480 --> 00:23:18,880
Who does that? Why is this woman who I knew reposted my photo thinking I wouldn't see it?

245
00:23:18,880 --> 00:23:24,960
And I didn't say anything at the time. But I think there's obviously been an issue with just

246
00:23:24,960 --> 00:23:29,440
culturally on the internet of lack of attribution. I think we're as, you know,

247
00:23:30,640 --> 00:23:35,440
cultural practices were getting better about that. But I think the difference I think is that with

248
00:23:35,440 --> 00:23:40,960
film is you do have a consent form there, because you're actively aware that someone generally is

249
00:23:40,960 --> 00:23:45,440
filming you. You know, when you, with the exception of general somebody's those reality shows where

250
00:23:45,440 --> 00:23:48,960
they like to film you and they're hidden, and then they ask for consent forms later on. If someone

251
00:23:48,960 --> 00:23:53,360
doesn't sign it, then they blur it. There is something like that kind of like active act happening

252
00:23:53,920 --> 00:23:58,720
versus you post something on the internet. Anyone can then like, you know, screenshot it, download

253
00:23:58,720 --> 00:24:02,240
it, carry it, and then post it. And you just don't know. And I do think we need to get better about

254
00:24:02,240 --> 00:24:08,000
that attribution because you know, this that work goes beyond just, you know, obviously, the AI space,

255
00:24:08,000 --> 00:24:13,840
it goes beyond it goes into thinking about like, just photographs that people have copyrights that

256
00:24:13,840 --> 00:24:20,880
people own. I have I did see one AI porn site that did try to be a little better. I don't remember

257
00:24:20,880 --> 00:24:25,040
which one it was, but they did have this thing that like if you wanted to upload a face, you could

258
00:24:25,040 --> 00:24:29,680
only do your own face and you had to like show like proof of like ID or something. And I was like,

259
00:24:29,680 --> 00:24:34,080
oh, that's kind of interesting. I haven't seen anyone else do that. So therefore, you're

260
00:24:34,080 --> 00:24:37,680
saying because you prove that this is my face and they were like very stringent, like we have the

261
00:24:37,680 --> 00:24:43,280
technology to do it, but we only want people who prove it's them to do it with just one way of doing

262
00:24:43,280 --> 00:24:48,880
it when it comes to a generation component if you're presented your own imagery. But I think it's

263
00:24:48,880 --> 00:24:55,440
something that we haven't even really like there's no one I think incentivize to really work on that

264
00:24:55,440 --> 00:25:01,760
component right now because you know, that takes time, it takes effort, it's difficult, it's a hard

265
00:25:01,760 --> 00:25:08,880
problem. And there's there's no one forcing people to do that right now. You know, I think like,

266
00:25:08,880 --> 00:25:14,640
even think about something as far as like the development of vehicles, like there was a time

267
00:25:14,640 --> 00:25:18,480
and then space when we didn't have breaks lights in one car company decided like we're going to

268
00:25:18,480 --> 00:25:21,760
this problem because we see this issue, we're going to put in the time extra to develop a

269
00:25:21,760 --> 00:25:27,280
break lights assembly and then everyone adopted it afterwards. And that's a very shortened version

270
00:25:27,280 --> 00:25:30,880
on the history of break lights, which I know sounds like why are you talking about that

271
00:25:30,880 --> 00:25:35,440
history of car development here? But we think about tech development and taking the time and

272
00:25:35,440 --> 00:25:39,440
the money and the investment of a technology that actually then ends up benefiting people

273
00:25:40,160 --> 00:25:47,200
everywhere about it. And so I think there is a bit there's something we can think about consent.

274
00:25:47,200 --> 00:25:52,400
I know that's when we come to think about like SAG-AFTRA and the Vardens Guild and all those

275
00:25:52,400 --> 00:25:57,440
protests we had in the US about in relationships also like AI, like people consenting to their

276
00:25:57,440 --> 00:26:03,120
voice, their images, their videos on themselves being used in AI technology, that is another

277
00:26:03,120 --> 00:26:08,400
component of it. And I think if we push towards this idea of like, what are we consenting to?

278
00:26:09,440 --> 00:26:13,920
I think that would be a healthier space because there's lots of people out there who wouldn't want

279
00:26:13,920 --> 00:26:18,000
to consent to their images being used. There's a lot of artists out there who don't want to consent

280
00:26:18,000 --> 00:26:23,200
to their images being used in AI generation. I think consent is a good thing. And I think a

281
00:26:23,200 --> 00:26:29,920
lot of companies are against that because it is costly, it is prohibitive, it lessens remodeled,

282
00:26:29,920 --> 00:26:36,000
people don't consent to their images being used in an AI generation model. It's a part problem

283
00:26:36,000 --> 00:26:39,760
that I think people don't want to tackle because they're not, instead of actually tackling it.

Speaker 1
00:26:39,760 --> 00:26:49,760
You highlighted how popular AnyDream was as a server in a community. The other platforms you've come across in terms of the majority of them, the majority of them have this access to kind of upload your own photograph, upload other people's photographs. Do the majority of them have that feature? And are all of the platforms you've come across fairly popular? Is AnyDream kind of something unique that you decided to look into? 

Speaker 2
00:27:15,040 --> 00:27:20,640
AnyDream I thought was particularly unique because I thought their Discord community was particularly gross. And they're unique in the

290
00:27:20,640 --> 00:27:25,120
component that they had like this active, they call their discover section. They can go onto their

291
00:27:25,120 --> 00:27:31,600
website and see images that other people were generating. So you saw just like tons of

292
00:27:31,600 --> 00:27:39,200
non-consensual adult intimate imagery that people are like happy to share publicly. And so this is

293
00:27:39,200 --> 00:27:43,840
also where you can find like public CSAM. I unfortunately saw public CSAM, like AI generated

294
00:27:43,840 --> 00:27:49,040
CSAM just on this like, it kind of boggled my mind. Why is this like a celebrated component of this

295
00:27:49,040 --> 00:27:54,400
website? So I focused on it because it just was like, there's so much here that was particularly

296
00:27:54,400 --> 00:28:00,240
gross. And there was like very little, you know, sort of like shame or abas... like there's so much

297
00:28:00,240 --> 00:28:04,080
sort of like shame or abas... like people were very proud to like share their images that they were

298
00:28:04,080 --> 00:28:08,800
creating within the Discord as well. So that's kind of like the reason why. I find that they were

299
00:28:08,800 --> 00:28:13,440
probably like in the mid-small ways, they were kind of like starting level. I wouldn't, they're not

300
00:28:13,440 --> 00:28:18,080
like obviously, as you just say, some of the like mid-journey or like other like companies are trying

301
00:28:18,080 --> 00:28:22,400
to go into that space. Like there are companies that are like much, much, much larger. I picked

302
00:28:22,400 --> 00:28:26,400
this one because I think they were kind of a little bit under the radar, but they're coming up,

303
00:28:26,400 --> 00:28:31,120
they were like on that way up because their image stuff was good and they were particularly,

304
00:28:32,160 --> 00:28:37,280
had not been like, I don't know, stopped or realized that part of the issue of

305
00:28:37,280 --> 00:28:41,600
what they were doing was this like non-censorial bit. Because even when you look at some of the

306
00:28:41,600 --> 00:28:49,360
like major platforms out there, most of them can become mainstream to be made really profitable,

307
00:28:49,360 --> 00:28:55,760
realize they cannot do, they should not be doing non-concensual intimate imagery, right?

308
00:28:55,760 --> 00:29:03,120
So like this AI porn space is in this sort of like area where people are trying to figure out

309
00:29:03,120 --> 00:29:08,640
how to make money in it and how to like be accepted on platforms. And to be clear, when I

310
00:29:08,640 --> 00:29:12,960
talk about this, like the AI pornography, there is a variety of different services on there.

311
00:29:12,960 --> 00:29:16,960
There are ones where people are generating and their services are designed where they only generate

312
00:29:16,960 --> 00:29:24,320
like from like text images or you don't say have a consistent character or if a consistent character

313
00:29:24,320 --> 00:29:30,960
is like a preset one, right? So there's ones where every image isn't necessarily based on a human

314
00:29:30,960 --> 00:29:36,480
person, right? And that's kind of like towards our less, the more ethical ones. And then there's

315
00:29:36,480 --> 00:29:41,920
ones where you can like really based on a particular character, you can make a blood particular face,

316
00:29:41,920 --> 00:29:45,360
you can do a face-fobbing. So one of the other components of any dream is like if you wanted to

317
00:29:45,360 --> 00:29:51,520
say how like a photo and you wanted to replace like the person's face or make whatever the position,

318
00:29:51,520 --> 00:29:57,440
so kind of copy that photo and have a human to your character's face and do that. So you want like,

319
00:29:57,440 --> 00:30:02,320
I want this person doing in a much more ethical, like a parkour imaging or like I want them to do

320
00:30:02,320 --> 00:30:06,800
this really intense imagery that's hard to generate through text, but copy from that image by what my

321
00:30:06,800 --> 00:30:11,600
face on or when this person's face. But oftentimes that's in this case not used in a parkour image.

322
00:30:11,600 --> 00:30:16,240
There are other adult images that were being used and people replacing their faces on that.

323
00:30:16,240 --> 00:30:21,200
And then the other bit, there's all these, these services that are called sort of a service meant

324
00:30:21,200 --> 00:30:26,480
to like make nude images. So the case in Spain, from a company that will say you could upload a

325
00:30:26,480 --> 00:30:32,480
photo of anyone in like clothing. So these are all like real non-AI images people are uploading

326
00:30:32,480 --> 00:30:37,520
and then generating an AI image of that same one, but with that person not wearing clothes.

327
00:30:37,520 --> 00:30:41,360
And so there's a variety of different types of services in those. And these services will vary

328
00:30:41,360 --> 00:30:50,800
in like costs and popularity and options out there. Like it is, it's, yeah, there's variety

329
00:30:50,800 --> 00:30:54,720
of range, like any dreams communities, you know, sometimes range from like a few thousands,

330
00:30:54,720 --> 00:30:58,560
you know, so like their discord had like, I think three, four, maybe 5,000, I can't remember the

331
00:30:58,560 --> 00:31:04,960
exact number in our article. But like, you know, definitely I think 10,000 people. But some of the

332
00:31:04,960 --> 00:31:11,280
other spaces, I'm afraid if I go to switch communities in discord outlaw, so I'm afraid

333
00:31:11,280 --> 00:31:15,920
I'm gonna log myself out of this. But some of these other ones have like, you know, 20,000,

334
00:31:15,920 --> 00:31:19,760
some have less. It's just people are in different stages of their startup development.

335
00:31:20,560 --> 00:31:27,760
And then if you go to like even, because discord policies are against sharing non-censorial images.

336
00:31:27,760 --> 00:31:32,560
So I think it's pretty bold when people are choosing to use a very popular platform and

337
00:31:32,560 --> 00:31:37,760
very clearly saying go to our platform here. And it's against policies. So if you go to telegram,

338
00:31:37,760 --> 00:31:41,680
those communities are much larger because there aren't any rules or regulations on that platform.

339
00:31:43,360 --> 00:31:48,320
Like there's rules on reddit has like pretty strong rules generally on non-censorial

340
00:31:48,320 --> 00:31:53,760
intimate images. And so whatever that does gets, if you do find something out there,

341
00:31:53,760 --> 00:31:57,920
you can actually report it and people do generally take it out. It is frowned upon on that site.

342
00:31:58,720 --> 00:32:02,240
As like, it's not just by like community, but like as an entire site in total.

Speaker 1
00:32:02,240 --> 00:32:55,760
I want to get onto some of the risks of children and also some of the kind of legislation that could come in as that's kind of the main conversation in the chat right now in a second.
But Emma Miller asked a question earlier that I thought I should put in front of you. They asked, hi, Kolina. I'm also part of a group that has done a lot of research on AI generated NCI and your story sounds so similar to some sites we found. However, Stripe and some other services didn't respond by taking their access to the payment platform away. Is there any reporting template or key piece of info that you sent them that you think got them to take action so immediately? I'd really appreciate any insight you might be able to share. What would you say to that particular question? 

Speaker 2
00:32:55,760 --> 00:33:03,520
First of all, I'm glad that you're doing the work in that space. It's great because there are so many different companies that need that scrutiny. I don't know what

353
00:33:03,520 --> 00:33:09,520
necessarily did special. If we can follow up afterwards, I can see if I can get you a copy

354
00:33:09,520 --> 00:33:14,560
of what we did. But essentially the email that we sent was like, hey, we're reporting on the story.

355
00:33:14,560 --> 00:33:20,960
This is kind of what we found. Here's the proof that they're using Stripe. Any comments? And then

356
00:33:20,960 --> 00:33:25,680
it was taken down. We didn't do anything particularly. I think that stuff just made me

357
00:33:25,680 --> 00:33:29,200
very responsive. We got the right person who was immediately like, oh, no, this is not good.

358
00:33:30,560 --> 00:33:38,240
And generally speaking, I think Stripe will at least take it down once they are alerted of the

359
00:33:38,240 --> 00:33:45,600
nature. Other platforms, it's a little bit harder. So we can definitely, I think, touch base afterwards

360
00:33:45,600 --> 00:33:53,200
and see what we can do to help encourage Stripe to take action. 

Speaker 1
00:33:53,200 --> 00:34:40,320
Great. So feel free to DM Kolina afterwards. Keep putting your questions in the chat. We've got about 30 minutes left. I wanted to quickly get on to the risks to children with this type of technology. I think that's a good question. You mentioned that CSAM was in the server that you were exploring, but you also mentioned the case in Spain. And then recently there's been a death of a young girl in London. Could you tell us a little bit about the London case and what kind of risks there are to children? And if there's any controls to stop children generating this kind of imagery? 

Speaker 2
00:34:40,320 --> 00:34:45,280
Yeah. The case in London, I don't know if someone has put in a link to that article, but it hasn't been done already on some reporting

368
00:34:45,280 --> 00:34:52,400
that's been done on it. But the quick gist of that is that there was a young girl who was 14 years old

369
00:34:52,400 --> 00:34:59,200
who was being bullied and she'd been bullied for a long period of time. I don't remember exactly

370
00:34:59,200 --> 00:35:06,000
how many months, but was being bullied. There was a Snapchat group, I think, of classmates,

371
00:35:06,000 --> 00:35:12,720
so other high school boys who were making images of her and I think of a few other girls in

372
00:35:13,280 --> 00:35:20,480
Bath for School and sharing them a teasing batter. And they were making fun of her appearance that

373
00:35:20,480 --> 00:35:25,920
was part of it. And so making non-consensual imagery, making fun of her. I don't know what those

374
00:35:25,920 --> 00:35:33,680
images fully depicted, but things that made the situation feel so horrible. So I thought,

375
00:35:33,680 --> 00:35:39,440
obviously, she ended up taking her own life. And part of that is there's already obviously an ongoing

376
00:35:39,440 --> 00:35:44,480
bullying situation that's happening. But the fact that you're adding onto this level of

377
00:35:45,440 --> 00:35:50,240
images that are being AI generated of her non-censural lettering, a 14-year-old girl

378
00:35:50,240 --> 00:35:56,640
abusing it to continue to bully and harass her. And it ended up in this situation that's

379
00:35:56,640 --> 00:36:03,120
absolutely awful. And I think, again, high school girls are already a victim of this. I can't imagine.

380
00:36:03,120 --> 00:36:06,800
I have some of my cousins that are having kids that are in high school now, which makes me feel

381
00:36:06,800 --> 00:36:13,200
A, very old, but B, they also have cousins who have younger kids who are five. And I can't imagine

382
00:36:13,200 --> 00:36:19,120
the future that they're growing up in where in an 80s movie you'd have maybe someone cut someone's

383
00:36:19,120 --> 00:36:24,160
face and put it on someone's body that you might see in a magazine and that'd be a way to harass

384
00:36:24,160 --> 00:36:33,120
them or something. Now a kid could go onto any website, easily spend like, some of these websites

385
00:36:33,120 --> 00:36:39,360
do it for like, spend a dollar, spend a few dollars and here's a ton of credits. You can generate

386
00:36:40,800 --> 00:36:46,400
X amount of many images for a few bucks, very, very, very cheaply. The barriers of financial

387
00:36:46,400 --> 00:36:52,960
injury on this, very low. And the ability to find it, and basically people have interest in themselves

388
00:36:52,960 --> 00:36:59,600
on the internet. So I think as far as stopping kids to do that, I think any kid that's competent

389
00:36:59,600 --> 00:37:03,760
to do that can click, easily click a button that says, are you over 18? And they click yes.

390
00:37:05,120 --> 00:37:11,120
You know, that's without a braider, but I think there needs to be culturally this conversation

391
00:37:11,120 --> 00:37:18,880
we need to have on a larger scale, like what makes it seemingly okay to young boys across the world,

392
00:37:18,880 --> 00:37:24,320
not just in one country, but globally, that young boys feel that it is okay to do this to their

393
00:37:25,520 --> 00:37:33,120
female classmates, because it's not. And we can theorize about Andrew Tate, we can theorize about

394
00:37:33,120 --> 00:37:38,160
the ideology and sort of the narratives that are really appealing to young boys, and like what kind

395
00:37:38,160 --> 00:37:47,120
of point we are culturally, where the objectification and the dehumanizing act of creating

396
00:37:47,120 --> 00:37:52,320
and generating images of other classmates, of your friends, of people that you interact with,

397
00:37:52,320 --> 00:37:57,440
these are not strangers, they're doing the people they know. And they're making these images to

398
00:37:57,440 --> 00:38:04,000
bully and harass, to stalk, to threaten. And it's happening on a global scale. These are just the

399
00:38:04,000 --> 00:38:09,040
stories we know about that have made national headlines. There are probably hundreds of thousands

400
00:38:09,040 --> 00:38:13,520
more cases we don't know about because they aren't making the news. And I think the takeaway is that

401
00:38:13,520 --> 00:38:19,680
we need to really think about where we are societally on the acceptability of like AI technology,

402
00:38:20,720 --> 00:38:27,440
how acceptable it is to make non-consensual images of other people, and the, you know,

403
00:38:27,440 --> 00:38:33,360
the boys' relationships to girls. There is a lot we need to do on a large scale. That's probably

404
00:38:33,920 --> 00:38:38,480
a lot we could talk about that, and not the amount of time that we have.

Speaker 1
00:38:40,800 --> 00:39:36,720
Yeah, I mean, the sharing, especially in school settings, of non-consensual images of girls and kind of revenge porn isn't a new concept in schools, at least in the UK, that I'm familiar with. But now the tools are making it a lot easier, particularly without even the girls' involvement or any coercion of any kind, which is terrifying. We've got a few questions in about CSAM and
reporting on it. Michael, your colleague, has asked, how can someone wanting research slash investigate this do so in a way that protects their mental health and their safety, let alone their sanity, prompted to ask this based on you mentioning having involuntarily come across AI CSAM?

Speaker 2
00:39:38,400 --> 00:39:50,720
That's great. I don't know. You know, I think it's tough. I guess I have, I don't want to say I have

413
00:39:50,720 --> 00:39:54,080
a high tolerance for awful stuff on the Internet, but I think I have a higher tolerance than most

414
00:39:54,080 --> 00:39:57,280
for awful stuff on the Internet because I've been looking at awful stuff on the Internet for

415
00:39:58,560 --> 00:40:07,120
over a decade now. And like professionally, academically, even longer beforehand. Like

416
00:40:07,840 --> 00:40:11,520
I remember like even when I was in high school, not high school, excuse me, when I was in college,

417
00:40:12,160 --> 00:40:19,120
I got really into researching Stormfront, which was the popular like neo-Nazi white supremacist

418
00:40:19,120 --> 00:40:24,720
like forum at the time, created an account like lurked and like, you know, did there and that was

419
00:40:24,720 --> 00:40:28,320
and I wrote reports on it. And that was back when no one was really talking about

420
00:40:28,320 --> 00:40:33,440
Stormfront that I was aware of because I was in college. No one was anyway. So I've been liking

421
00:40:35,440 --> 00:40:40,800
to be an awful place. CSAM, I think, you know, like I never went to see CSAM. And I think

422
00:40:40,800 --> 00:40:46,400
immediately when I first started researching this, even the AI girlfriend topic, like I was looking

423
00:40:46,400 --> 00:40:53,440
at accounts on Instagram that were doing AI images and I like very quickly, like within the first,

424
00:40:53,440 --> 00:41:00,240
like 10 minutes of searching, found an Instagram account that essentially was promoting the sale

425
00:41:00,240 --> 00:41:04,240
of AI CSAM. Like you have to go to their Telegram and they were you could purchase and subscribe to

426
00:41:04,240 --> 00:41:10,000
AI CSAM. But then 10 minutes found that it is and that account had like thousands of followers.

427
00:41:10,000 --> 00:41:16,240
So I think it is unfortunately something that is the very reality is very easily there. You find it

428
00:41:16,240 --> 00:41:20,240
as far as protecting yourself about it. I think take breaks. Like when I found it, I like basically

429
00:41:20,720 --> 00:41:27,120
threw my phone and like walked away and went on a walk because it made me really upset. But I think,

430
00:41:27,120 --> 00:41:32,160
you know, obviously taking mental health breaks, very self, reminding yourself like motivated to

431
00:41:32,880 --> 00:41:38,640
why you do the work, you know, I should be a look at office on the internet because I want

432
00:41:38,640 --> 00:41:44,160
that awful stuff to be gone. I want to help people who are victims of that. There's like other legal

433
00:41:44,160 --> 00:41:49,360
stuff. There is like organizations that are specifically set up to like look at CSAM and AI

434
00:41:49,360 --> 00:41:55,360
CSAM. We were working with the organization, which when we found suspected AI CSAM hosted in the

435
00:41:55,360 --> 00:42:00,400
Discord server of anything that the images are still there, we could send those on there. It's

436
00:42:00,400 --> 00:42:04,400
like it gets very tricky. So rules, there are really strong rules in some countries about CSAM,

437
00:42:04,400 --> 00:42:09,600
even like viewing it to be a crime, even if you're as a researcher. And so there are particular

438
00:42:09,600 --> 00:42:15,120
organizations that are designed to look and search for CSAM and a way to actually stop it.

439
00:42:15,760 --> 00:42:20,640
So I think it's a really, really difficult area for like just like a lot of researchers to go into

440
00:42:20,640 --> 00:42:25,520
because there is a legal component to it. So I think working with an organization that already

441
00:42:25,520 --> 00:42:31,040
has sort of a legal authority to research into it, even though, and my goal here, I wasn't

442
00:42:31,040 --> 00:42:36,640
specifically trying to research into the AI CSAM. So whenever I did find about it, I reported it,

443
00:42:36,640 --> 00:42:39,920
got it. I didn't care. Like I'm not going to be like some of the stuff that we find out,

444
00:42:39,920 --> 00:42:43,840
like we'll wait until like we're interacting about the report before like contenting authorities.

445
00:42:43,840 --> 00:42:49,280
When it comes to CSAM, that's an immediate like need to stop right away. Some organizations are

446
00:42:49,280 --> 00:42:54,560
really better than others. I think, you know, do a little bit of research. I don't know the list of

447
00:42:54,560 --> 00:42:59,280
the top ones off top of my head. And really, I think be kind to yourselves. You don't have

448
00:42:59,280 --> 00:43:03,920
to do that work. There are people doing that work. You don't feel like this duty about this obligation

449
00:43:03,920 --> 00:43:09,440
if that work is really tough to take on to force yourself to do it. I think if I came up to Bellingcat

450
00:43:09,440 --> 00:43:13,280
like, and be like, you know, guys, I just emotionally spent on the topic. I don't want to

451
00:43:13,280 --> 00:43:17,520
look at it. I don't want to do anymore. No one would be like, you have to keep doing it. You know,

452
00:43:17,520 --> 00:43:21,440
they'd be like, sure, take whatever you need. And I think that's important parts. You don't force

453
00:43:21,440 --> 00:43:27,120
yourself to do it. You take the breaks. You set your sights gold. And then also what I would do

454
00:43:27,120 --> 00:43:31,680
this is actually a technique I learned from one of our other colleagues that I feel so different

455
00:43:31,680 --> 00:43:35,840
out there about that. I find I'm looking at a particular area on the screen or an image. I

456
00:43:35,840 --> 00:43:40,640
like literally would take my hand or I'll pick a piece of paper and I'll cover part of the screen.

457
00:43:40,640 --> 00:43:45,280
So I don't need to see certain images if I'm researching a bit. So like on Instagram,

458
00:43:45,280 --> 00:43:49,600
I know it was kind of popping in a particular area. It's just like defending yourself from

459
00:43:49,600 --> 00:43:53,920
seeing other bits too, like on a visual level. Things like that can be helpful.

Speaker 1
00:43:53,920 --> 00:44:16,160
Thanks. You've actually answered a couple of people's questions in that one answer as well, because people are asking about the legal repercussions of even searching or viewing CSAM. So you answered that. Sarah's put in the chat, who's one of our lovely mods, has said, New York Times looked into this and they built a whole program to prevent them from looking at the imagery itself and read the fingerprint of known images, which is really interesting.

00:44:16,160 -->  00:45:41,200
Thanks for sharing that. We've had quite a lot of questions throughout the hour. And by the way, by the way, we have 15 minutes left. So if you have any questions, please put them in the chat.
But we've had quite a few questions about some of the countermeasures for AI based deep fake porn. Clockwork, for example, is a good question, at least in the anti-states. There's section 230 of the Communications DTC Act, which basically gives full blanket immunity to tech firms from civil litigation. There's a small gap in this for foster sester, which is meant to hold firms accountable for human trafficking and prostitution rings. Could existing laws like that be used to litigate civil cases regarding AI content that is either illegal or should be? Or do we need new legislation?
And is there anything that you've come across that could work in terms of helping restrict the distribution of this? A couple of people in the chat had mentioned maybe digitally watermarking the images as AI generated, but others have disagreed with the effectiveness of that. Others have said, should this kind of content be treated the same way international crime treats child porn? What kind of legislation or moves have you come across or would suggest to any policymakers or change makers listening? 

Speaker 2
00:45:41,200 --> 00:45:47,440
Oh, Dash. Well, first of all, I'm not a lawyer. So we'll say very clearly that is not what my degree is in. But

480
00:45:47,440 --> 00:45:53,440
I do think we need new legislation because I think, you know, like individual states, like

481
00:45:54,000 --> 00:45:57,760
this is the US context because I live in the US and I know this is very subtle, and that's also

482
00:45:57,760 --> 00:46:03,440
the other great assault about that. I think, you know, there is section two there, if I remember,

483
00:46:03,440 --> 00:46:08,720
really focuses particularly on social media platforms and these sites go beyond just like

484
00:46:08,720 --> 00:46:12,480
social media platforms and like other digital platforms. And if my reading of that is like

485
00:46:12,480 --> 00:46:17,360
incorrect, please let me know. And I think that's the biggest issue. And I think that's

486
00:46:17,360 --> 00:46:22,640
the biggest issue. But I think there does seem to be, I think, really clear legislation because

487
00:46:22,640 --> 00:46:27,920
the way that some of the legislation has been originally written is not with the intention of

488
00:46:27,920 --> 00:46:34,640
how it's being how the Internet landscape is today. So as we get as our technology changes,

489
00:46:36,240 --> 00:46:41,040
oftentimes our legislation needs to change with it. As far as like whether or not you can't use a

490
00:46:41,040 --> 00:46:45,840
particular law there, I can't speak to that. Maybe it depends on your lawyer, depends on your judge,

491
00:46:45,840 --> 00:46:51,280
depends on your circumstances. But I think, you know, we had to, you know, a lot of states have

492
00:46:51,280 --> 00:46:56,320
like laws against like revenge board and if you're not sure, but the defense board essentially is

493
00:46:56,320 --> 00:47:02,320
like, imagine you're dating someone, you create adult content together, you record yourself,

494
00:47:02,320 --> 00:47:08,080
photos, images, as you would as a couple would do, you break up. And then after that break up,

495
00:47:08,080 --> 00:47:13,280
one person in that couple and that former relationship decides to share those images

496
00:47:13,280 --> 00:47:17,120
without the consent of the other person. So even though there was equal consent, both parties

497
00:47:17,120 --> 00:47:22,560
consented to the images being taken or recorded at the time, the other person is now not consented

498
00:47:22,560 --> 00:47:27,360
that images being distributed or being used to be harassed or threatened. So like, you know,

499
00:47:27,360 --> 00:47:31,440
get back together or we're going to, I'm going to post it and send this photo to your friends

500
00:47:31,440 --> 00:47:37,680
and family. And that's like the idea of like revenge porn. Because and there are laws against

501
00:47:37,680 --> 00:47:46,800
that. But there aren't necessarily laws that are designed with AI in mind because these images

502
00:47:46,800 --> 00:47:51,680
aren't some people might argue that those aren't real images or those like didn't exactly happen.

503
00:47:52,720 --> 00:48:04,480
You know, there are doctors. I do think we need like strong, clear, you know, sweeping legislation,

504
00:48:04,480 --> 00:48:10,240
not just on a state to state basis. And I think beyond just the US, of course, globally, because

505
00:48:10,240 --> 00:48:14,720
the way that sometimes these companies, so the EDG, they had a corporation in the UK,

506
00:48:14,720 --> 00:48:18,800
but then they were like headquartered here in the US. But they had like people they were making

507
00:48:18,800 --> 00:48:23,440
content of across the world, you know, like people could anyone could join that community.

508
00:48:23,440 --> 00:48:30,800
And so the sort of like borders of where does this responsibility lie needs to be clear because

509
00:48:30,800 --> 00:48:37,040
obviously these communities are globally span in keto, even where people are forming their businesses

510
00:48:37,040 --> 00:48:43,040
on this. There is as far as like other like stuff like the things with the watermark stash,

511
00:48:43,680 --> 00:48:47,760
people can remove watermarks. I don't know that's going to be good enough you crop a little bit out

512
00:48:47,760 --> 00:48:51,760
or something. Some of these services if you do like a free version that you get to see like

513
00:48:51,760 --> 00:48:58,240
their little logo at the bottom. Like, without like, I don't want to name any of them that we

514
00:48:58,240 --> 00:49:02,240
haven't reported on yet. But like some of them like also just like automatically just kind of

515
00:49:02,240 --> 00:49:05,040
like put their little like their products, like some of these companies are proud of it. They

516
00:49:05,040 --> 00:49:08,880
automatically just put their little like, oh, yeah, this image is here was generated on

517
00:49:08,880 --> 00:49:15,520
bubbleblonde.com. They are like, what? Pay extra that are moved. So I don't know. And like,

518
00:49:15,520 --> 00:49:19,200
and Graham, there's like this watermarking bit of it. I don't know if that's going to be enough.

519
00:49:19,200 --> 00:49:26,720
I think it's it's it's a thought that does some little bit. But I don't know if that does anything

520
00:49:26,720 --> 00:49:32,560
if that does anything without consequence. Like I can go on Twitter right now and find

521
00:49:33,760 --> 00:49:36,160
like Nikki Haley is one of the people who are running to be a

522
00:49:37,680 --> 00:49:43,360
GOP nominee for the president's election. And, you know, if I'm finding inocular central intimate

523
00:49:43,360 --> 00:49:49,760
images of her just like not searching for it. But my new I find it on Twitter, just like searching

524
00:49:49,760 --> 00:49:54,560
out research in the platforms themselves. And so like I think, you know, and they're very clearly

525
00:49:54,560 --> 00:50:02,640
people are not ashamed. I think people need more shame. Maybe that's a stronger bit. I'm kind of

526
00:50:02,640 --> 00:50:06,400
like going all over the place. I think I forgot the question here, because there's a lot of questions

527
00:50:06,400 --> 00:50:11,520
in that. But legislation, I do think we need new legislation. I think we need legislation that needs

528
00:50:11,520 --> 00:50:21,120
teeth, not just on. I think we need legislation for the companies. I need legislation for distributors,

529
00:50:21,120 --> 00:50:25,600
people who are producing the content. We need people. And even if you want to go really hard,

530
00:50:25,600 --> 00:50:29,600
people who are like willingly searching out the content and getting to it. Like we think about

531
00:50:29,600 --> 00:50:34,080
the strong laws right now about CSAM. Like there's a case in Canada, like I think three years ago,

532
00:50:34,080 --> 00:50:39,280
where there was where they did arrest a guy who was producing AI CSAM, even though there wasn't

533
00:50:39,280 --> 00:50:43,680
much strong laws or really explicit laws, I should say against AI CSAM, the fact that it was still

534
00:50:43,680 --> 00:50:48,880
considered CSAM. But they were able to still like arrest that guy. But that's like, like when you

535
00:50:48,880 --> 00:50:52,400
read articles or reading about it, you're like, why is there only one guy that's been arrested

536
00:50:52,400 --> 00:50:57,360
in the past three years is generating AI CSAM, but I can go and easily find it unintentionally,

537
00:50:57,360 --> 00:51:05,680
mind you, of looking at AI content on Instagram, on Reddit, on have it at platform. So like,

538
00:51:05,680 --> 00:51:10,720
obviously, the laws are not there, even though they can be applied. And in case I think it needs,

539
00:51:11,520 --> 00:51:18,000
you know, a big task force with some budget that's really dedicated, bring back Dateline, NBC,

540
00:51:18,000 --> 00:51:23,680
you know, bring back like, hey, please have a seat here. Let's talk about you generating

541
00:51:23,680 --> 00:51:34,720
all this AI content of my own, you know. So you might know what I'm talking about. But

Speaker 1
00:51:34,720 --> 00:52:38,640
I think that's also as researchers, kind of, it's kind of that question, what kind of impact can my research can my journalism have in this particular space? And I think that's why at Bellingcat, with yourself, you know, we decided to contact the Stripe, we decided to flag it to the payment companies and to the company, the social media companies or the spaces that they have these communities, because you can hit them in their wallet, right? That's an easy way to kind of help stop the practice of the company enabling this to a point. And it has had a little bit of an effect on any dream, hasn't it? Do you want to speak a little bit about how they've, how, you know, how they've been able to do that?
How they've, how your work has impacted their progress? 

Speaker 2
00:52:38,640 --> 00:52:45,760
Yeah, it was great. I loved making their days miserable. So after, in the course of leading up to publishing, and then of course,

552
00:52:45,760 --> 00:52:50,400
after publishing, I kept a document of like, out of stuff that happened because I wanted to make

553
00:52:50,400 --> 00:52:58,160
sure we kept records of that impact. But Stripe cut off their payment processing, Discord kicked

554
00:52:58,160 --> 00:53:04,800
them off, Reddit kicked them off of their other subreddits. People's accounts got banned. So some

555
00:53:04,800 --> 00:53:08,000
of the moderates accounts, the owners, people that add this, they didn't just like lose the

556
00:53:08,000 --> 00:53:15,120
Discord, they lost their Discord accounts. The PayPal venture kicked them off. They basically lost

557
00:53:15,120 --> 00:53:22,640
all their processing, but then they tried to go and pivot, if you will, to a site called FanView.

558
00:53:22,640 --> 00:53:25,920
And so FanView is like sort of like one of those like creator sites where you could like

559
00:53:25,920 --> 00:53:30,480
pay like a subscription fee or pay like to like pay to get access to content. But they were wanting

560
00:53:30,480 --> 00:53:36,400
to use this people pay for FanView. And then they would get tokens in return through proof of those

561
00:53:36,400 --> 00:53:41,760
purchases. And we contacted FanView and we actually got like email back from I think like your CEO

562
00:53:41,760 --> 00:53:47,520
saying like, oh my gosh, this was not okay. This file is they were like, they do adult content,

563
00:53:47,520 --> 00:53:51,360
their adult content is on their platform. But because they were like trying to like hide what

564
00:53:51,360 --> 00:53:55,200
they didn't consent to like what any dream was doing. So even this adult platform was like,

565
00:53:55,200 --> 00:54:01,840
no. And they kicked those guys off right away. We so basically like every time they try to do

566
00:54:01,840 --> 00:54:05,920
something, they want to get another payment thing we were like on them and just emailed the new

567
00:54:05,920 --> 00:54:08,720
provider they went to and be like, this is what they're doing. The provider was like,

568
00:54:08,720 --> 00:54:14,960
oops, no, we're not touching that. So right now you can only pay with crypto. Unfortunately, crypto

569
00:54:14,960 --> 00:54:21,440
is much, much harder because it's so unregulated and it has so much anonymity. But it did take

570
00:54:21,440 --> 00:54:26,960
past rate like we had to pass their PayPal, a ton for PayPal took action, send them multiple emails,

571
00:54:26,960 --> 00:54:32,080
have other places like encouraging but like, one fortunate thing for most large major payment

572
00:54:32,080 --> 00:54:37,520
processors was it's brought to their attention. They usually are like, it's not a good sign for

573
00:54:37,520 --> 00:54:41,680
us, especially when we are coming to them for comment. Most of the comments we get are all very

574
00:54:41,680 --> 00:54:46,240
generic, like we don't approve of this kind of stuff. But if we're very supposed to like

575
00:54:46,240 --> 00:54:50,000
hear these accounts, like here's the thing that's being done and here's all the evidence of it.

576
00:54:50,000 --> 00:54:54,800
Do you have any comments on that? And then like usually actually gets taken down. So I think what

577
00:54:54,800 --> 00:55:00,880
was nice is that like the digital spaces that these, the people were congregating around to have this

578
00:55:00,880 --> 00:55:06,560
community around, we were able to shut down all major prominent processors, with the exception

579
00:55:06,560 --> 00:55:11,600
of crypto, we got those taken down. Once they decided to like divert to, we got those taken down.

580
00:55:11,600 --> 00:55:17,200
And like their community, I think is only has like 300 people in it. So much, much, much smaller

581
00:55:17,200 --> 00:55:22,080
than what they had in those other spaces. It's seemingly a lots of issues because a lot of people

582
00:55:22,080 --> 00:55:28,000
don't want to pay and use crypto and the guys are doing a terrible job. You know, I think it is,

583
00:55:30,240 --> 00:55:34,080
that's like probably the best case there, which I know is awful because I can still go on there

584
00:55:34,080 --> 00:55:38,240
and people are still sharing content. There are people who are still making non-consensual images

585
00:55:38,240 --> 00:55:43,520
that I would want. I unfortunately can't really go on that one behalf to say like, she doesn't want

586
00:55:43,520 --> 00:55:49,120
this. I need her, like I need the women themselves to also like message that, but like that's like

587
00:55:49,120 --> 00:55:55,360
a huge burden to ask people to be like, go on these sites, go find images of yourself, and then

588
00:55:55,360 --> 00:56:01,680
co-report that. That's like, that's not a sustainable or practical solution, but that's the,

589
00:56:03,760 --> 00:56:09,680
I think the impact on that has been great and it's obviously diminished him, but it hasn't

590
00:56:09,680 --> 00:56:16,400
stopped it. I think for now, the power that we have with reporting on it, talk about publicly,

591
00:56:16,400 --> 00:56:21,040
having outcry puts pressure on the large organizations that do have the power on it

592
00:56:21,040 --> 00:56:26,560
and that's great. And I think we need to work towards it just being an absolute no, like no,

593
00:56:26,560 --> 00:56:35,280
this is just not okay. Yeah, and I think that the fact that they're able to still go on currently

594
00:56:35,280 --> 00:56:42,400
without any further pressure beyond the loose and heavy air riders is not enough. It's like

595
00:56:42,400 --> 00:56:46,400
a little bit sort of like pet peeve of mine that will continue to go on until like they're gone

596
00:56:46,400 --> 00:56:50,880
and dead. Obviously we were able to really, really stall their operations, like the amount of images

597
00:56:50,880 --> 00:56:55,520
that are being shared before to their discourse now has decreased and like, I don't have a

598
00:56:55,520 --> 00:56:59,600
particular number, so it wasn't counting, I mean, it's just being generated daily, but like obviously

599
00:56:59,600 --> 00:57:05,520
it's decreased because like their numbers are not disabled would be on telegram. It is hopefully

600
00:57:05,520 --> 00:57:09,440
slowly dying out and hopefully it'll be gone. And I think hitting them in their wallet, which is where

601
00:57:09,440 --> 00:57:14,640
the incentive is for these companies and these organizations is to make money. It costs them

602
00:57:14,640 --> 00:57:19,600
time and money to be able to generate the images and if they are not making enough money making it,

603
00:57:20,480 --> 00:57:25,760
it will stop. And that is where our power lies, is putting pressure on it. Legislation makes a long

604
00:57:25,760 --> 00:57:30,640
time. We want the legislation to happen until then we've got to put pressure on the payment companies

605
00:57:30,640 --> 00:57:37,680
to really lock down on this. Crypto does not have enough widespread appeal yet for it to be a long

606
00:57:37,680 --> 00:57:42,480
term, I think sustainable option. And I don't think regulations have to get in time to see when it

607
00:57:42,480 --> 00:57:50,240
comes to crypto purchases. So it's just, yeah, that's like, I think. The solution unfortunately,

608
00:57:50,240 --> 00:57:56,880
but our work we're doing here is meaningful. Without the academics, without the journalists,

609
00:57:56,880 --> 00:58:01,440
without the public, without the people interested in this and pressure about it, no change happens

610
00:58:01,440 --> 00:58:06,400
on a large scale. For change to happen, it's not just a couple people that they're going to talk

611
00:58:06,400 --> 00:58:13,280
about. It needs to be like, we're all in on it. So hopefully it's tailored all the photos of stuff,

612
00:58:13,280 --> 00:58:16,320
but it's a lot more public pressure on it. It doesn't just go away in like a week.

Speaker 1
00:58:16,320 --> 00:58:53,360
Yeah. I mean, the US government has responded to other Taylor Swift stuff this week, haven't they? Kind of expressing outrage and want to do something. We are running out of time very quickly.
But I know that a couple of people in the chat have asked this and I wanted to ask it before we wrap. So perhaps you could give a quick answer and then we will end. What are your biggest fears that this will develop into? What are you most worried about that this will become if legislation or more reporting on this isn't done?

Speaker 2
00:58:56,960 --> 00:59:04,640
No one to say about it. It's kind of insane to me that right now, one of the most wealthy,

620
00:59:05,280 --> 00:59:11,760
popular, famous, wealthy women on the planet, women who's got this name saying Taylor Swift

621
00:59:11,760 --> 00:59:18,320
has enough, only barely more protections than a 14 year old girl in England. We just need

622
00:59:18,320 --> 00:59:23,520
protection for these women because if it could happen to Taylor Swift, it could happen to a 14

623
00:59:23,520 --> 00:59:28,080
year old girl. It doesn't stop anyone. In theory right now, no one's stopping someone taking my

624
00:59:28,080 --> 00:59:31,760
face and making an image of me and distributing on the internet. And there's no consequences for

625
00:59:31,760 --> 00:59:39,680
that. I think if anyone knows a woman in their life, they're victims. I saw someone do this and

626
00:59:39,680 --> 00:59:48,560
they make images of iced tea the other day. Sorry, it was ice cube not ice tea. And made a bunch of

627
00:59:49,360 --> 00:59:54,720
NCI to harass him on Twitter. It's not just women who are victims, it's men too. And it'll

628
00:59:54,720 --> 01:00:04,240
often start off being celebrity men, but then it'll be everyone. So I think having this go unchecked,

629
01:00:04,240 --> 01:00:13,440
I would hate to see a world in which NCII, non-essential intimate images, are so commonplace

630
01:00:13,440 --> 01:00:18,560
that anyone could be a victim. In the same way that you would call, you know, put someone's face

631
01:00:18,560 --> 01:00:24,320
on like a meme to make fun of them. It becomes so commonplace that you could just like sling around,

632
01:00:24,320 --> 01:00:28,240
oh I'm going to make a nude of you and post on the internet just to like, because I didn't like what

633
01:00:28,240 --> 01:00:34,800
you said. You know, like if it's, that's kind of like the love word. If it's easy enough that a 12

634
01:00:34,800 --> 01:00:40,320
year old boy can figure it out, and easy enough that probably someone who's like 80, like my dad,

635
01:00:40,320 --> 01:00:47,600
could like figure it out. It's, the fear is that it just becomes normal. We accept that as that,

636
01:00:47,600 --> 01:00:51,040
oh that's just something that happens on the internet. You posted your image, there's a picture

637
01:00:51,040 --> 01:00:56,080
of you on the internet, oh well now you're a potential victim of AI and NCII and that's just

638
01:00:56,080 --> 01:00:59,440
normal. And I don't hate for that to be considered something that's a normal thing that we see on the internet.

Speaker 1
01:00:59,440 --> 01:01:36,000
Thank you so much for the insightful discussion, Kolina, and for sharing more about your research and kind of the motives behind it. And thank you to everyone who's tuned in today
and for discussing solutions to this in the chat. There's some really interesting conversations going
on. Feel free to continue talking, but we're going to end the podcast there. We'll be back
in two weeks time with another stage talk and you'll find this recording on our soundcloud and on podcast platforms, hopefully by the weekend. But thank you again Kolina for joining us.

Speaker 2
01:01:37,440 --> 01:01:49,040
Thanks for having me and see you. We will be having an article on something on this topic in the near future, so look out for that. 

Speaker 1
01:01:49,040 --> 01:02:19,680
Thank you for listening to the stage talk. If you'd like
to catch a stage talk live where you can ask the guest questions, join the Bellingcat Discord server by visiting www.discord.gg slash Bellingcat. The music you've heard is titled Dawn by Newer Self and is courtesy of Artlist.