You're listening to a Bellingcat Discord server stage talk titled The Secretive World of AI Pornography. This week we were joined by Bellingcat researcher Kolina Koltaj. She is an expert in how socio-technical systems influence the decision-making of social groups with a PhD from the University of Texas School of Information. She has also previously worked at the Center for an Informed Public at the University of Washington. Now at Bellingcat, her focus is into online ideologies and her latest research has been into the phenomenon of AI girlfriends and AI sexual content. In the Discord stage talk, Kolina spoke about why she started looking into this area and why she feels so passionately about doing more research, slash speaking for regulation on this topic. A small content warning. In the discussion we reference child sexual abuse material and the death of a child, as well as revenge porn. You can find links to Kolina's research in the description of this podcast. The talk was hosted by me, Charlotte Maher, on Thursday the 1st of February in the Bellingcat Discord server. Welcome to this week's Bellingcat Discord stage talk. Non-consensual sexual deep fakes have been in the headlines this week. As faked, explicit images of Taylor Swift went viral. For many, this type of exploitative content was a new concept, but for Bellingcat researcher Kolina Koltai, who joins us today, it is a trend she's been following for a while. Kolina joins us today to discuss the AI porn industry and the platform set up to profit Any Dream. In late November, Kolina published research on a US artificial intelligence company that was collecting money for a service that can create non-consensual pornographic deep fakes using financial services company Stripe, which bans processing payments for adult material. After joining the Discord community, may I note, fostered by the company for creation of these images, she discovered references to child sexual material being created and distributed, and non-consensual generated images of celebrities and real women being shared. This kind of generated material has sadly recently led to the death of a young girl in London, which we will also speak to during the course of this talk. As Kolina talks, please put all of your questions in the chat. I'll ask them as we move into the Q&A section. So please put them in the chat and I'll be moderating that. Thanks so much for coming, Kolina. The stage is yours. Speaker 2
Thanks so much for having me here, Charley. I am excited, but it feels like sometimes a weird word to say when you're talking about a icky topic that you're excited to talk about the topic. Speaker 2
But this is obviously a really important topic that's happening today and that I am thrilled that it is getting so much more attention because it's something that's been awful, awful stuff happening, occurring at sometimes different corners of the internet. And we like to think they're in these like dark, shady quarters where you have to go on the dark web. But no, you can go in their websites. You could access easily where you can create this content and find this content. So let me start from the beginning before I kind of just derail myself because there is so much to cover and talk about that I feel like I can just go a million miles a minute talking about it. But I hope everyone is having a good morning, good evening or good afternoon, Speaker 2
depending on where you are. And I want to talk a little bit first about some of the initial research we did and we published on in November, which we were looking at a company that we found to be based here in the US where I am, but actually have like incorporation in the UK. And obviously with a lot of stuff that you see on the internet, there's some really clear borders. Speaker 2
At first, we kind of started looking into this topic of AI girlfriends. Actually, on like a little bit of a whim. I was scrolling Twitter and someone said they found a popular influencer and that she had not disclosed she was actually using AI images. But I was like, oh, this is kind of interesting. I wonder how many people on Instagram are using AI images of themselves or other women pretending to be someone that they're not. And I started digging into this topic. And Charley's
there too, because I actually talked to Charley a little bit about this topic last year. I started
getting questions like, man, how are all these people creating these images? There are, you know,
dozens of hundreds, sometimes thousands if you think about it, people who are using not just
a sort of shop, but like using AI technology to create new images, sometimes edited versions of
themselves or completely new women. And the seemingly the barrier to entry and creating this
content was drastically dropping at a speed that we had anticipated, I think, it comes to like
visual media. So I started looking into the tech component. And very quickly, a company's name kind
of started consistently popping up, which was one called AnyDream. I went out and checked out their
site and found that they had a Discord community. And also what they still have, and it's still up
unfortunately, is they have something called a Discover section on their website in which
you can go there and view all the images people are creating and publicly share. So people are
electing to share these images publicly on AnyDream's website, including their prompts. So
I think the goal of the developers, people who own this, would be like, oh, you can see the images
and what kind of like words and terms people are using to generate the images, but then use that
to build off it. Or you see this image, you're like, oh, I'm going to take this image, but then
do a little bit of something different with it. So this can work really great in a community
that is not necessarily steeped in non-consensual sexual imagery. If you're like, oh, you know,
I want to make images around like making these animals or something like that. But when we come
to the idea that you're making images of women who are not just made up women, not completely AI
women, but women that are actually based within real people. And one of the particularly unique
features of AnyDream is some other components that you can upload the face of anyone. So like,
you can upload like our founders, Ali Hinkins face, or you can upload the Taylor Smith face.
You can upload the face of your neighbors or anyone that you know, a celebrity, real, whoever,
and then generates images of them and sexual images of them, generate photography, and it
worked really well. And that's the thing. Like there is, as with any sort of AI image technology,
sometimes a little bit of a play on it, but actually worked pretty well. So immediately,
I saw this like, do we got to focus on this company? And while they were trying really hard,
I think, to be secretive, I think people, they, people get conky. They don't realize
how much information they're sharing about themselves so that people care about this,
right? There hasn't been very much scrutiny on the companies and the communities that are
focused on this. Like I'm creating it. I have joined plenty of Discord communities that are
generating this content for different companies, and that's just one of them. And people are really
bold. They try using a pseudonym, but with a little bit about some OZIP skills, you can usually
figure out who's behind certain pseudonym because people like to use it consistently. So one other
company, a much smaller one that I was looking into, and I'm actively monitoring their Discords,
their Discords, they, you know, this guy's, what are their devs slash their owner uses the same
username across all his profiles, which made it very easy that defined like his GitHub, his name,
is like, you know, if you use the same profile image and username across it, it becomes very easy.
But it's, I'm derailing. So what we decided to look at is not just obviously look at this community
in which we found, you know, not just like the sharing of like AI generated non-custential imagery,
but like I think the act of like sort of encouragement of it. They had channels completely
dedicated to sharing those images and seemingly really no pushback when it came to showing images
of celebrities, of real people. You would have users very clearly say, hey, I made nudes of my
wife's frag, here they are. I've made nudes of this person. But there is, from the community,
they did not seem to really improve CSAM, which is good, but there is so little work. Like whenever
someone would question the devs or the owners about saying, hey, you know, hey, keep finding
CSAM. Hey, there's really disgusting stuff here. Hey, do something about it. And he really was doing
the bare minimum and kind of was like, oh, I'm just going to leave it for someone else's problem. I'm
not focused on that. So it is, it was really, it still is about that small company. After we published
the story, because we found that they were hiding their payments. So one thing that they were doing
to disguise their payments from Stripe was they would use a redirect. So if you click to say, oh,
I'm going to pay another sort of site, like a loading window would come up very briefly. And
if you will look at the address bar, it would change the name of that website. And then it would
say like Stripe's payments and the sales screen. And you would see that it wasn't being processed
by imagery. It was being crossed payment from some other site. And not only does this give protection
to people who are purchasing it, because it doesn't show up like you're buying tokens or coins to use
a AI generation service. But it also hides some Stripe who had no idea. Because right after,
when we started contacting Stripe for comments about it, immediately, like within the hour after
we contacted Stripe, their payment system got shut down. So Stripe took it very, very seriously,
which is great on Stripe. And later after the fact, one of my friends contacted me about it.
And she was like, hey, I didn't know at the time, but like her husband, I was working at Stripe,
or still is working at Stripe. And she was like, I said, I could have like got that shut down for
you just let me know. They take it very seriously. So good on Stripe for taking it seriously. But
of course, there's no way for them to know that like going in and investigating that these are
using a redirect to match this payment, which can be a really common way to match payments from
these kind of providers. We just set up another website and it seems like payments are going to
that website. They also were very bold to be like, hey, you can just PayPal me. Here's my like personal
PayPal address. And it took us a while, but we did eventually get PayPal to also agree to like shut
down those accounts. There is definitely a variance in response time from some of these financial
companies for financial payment processors, even though this very clearly breaks the rules.
And also, for example, there are the websites being hosted through Squarespace. Be fantastic,
Squarespace. Hey, this violates your policy on adult content. There is very clearly adult content
being hosted on Squarespace. Websites that love Squarespace haven't done anything. And so
when we talk about it, think about the public pressure that's happening right now.
You know, there's talk about new legislation. There's public outcry. And I'm like, this is
great because me sitting here at my keyboard and telling y'all is just part of that process. It is
getting upset at people who are in positions to actually do something. Be reporting on it.
But they on belly cat writing stories is how we're going to actually, I think, institute some change
when it comes to putting pressure on systems like the payment processor, because we could address
the issue of payment. You know, it really incentivize people. Now it's always possible to go to crypto,
but a lot of people still don't want to necessarily want to do the crypto. It's much easier to just say
take a credit card, use PayPal, use one of these mainstream payment providers. The other thing is
we can push them off to other parts of the internet. So it's not just like an easily found
thing on, you know, social media platforms on through Google search. You know, these services,
and a dream and other ones are being used to generate hundreds, thousands and sort of like
almost incomprehensible amounts of non-consensual delimitry. And most arguably, I doubt that the,
I can guarantee you I'd be willing to bet money that it's overwhelmingly women who are being
targets. And I think that's really, unfortunately, not everyone who is above the age of 18 or targets.
And I think that is the really tricky part, because even when I go to some of the telegram
groups that are directed to it, these groups are huge. Some of them, like the smaller ones,
as for services that are paying for this on the small end, have like 10, 15,000 people in it.
On the larger end, the largest group I've seen on Telegram has 800,000 people on the singular channel
dedicated to the creation of this to pay for it. 800,000 people, that's more Twitter followers,
Twitter followers than, but I guess like one of the largest telegram groups I think I've ever seen.
It is huge. And so this is a problem, I think that, you know, people will say like, oh, we've had
Photoshop for a really long time. How does this change anything? Just sometimes people will go
and talk about like personal. Yes, we've had Photoshop. But think about the e-photoshop
expensive, you can of course, they don't hire copyers, other sort of knockoffs out there,
but the barrier to entry are creating a convincing bit of non-consensual intimate
media was much higher five years ago. You know, in the past like year, you know, 12, 18 months,
things have changed dramatically. And that barrier to entry, the barrier and the ease of it, right?
You don't even need to be that sophisticated and creating AI imagery, right? You know, you can go
and use something like mid-journey or any of the other platforms, sample diffusion, and sort of
just play around with it and create it. You can even go into Bing, you know, even my father, who's
80, could probably figure out how to generate a basic image. And so really think about it, it
becomes incredibly easy to create these kinds of content and then people figure out how to get
around it. People have not figured out the great way to go around modifying or, you know, really
about how to moderate these spaces well, because, you know, you know that you can write a name like
Taylor Swift, and if you want to ban the word Taylor Swift, you can put a singer that looks like
Taylor Swift, and sometimes that will work. Or you misspell Taylor Swift slightly and the system
goes, oh, I think you're trying to misspell Taylor Swift and it still works. Or you put in the word
like singer and between it and break it up into your bit, right? Or if they're really lazy, you
can do lowercase so that it works. Like it is, there's so, people are a, they haven't thought
enough about moderation, two are lazy, and three people are really smart to figure around
moderation. Some of my other previous work was looked at how communities got around moderating
systems, and humans are really, really smart at figuring it out. So it's something that we need to
be really proactive. We can't be just reactive. Like right now we're reacting to this image being
generated like that's out there, but I can tell you that those images about Taylor Swift,
I mean, not those specific ones, but AI non-consensual intimate imagery of Taylor Swift
and thousands of other women are being generated daily and have been for months, years. I think
it is what I think is also a little disheartening that it takes like a giant global pop superstar
to do it, but I am glad in a sense because if something actually happens, it's going to be
really frustrating if nothing does happen. And I think we need something on a global scale.
We talked about legislation as who is responsible. It's not just supposed to be just on one country.
I think there does need to be pressure on obviously the services that are doing this,
payment processors who are allowing it to kind of continue on to accept payment,
people who are sharing, disseminating that content. There needs to be restrictions in a similar way
we think about CSAM, which is child sex abuse material. We are globally like most countries
have really strong rules and regulations around it. We are still behind on AI CSAM. We are definitely
behind on deep fake content and non-consensual deep fake content because our legislation is slow.
Things need to happen. So legislation is slow everywhere. We know technology on pace and I feel
like I'm going on and on about this. Let's see what else about this? I think there are as far as
where these spaces are, it is literally I think it doesn't take that long to find it. There are
forums dedicated to it. You can find discords dedicated to it. You can find it on Twitter.
I was just searching, there's a recent report from Graphica I think. I might have to pull up the link.
The Graphica did a recent short report on referral links and sort of the rise of that space.
I think they saw like it just in 2023, so last year, something like a 2600,000% increase in
referral links for this kind of content just on Reddit and Twitter, which is like skyrocketing high.
I think people are also really just unabashed with the people who are creating this. It's like
people don't care. I kept following the anydream community space after they moved to Telegram
and I kind of lurked in their Telegram. They felt like they were being discriminated,
like literally they said we're being discriminated against because we got kicked off Stripe and we
got kicked off Discord. They don't understand us. We're not doing anything wrong. We just have to
stick to Cripo for now because people don't understand what we're doing. And remember,
this is the community of people who are generating this content. And some people are like,
oh, I don't understand what my Discord account, like I wasn't the one creating the celebrity
content, but you're in there. Like some of the mods were upset and it's like the community people
and they are really unabashed. And these are people from sort of all walks of life. There are
people all over the globe. I don't think that many people who are really actively in that space feel
guilty about it because that's what they're saying in those spaces. So I think it is a,
I don't know how many, how pervasive this ideology is becoming, but it is overwhelmingly women who
are victims of this and overwhelmingly the men who are in those spaces don't see it as like an issue.
I will say there is sort of a room and space. There are say women who use AI technology
because they themselves are consenting to it. They're making AI images of themselves and that's a
very different thing, right? There are, that's the consensual commit for overwhelmingly this kind of
content and this type of technology, it is non-consensual. Women are not consenting to it
and overwhelmingly targets and oftentimes the onerous is on the woman. So like for the
Eddie Dre example, when we were questioning them about like the kind of content and the non-consensual
imagery of like the women who are real women and people were using their names and I was able to
Google them and find them incredibly easily on the site. And they're like, it is then up to that
woman to A, find out that images of her being created and then contact some brand and developer,
some brand guys running a website being like, hey, there's all these images of me that I didn't agree
to. And then hopefully those images get taken down versus like this guy who's making, still making
those images, posting the images of this woman on Twitter, on Reddit, on everywhere. And it's on her,
it's on the woman to ask places to stop. And that's unfair. I would hate like, imagine how hard
that would be to go on like the internet and suddenly see fake nudes being made of you. Like
you read the stories, like there's this case in Spain that made huge news in September last year
because they were using one of these AI services and it was high school boys who were targeting
their female classmates. So like even though the boys themselves ranged from ages 12, 14,
the girls in question were aged range from 11 to 17. So you're all minors involved. And so,
you know, I think the, assuming these boys, you can't just say boys will be boys because they
thought, so they really thought there was nothing wrong with creating a group in which they made
these images of their classmates, share those images. And then even some of the kids tried to
extort their high school classmates saying like, I have these fake images of you, send me money,
send me other fake nudes of you or send me other news of you, otherwise I better share these around.
And huge outcry. Women are minors, girls are at the core of this, that company still up and going,
no consequence. And I think there's, things need to be done. I don't know, maybe I'll leave it there.
That's probably a terrible story to end it on, but like I think that's made such huge news and
absolutely nothing happened except, you know, a bunch of girls being victims. And, you know,
we can talk a little bit about ideology, ideology, what makes a teenage woman to do that, but I think
we're making it okay. There's no consequence besides the slap on the wrist. There's no
consequence of the companies that are happening. Something needs to change. Speaker 1
Thanks, Kolina. Thanks for explaining that. Maybe one of the mods could put your piece of research in the chat. I think that would be really useful, if you're out there. Thank you so much. You can feel your passion for this topic coming through the screen. And I think it's a topic that all of us should be highly passionate about. We've had some really interesting questions while you've been talking, coming into the chat, including discussions on what we can do about this problem. But I first wanted to start with the Simian's question, because I think it relates to a lot of what you were saying. They were mentioning that in film class, they were taught that before releasing any media of any people who start in projects, they have to get them to sign media consent and release forms. How come AI generated pornographic content isn't in the same area? Is it possible nowadays to kind of own your own image in this kind of social media world? And is this kind of consent possible in terms of this kind of AI generated pornography? Speaker 2
We see people share images of other people all the time without their consent.
Gosh, I remember even when this is like Percy's work, but even on my Instagram, I remember I shared
a photo of me from a Halloween costume. So my face is covered, but I remember I posted this on
Instagram. And someone who followed me took my photo and reposted it as if it was her,
without attributions. It's like that was her Halloween costume. And I was like, what?
Who does that? Why is this woman who I knew reposted my photo thinking I wouldn't see it?
And I didn't say anything at the time. But I think there's obviously been an issue with just
culturally on the internet of lack of attribution. I think we're as, you know,
cultural practices were getting better about that. But I think the difference I think is that with
film is you do have a consent form there, because you're actively aware that someone generally is
filming you. You know, when you, with the exception of general somebody's those reality shows where
they like to film you and they're hidden, and then they ask for consent forms later on. If someone
doesn't sign it, then they blur it. There is something like that kind of like active act happening
versus you post something on the internet. Anyone can then like, you know, screenshot it, download
it, carry it, and then post it. And you just don't know. And I do think we need to get better about
that attribution because you know, this that work goes beyond just, you know, obviously, the AI space,
it goes beyond it goes into thinking about like, just photographs that people have copyrights that
people own. I have I did see one AI porn site that did try to be a little better. I don't remember
which one it was, but they did have this thing that like if you wanted to upload a face, you could
only do your own face and you had to like show like proof of like ID or something. And I was like,
oh, that's kind of interesting. I haven't seen anyone else do that. So therefore, you're
saying because you prove that this is my face and they were like very stringent, like we have the
technology to do it, but we only want people who prove it's them to do it with just one way of doing
it when it comes to a generation component if you're presented your own imagery. But I think it's
something that we haven't even really like there's no one I think incentivize to really work on that
component right now because you know, that takes time, it takes effort, it's difficult, it's a hard
problem. And there's there's no one forcing people to do that right now. You know, I think like,
even think about something as far as like the development of vehicles, like there was a time
and then space when we didn't have breaks lights in one car company decided like we're going to
this problem because we see this issue, we're going to put in the time extra to develop a
break lights assembly and then everyone adopted it afterwards. And that's a very shortened version
on the history of break lights, which I know sounds like why are you talking about that
history of car development here? But we think about tech development and taking the time and
the money and the investment of a technology that actually then ends up benefiting people
everywhere about it. And so I think there is a bit there's something we can think about consent.
I know that's when we come to think about like SAG-AFTRA and the Vardens Guild and all those
protests we had in the US about in relationships also like AI, like people consenting to their
voice, their images, their videos on themselves being used in AI technology, that is another
component of it. And I think if we push towards this idea of like, what are we consenting to?
I think that would be a healthier space because there's lots of people out there who wouldn't want
to consent to their images being used. There's a lot of artists out there who don't want to consent
to their images being used in AI generation. I think consent is a good thing. And I think a
lot of companies are against that because it is costly, it is prohibitive, it lessens remodeled,
people don't consent to their images being used in an AI generation model. It's a part problem
that I think people don't want to tackle because they're not, instead of actually tackling it. Speaker 1
You highlighted how popular AnyDream was as a server in a community. The other platforms you've come across in terms of the majority of them, the majority of them have this access to kind of upload your own photograph, upload other people's photographs. Do the majority of them have that feature? And are all of the platforms you've come across fairly popular? Is AnyDream kind of something unique that you decided to look into? Speaker 2
AnyDream I thought was particularly unique because I thought their Discord community was particularly gross. And they're unique in the
component that they had like this active, they call their discover section. They can go onto their
website and see images that other people were generating. So you saw just like tons of
non-consensual adult intimate imagery that people are like happy to share publicly. And so this is
also where you can find like public CSAM. I unfortunately saw public CSAM, like AI generated
CSAM just on this like, it kind of boggled my mind. Why is this like a celebrated component of this
website? So I focused on it because it just was like, there's so much here that was particularly
gross. And there was like very little, you know, sort of like shame or abas... like there's so much
sort of like shame or abas... like people were very proud to like share their images that they were
creating within the Discord as well. So that's kind of like the reason why. I find that they were
probably like in the mid-small ways, they were kind of like starting level. I wouldn't, they're not
like obviously, as you just say, some of the like mid-journey or like other like companies are trying
to go into that space. Like there are companies that are like much, much, much larger. I picked
this one because I think they were kind of a little bit under the radar, but they're coming up,
they were like on that way up because their image stuff was good and they were particularly,
had not been like, I don't know, stopped or realized that part of the issue of
what they were doing was this like non-censorial bit. Because even when you look at some of the
like major platforms out there, most of them can become mainstream to be made really profitable,
realize they cannot do, they should not be doing non-concensual intimate imagery, right?
So like this AI porn space is in this sort of like area where people are trying to figure out
how to make money in it and how to like be accepted on platforms. And to be clear, when I
talk about this, like the AI pornography, there is a variety of different services on there.
There are ones where people are generating and their services are designed where they only generate
like from like text images or you don't say have a consistent character or if a consistent character
is like a preset one, right? So there's ones where every image isn't necessarily based on a human
person, right? And that's kind of like towards our less, the more ethical ones. And then there's
ones where you can like really based on a particular character, you can make a blood particular face,
you can do a face-fobbing. So one of the other components of any dream is like if you wanted to
say how like a photo and you wanted to replace like the person's face or make whatever the position,
so kind of copy that photo and have a human to your character's face and do that. So you want like,
I want this person doing in a much more ethical, like a parkour imaging or like I want them to do
this really intense imagery that's hard to generate through text, but copy from that image by what my
face on or when this person's face. But oftentimes that's in this case not used in a parkour image.
There are other adult images that were being used and people replacing their faces on that.
And then the other bit, there's all these, these services that are called sort of a service meant
to like make nude images. So the case in Spain, from a company that will say you could upload a
photo of anyone in like clothing. So these are all like real non-AI images people are uploading
and then generating an AI image of that same one, but with that person not wearing clothes.
And so there's a variety of different types of services in those. And these services will vary
in like costs and popularity and options out there. Like it is, it's, yeah, there's variety
of range, like any dreams communities, you know, sometimes range from like a few thousands,
you know, so like their discord had like, I think three, four, maybe 5,000, I can't remember the
exact number in our article. But like, you know, definitely I think 10,000 people. But some of the
other spaces, I'm afraid if I go to switch communities in discord outlaw, so I'm afraid
I'm gonna log myself out of this. But some of these other ones have like, you know, 20,000,
some have less. It's just people are in different stages of their startup development.
And then if you go to like even, because discord policies are against sharing non-censorial images.
So I think it's pretty bold when people are choosing to use a very popular platform and
very clearly saying go to our platform here. And it's against policies. So if you go to telegram,
those communities are much larger because there aren't any rules or regulations on that platform.
Like there's rules on reddit has like pretty strong rules generally on non-censorial
intimate images. And so whatever that does gets, if you do find something out there,
you can actually report it and people do generally take it out. It is frowned upon on that site.
As like, it's not just by like community, but like as an entire site in total. Speaker 1
I want to get onto some of the risks of children and also some of the kind of legislation that could come in as that's kind of the main conversation in the chat right now in a second. But Emma Miller asked a question earlier that I thought I should put in front of you. They asked, hi, Kolina. I'm also part of a group that has done a lot of research on AI generated NCI and your story sounds so similar to some sites we found. However, Stripe and some other services didn't respond by taking their access to the payment platform away. Is there any reporting template or key piece of info that you sent them that you think got them to take action so immediately? I'd really appreciate any insight you might be able to share. What would you say to that particular question? Speaker 2
First of all, I'm glad that you're doing the work in that space. It's great because there are so many different companies that need that scrutiny. I don't know what
necessarily did special. If we can follow up afterwards, I can see if I can get you a copy
of what we did. But essentially the email that we sent was like, hey, we're reporting on the story.
This is kind of what we found. Here's the proof that they're using Stripe. Any comments? And then
it was taken down. We didn't do anything particularly. I think that stuff just made me
very responsive. We got the right person who was immediately like, oh, no, this is not good.
And generally speaking, I think Stripe will at least take it down once they are alerted of the
nature. Other platforms, it's a little bit harder. So we can definitely, I think, touch base afterwards
and see what we can do to help encourage Stripe to take action. Speaker 1
Great. So feel free to DM Kolina afterwards. Keep putting your questions in the chat. We've got about 30 minutes left. I wanted to quickly get on to the risks to children with this type of technology. I think that's a good question. You mentioned that CSAM was in the server that you were exploring, but you also mentioned the case in Spain. And then recently there's been a death of a young girl in London. Could you tell us a little bit about the London case and what kind of risks there are to children? And if there's any controls to stop children generating this kind of imagery? Speaker 2
Yeah. The case in London, I don't know if someone has put in a link to that article, but it hasn't been done already on some reporting
that's been done on it. But the quick gist of that is that there was a young girl who was 14 years old
who was being bullied and she'd been bullied for a long period of time. I don't remember exactly
how many months, but was being bullied. There was a Snapchat group, I think, of classmates,
so other high school boys who were making images of her and I think of a few other girls in
Bath for School and sharing them a teasing batter. And they were making fun of her appearance that
was part of it. And so making non-consensual imagery, making fun of her. I don't know what those
images fully depicted, but things that made the situation feel so horrible. So I thought,
obviously, she ended up taking her own life. And part of that is there's already obviously an ongoing
bullying situation that's happening. But the fact that you're adding onto this level of
images that are being AI generated of her non-censural lettering, a 14-year-old girl
abusing it to continue to bully and harass her. And it ended up in this situation that's
absolutely awful. And I think, again, high school girls are already a victim of this. I can't imagine.
I have some of my cousins that are having kids that are in high school now, which makes me feel
A, very old, but B, they also have cousins who have younger kids who are five. And I can't imagine
the future that they're growing up in where in an 80s movie you'd have maybe someone cut someone's
face and put it on someone's body that you might see in a magazine and that'd be a way to harass
them or something. Now a kid could go onto any website, easily spend like, some of these websites
do it for like, spend a dollar, spend a few dollars and here's a ton of credits. You can generate
X amount of many images for a few bucks, very, very, very cheaply. The barriers of financial
injury on this, very low. And the ability to find it, and basically people have interest in themselves
on the internet. So I think as far as stopping kids to do that, I think any kid that's competent
to do that can click, easily click a button that says, are you over 18? And they click yes.
You know, that's without a braider, but I think there needs to be culturally this conversation
we need to have on a larger scale, like what makes it seemingly okay to young boys across the world,
not just in one country, but globally, that young boys feel that it is okay to do this to their
female classmates, because it's not. And we can theorize about Andrew Tate, we can theorize about
the ideology and sort of the narratives that are really appealing to young boys, and like what kind
of point we are culturally, where the objectification and the dehumanizing act of creating
and generating images of other classmates, of your friends, of people that you interact with,
these are not strangers, they're doing the people they know. And they're making these images to
bully and harass, to stalk, to threaten. And it's happening on a global scale. These are just the
stories we know about that have made national headlines. There are probably hundreds of thousands
more cases we don't know about because they aren't making the news. And I think the takeaway is that
we need to really think about where we are societally on the acceptability of like AI technology,
how acceptable it is to make non-consensual images of other people, and the, you know,
the boys' relationships to girls. There is a lot we need to do on a large scale. That's probably
a lot we could talk about that, and not the amount of time that we have. Speaker 1
Yeah, I mean, the sharing, especially in school settings, of non-consensual images of girls and kind of revenge porn isn't a new concept in schools, at least in the UK, that I'm familiar with. But now the tools are making it a lot easier, particularly without even the girls' involvement or any coercion of any kind, which is terrifying. We've got a few questions in about CSAM and reporting on it. Michael, your colleague, has asked, how can someone wanting research slash investigate this do so in a way that protects their mental health and their safety, let alone their sanity, prompted to ask this based on you mentioning having involuntarily come across AI CSAM? Speaker 2
That's great. I don't know. You know, I think it's tough. I guess I have, I don't want to say I have
a high tolerance for awful stuff on the Internet, but I think I have a higher tolerance than most
for awful stuff on the Internet because I've been looking at awful stuff on the Internet for
over a decade now. And like professionally, academically, even longer beforehand. Like
I remember like even when I was in high school, not high school, excuse me, when I was in college,
I got really into researching Stormfront, which was the popular like neo-Nazi white supremacist
like forum at the time, created an account like lurked and like, you know, did there and that was
and I wrote reports on it. And that was back when no one was really talking about
Stormfront that I was aware of because I was in college. No one was anyway. So I've been liking
to be an awful place. CSAM, I think, you know, like I never went to see CSAM. And I think
immediately when I first started researching this, even the AI girlfriend topic, like I was looking
at accounts on Instagram that were doing AI images and I like very quickly, like within the first,
like 10 minutes of searching, found an Instagram account that essentially was promoting the sale
of AI CSAM. Like you have to go to their Telegram and they were you could purchase and subscribe to
AI CSAM. But then 10 minutes found that it is and that account had like thousands of followers.
So I think it is unfortunately something that is the very reality is very easily there. You find it
as far as protecting yourself about it. I think take breaks. Like when I found it, I like basically
threw my phone and like walked away and went on a walk because it made me really upset. But I think,
you know, obviously taking mental health breaks, very self, reminding yourself like motivated to
why you do the work, you know, I should be a look at office on the internet because I want
that awful stuff to be gone. I want to help people who are victims of that. There's like other legal
stuff. There is like organizations that are specifically set up to like look at CSAM and AI
CSAM. We were working with the organization, which when we found suspected AI CSAM hosted in the
Discord server of anything that the images are still there, we could send those on there. It's
like it gets very tricky. So rules, there are really strong rules in some countries about CSAM,
even like viewing it to be a crime, even if you're as a researcher. And so there are particular
organizations that are designed to look and search for CSAM and a way to actually stop it.
So I think it's a really, really difficult area for like just like a lot of researchers to go into
because there is a legal component to it. So I think working with an organization that already
has sort of a legal authority to research into it, even though, and my goal here, I wasn't
specifically trying to research into the AI CSAM. So whenever I did find about it, I reported it,
got it. I didn't care. Like I'm not going to be like some of the stuff that we find out,
like we'll wait until like we're interacting about the report before like contenting authorities.
When it comes to CSAM, that's an immediate like need to stop right away. Some organizations are
really better than others. I think, you know, do a little bit of research. I don't know the list of
the top ones off top of my head. And really, I think be kind to yourselves. You don't have
to do that work. There are people doing that work. You don't feel like this duty about this obligation
if that work is really tough to take on to force yourself to do it. I think if I came up to Bellingcat
like, and be like, you know, guys, I just emotionally spent on the topic. I don't want to
look at it. I don't want to do anymore. No one would be like, you have to keep doing it. You know,
they'd be like, sure, take whatever you need. And I think that's important parts. You don't force
yourself to do it. You take the breaks. You set your sights gold. And then also what I would do
this is actually a technique I learned from one of our other colleagues that I feel so different
out there about that. I find I'm looking at a particular area on the screen or an image. I
like literally would take my hand or I'll pick a piece of paper and I'll cover part of the screen.
So I don't need to see certain images if I'm researching a bit. So like on Instagram,
I know it was kind of popping in a particular area. It's just like defending yourself from
seeing other bits too, like on a visual level. Things like that can be helpful. Speaker 1
Thanks. You've actually answered a couple of people's questions in that one answer as well, because people are asking about the legal repercussions of even searching or viewing CSAM. So you answered that. Sarah's put in the chat, who's one of our lovely mods, has said, New York Times looked into this and they built a whole program to prevent them from looking at the imagery itself and read the fingerprint of known images, which is really interesting.
Thanks for sharing that. We've had quite a lot of questions throughout the hour. And by the way, by the way, we have 15 minutes left. So if you have any questions, please put them in the chat. But we've had quite a few questions about some of the countermeasures for AI based deep fake porn. Clockwork, for example, is a good question, at least in the anti-states. There's section 230 of the Communications DTC Act, which basically gives full blanket immunity to tech firms from civil litigation. There's a small gap in this for foster sester, which is meant to hold firms accountable for human trafficking and prostitution rings. Could existing laws like that be used to litigate civil cases regarding AI content that is either illegal or should be? Or do we need new legislation? And is there anything that you've come across that could work in terms of helping restrict the distribution of this? A couple of people in the chat had mentioned maybe digitally watermarking the images as AI generated, but others have disagreed with the effectiveness of that. Others have said, should this kind of content be treated the same way international crime treats child porn? What kind of legislation or moves have you come across or would suggest to any policymakers or change makers listening? Speaker 2
Oh, Dash. Well, first of all, I'm not a lawyer. So we'll say very clearly that is not what my degree is in. But
I do think we need new legislation because I think, you know, like individual states, like
this is the US context because I live in the US and I know this is very subtle, and that's also
the other great assault about that. I think, you know, there is section two there, if I remember,
really focuses particularly on social media platforms and these sites go beyond just like
social media platforms and like other digital platforms. And if my reading of that is like
incorrect, please let me know. And I think that's the biggest issue. And I think that's
the biggest issue. But I think there does seem to be, I think, really clear legislation because
the way that some of the legislation has been originally written is not with the intention of
how it's being how the Internet landscape is today. So as we get as our technology changes,
oftentimes our legislation needs to change with it. As far as like whether or not you can't use a
particular law there, I can't speak to that. Maybe it depends on your lawyer, depends on your judge,
depends on your circumstances. But I think, you know, we had to, you know, a lot of states have
like laws against like revenge board and if you're not sure, but the defense board essentially is
like, imagine you're dating someone, you create adult content together, you record yourself,
photos, images, as you would as a couple would do, you break up. And then after that break up,
one person in that couple and that former relationship decides to share those images
without the consent of the other person. So even though there was equal consent, both parties
consented to the images being taken or recorded at the time, the other person is now not consented
that images being distributed or being used to be harassed or threatened. So like, you know,
get back together or we're going to, I'm going to post it and send this photo to your friends
and family. And that's like the idea of like revenge porn. Because and there are laws against
that. But there aren't necessarily laws that are designed with AI in mind because these images
aren't some people might argue that those aren't real images or those like didn't exactly happen.
You know, there are doctors. I do think we need like strong, clear, you know, sweeping legislation,
not just on a state to state basis. And I think beyond just the US, of course, globally, because
the way that sometimes these companies, so the EDG, they had a corporation in the UK,
but then they were like headquartered here in the US. But they had like people they were making
content of across the world, you know, like people could anyone could join that community.
And so the sort of like borders of where does this responsibility lie needs to be clear because
obviously these communities are globally span in keto, even where people are forming their businesses
on this. There is as far as like other like stuff like the things with the watermark stash,
people can remove watermarks. I don't know that's going to be good enough you crop a little bit out
or something. Some of these services if you do like a free version that you get to see like
their little logo at the bottom. Like, without like, I don't want to name any of them that we
haven't reported on yet. But like some of them like also just like automatically just kind of
like put their little like their products, like some of these companies are proud of it. They
automatically just put their little like, oh, yeah, this image is here was generated on
bubbleblonde.com. They are like, what? Pay extra that are moved. So I don't know. And like,
and Graham, there's like this watermarking bit of it. I don't know if that's going to be enough.
I think it's it's it's a thought that does some little bit. But I don't know if that does anything
if that does anything without consequence. Like I can go on Twitter right now and find
like Nikki Haley is one of the people who are running to be a
GOP nominee for the president's election. And, you know, if I'm finding inocular central intimate
images of her just like not searching for it. But my new I find it on Twitter, just like searching
out research in the platforms themselves. And so like I think, you know, and they're very clearly
people are not ashamed. I think people need more shame. Maybe that's a stronger bit. I'm kind of
like going all over the place. I think I forgot the question here, because there's a lot of questions
in that. But legislation, I do think we need new legislation. I think we need legislation that needs
teeth, not just on. I think we need legislation for the companies. I need legislation for distributors,
people who are producing the content. We need people. And even if you want to go really hard,
people who are like willingly searching out the content and getting to it. Like we think about
the strong laws right now about CSAM. Like there's a case in Canada, like I think three years ago,
where there was where they did arrest a guy who was producing AI CSAM, even though there wasn't
much strong laws or really explicit laws, I should say against AI CSAM, the fact that it was still
considered CSAM. But they were able to still like arrest that guy. But that's like, like when you
read articles or reading about it, you're like, why is there only one guy that's been arrested
in the past three years is generating AI CSAM, but I can go and easily find it unintentionally,
mind you, of looking at AI content on Instagram, on Reddit, on have it at platform. So like,
obviously, the laws are not there, even though they can be applied. And in case I think it needs,
you know, a big task force with some budget that's really dedicated, bring back Dateline, NBC,
you know, bring back like, hey, please have a seat here. Let's talk about you generating
all this AI content of my own, you know. So you might know what I'm talking about. But Speaker 1
I think that's also as researchers, kind of, it's kind of that question, what kind of impact can my research can my journalism have in this particular space? And I think that's why at Bellingcat, with yourself, you know, we decided to contact the Stripe, we decided to flag it to the payment companies and to the company, the social media companies or the spaces that they have these communities, because you can hit them in their wallet, right? That's an easy way to kind of help stop the practice of the company enabling this to a point. And it has had a little bit of an effect on any dream, hasn't it? Do you want to speak a little bit about how they've, how, you know, how they've been able to do that? How they've, how your work has impacted their progress? Speaker 2
Yeah, it was great. I loved making their days miserable. So after, in the course of leading up to publishing, and then of course,
after publishing, I kept a document of like, out of stuff that happened because I wanted to make
sure we kept records of that impact. But Stripe cut off their payment processing, Discord kicked
them off, Reddit kicked them off of their other subreddits. People's accounts got banned. So some
of the moderates accounts, the owners, people that add this, they didn't just like lose the
Discord, they lost their Discord accounts. The PayPal venture kicked them off. They basically lost
all their processing, but then they tried to go and pivot, if you will, to a site called FanView.
And so FanView is like sort of like one of those like creator sites where you could like
pay like a subscription fee or pay like to like pay to get access to content. But they were wanting
to use this people pay for FanView. And then they would get tokens in return through proof of those
purchases. And we contacted FanView and we actually got like email back from I think like your CEO
saying like, oh my gosh, this was not okay. This file is they were like, they do adult content,
their adult content is on their platform. But because they were like trying to like hide what
they didn't consent to like what any dream was doing. So even this adult platform was like,
no. And they kicked those guys off right away. We so basically like every time they try to do
something, they want to get another payment thing we were like on them and just emailed the new
provider they went to and be like, this is what they're doing. The provider was like,
oops, no, we're not touching that. So right now you can only pay with crypto. Unfortunately, crypto
is much, much harder because it's so unregulated and it has so much anonymity. But it did take
past rate like we had to pass their PayPal, a ton for PayPal took action, send them multiple emails,
have other places like encouraging but like, one fortunate thing for most large major payment
processors was it's brought to their attention. They usually are like, it's not a good sign for
us, especially when we are coming to them for comment. Most of the comments we get are all very
generic, like we don't approve of this kind of stuff. But if we're very supposed to like
hear these accounts, like here's the thing that's being done and here's all the evidence of it.
Do you have any comments on that? And then like usually actually gets taken down. So I think what
was nice is that like the digital spaces that these, the people were congregating around to have this
community around, we were able to shut down all major prominent processors, with the exception
of crypto, we got those taken down. Once they decided to like divert to, we got those taken down.
And like their community, I think is only has like 300 people in it. So much, much, much smaller
than what they had in those other spaces. It's seemingly a lots of issues because a lot of people
don't want to pay and use crypto and the guys are doing a terrible job. You know, I think it is,
that's like probably the best case there, which I know is awful because I can still go on there
and people are still sharing content. There are people who are still making non-consensual images
that I would want. I unfortunately can't really go on that one behalf to say like, she doesn't want
this. I need her, like I need the women themselves to also like message that, but like that's like
a huge burden to ask people to be like, go on these sites, go find images of yourself, and then
co-report that. That's like, that's not a sustainable or practical solution, but that's the,
I think the impact on that has been great and it's obviously diminished him, but it hasn't
stopped it. I think for now, the power that we have with reporting on it, talk about publicly,
having outcry puts pressure on the large organizations that do have the power on it
and that's great. And I think we need to work towards it just being an absolute no, like no,
this is just not okay. Yeah, and I think that the fact that they're able to still go on currently
without any further pressure beyond the loose and heavy air riders is not enough. It's like
a little bit sort of like pet peeve of mine that will continue to go on until like they're gone
and dead. Obviously we were able to really, really stall their operations, like the amount of images
that are being shared before to their discourse now has decreased and like, I don't have a
particular number, so it wasn't counting, I mean, it's just being generated daily, but like obviously
it's decreased because like their numbers are not disabled would be on telegram. It is hopefully
slowly dying out and hopefully it'll be gone. And I think hitting them in their wallet, which is where
the incentive is for these companies and these organizations is to make money. It costs them
time and money to be able to generate the images and if they are not making enough money making it,
it will stop. And that is where our power lies, is putting pressure on it. Legislation makes a long
time. We want the legislation to happen until then we've got to put pressure on the payment companies
to really lock down on this. Crypto does not have enough widespread appeal yet for it to be a long
term, I think sustainable option. And I don't think regulations have to get in time to see when it
comes to crypto purchases. So it's just, yeah, that's like, I think. The solution unfortunately,
but our work we're doing here is meaningful. Without the academics, without the journalists,
without the public, without the people interested in this and pressure about it, no change happens
on a large scale. For change to happen, it's not just a couple people that they're going to talk
about. It needs to be like, we're all in on it. So hopefully it's tailored all the photos of stuff,
but it's a lot more public pressure on it. It doesn't just go away in like a week. Speaker 1
Yeah. I mean, the US government has responded to other Taylor Swift stuff this week, haven't they? Kind of expressing outrage and want to do something. We are running out of time very quickly. But I know that a couple of people in the chat have asked this and I wanted to ask it before we wrap. So perhaps you could give a quick answer and then we will end. What are your biggest fears that this will develop into? What are you most worried about that this will become if legislation or more reporting on this isn't done? Speaker 2
No one to say about it. It's kind of insane to me that right now, one of the most wealthy,
popular, famous, wealthy women on the planet, women who's got this name saying Taylor Swift
has enough, only barely more protections than a 14 year old girl in England. We just need
protection for these women because if it could happen to Taylor Swift, it could happen to a 14
year old girl. It doesn't stop anyone. In theory right now, no one's stopping someone taking my
face and making an image of me and distributing on the internet. And there's no consequences for
that. I think if anyone knows a woman in their life, they're victims. I saw someone do this and
they make images of iced tea the other day. Sorry, it was ice cube not ice tea. And made a bunch of
NCI to harass him on Twitter. It's not just women who are victims, it's men too. And it'll
often start off being celebrity men, but then it'll be everyone. So I think having this go unchecked,
I would hate to see a world in which NCII, non-essential intimate images, are so commonplace
that anyone could be a victim. In the same way that you would call, you know, put someone's face
on like a meme to make fun of them. It becomes so commonplace that you could just like sling around,
oh I'm going to make a nude of you and post on the internet just to like, because I didn't like what
you said. You know, like if it's, that's kind of like the love word. If it's easy enough that a 12
year old boy can figure it out, and easy enough that probably someone who's like 80, like my dad,
could like figure it out. It's, the fear is that it just becomes normal. We accept that as that,
oh that's just something that happens on the internet. You posted your image, there's a picture
of you on the internet, oh well now you're a potential victim of AI and NCII and that's just
normal. And I don't hate for that to be considered something that's a normal thing that we see on the internet. Speaker 1
Thank you so much for the insightful discussion, Kolina, and for sharing more about your research and kind of the motives behind it. And thank you to everyone who's tuned in today and for discussing solutions to this in the chat. There's some really interesting conversations going on. Feel free to continue talking, but we're going to end the podcast there. We'll be back in two weeks time with another stage talk and you'll find this recording on our soundcloud and on podcast platforms, hopefully by the weekend. But thank you again Kolina for joining us. Speaker 2
Thanks for having me and see you. We will be having an article on something on this topic in the near future, so look out for that. Speaker 1
Thank you for listening to the stage talk. If you'd like to catch a stage talk live where you can ask the guest questions, join the Bellingcat Discord server by visiting www.discord.gg slash Bellingcat. The music you've heard is titled Dawn by Newer Self and is courtesy of Artlist.