At the tail end of 2023, data poisoning became a major topic of discussion within the generative AI community. Today on The Old and the New, I will explore the generative AI and data poisoning world.
While data poisoning can take many forms, for this episode, when I talk about data poisoning I refer specifically to the process of altering an image’s pixels in a way that the change is imperceptible to a human but messes with the computer’s vision. Thus, if that image is ever scraped into a generative AI model, it makes the AI classify that image as something that the human eye would regard as being inaccurate. This can lead to the generator producing unreliable results. 
Examples of data poisoning are varied: a prompt to generate a balloon could result in an image of an egg, or a hand can be generated as having seven fingers.
This is the idea behind Nightshade. As Melissa Heikkilä a senior reporter at MIT Technology Review explains, “The tool, called Nightshade, is intended as a way to fight back against AI companies that use artists’ work to train their models without the creator’s permission. Using it to “poison” this training data could damage future iterations of image-generating AI models, such as DALL-E, Midjourney, and Stable Diffusion, by rendering some of their outputs useless”
“This, it seems, is the appeal of Nightshade to your average digital artist, who has spent the better part of a year watching their works being used as training data for generative image models like Midjourney and DALL-E,” writes Greg Noone for the Tech Monitor.
Then, why is this so bad? “Poisoning data can make the model sensitive to a malicious data pattern and produce the adversary’s desired output. It can create a security risk where adversaries can force model behavior for their own benefit. In addition to producing unintended and potentially malicious results, a model misalignment from data poisoning can result in business entities facing legal consequences or reputational harms.” writes IBM on their watsonx website.
As previously mentioned, data poisoning can take many forms. For example, poisoning for image-generating AI models could also be linked to the entered prompt keywords.
And researchers at Cornell University have demonstrated that data poisoning can be used by uploading insecure code to platforms such as GitHub with the hope that the code is then used by a code generation application like Copilot in order to create vulnerabilities in systems.
As with traditional poisons, the ultimate effect depends on the dosage. The more “poisoned” images that are fed into the AI model, the more the disruption.
So, how can companies resolve this?
First, they can be more careful about where their sources are coming from and from where they are scraping their data. However, this does mean that the data would be less varied.
Another method is to train multiple models on different subsets of data in order to weed out outliers.
Audits are another option.
But is data poisoning really that much of a problem? From using makeup to prevent surveillance and facial recognition technologies to sweaters that have patterns that do the same, humans trying to deceive and manipulate Artificial Intelligence systems are nothing new. As for whether this is just another issue for generative AI companies to overcome or a way to combat the intrusion of the rights of artists and users is up for interpretation.