Welcome to Tech Unplugged podcast. It's great
to be here. Today, we're going to be diving deep
into the world of AI security. Yeah. You've shared
some really fascinating material with me, and
our mission today is to unpack all of this. What
does all of this mean for our listeners out there,
whether they're tech professionals, business
leaders, or just someone who's really interested
in staying safe in this age of AI. So we're going
to be exploring some of the unique risks that
come with using AI, how security experts are
tackling these challenges, and what our listeners
should really be aware of to protect themselves
and their organization. So no need to feel overwhelmed.
We're going to be extracting the essential insights
that you need right now. Yeah, sounds good. You
know, I think a lot of people, when they hear
AI risk, kind of lump it all into one big bucket.
But it can be really helpful to think about AI
risk in two distinct categories, safety and security.
OK, so safety first, right? What does that mean
when we're talking about AI? Yeah, so AI safety
is all about making sure that the AI model itself,
like the actual model, isn't producing harmful
or unintended outputs. What kind of things are
we talking about here? Well, think about biased
results, for example. OK. We've seen AI generate
really weird visual representations or even data
that reflects biases that were present in the
information it was trained on. Oh, I see. And
here's the thing. This isn't always about developers
intentionally creating a biased model. Right,
right. It often comes down to problems with the
training data itself. Maybe it overemphasizes
certain viewpoints, or maybe it doesn't include
a diverse enough range of information to be truly
representative. Kind of like the old saying,
garbage in, garbage out. Makes sense. So how
does security differ from safety when it comes
to AI? So with AI security, we're taking all
of those concepts that we've learned from traditional
cybersecurity over the years. Things like protecting
our networks and applications. And we're applying
them to the unique challenges that AI brings
to the table. What's interesting here is that
when you're actually testing an AI system's defenses,
something we call red teaming, it's not always
the same as your typical penetration test. How
so? Well, you have to really dig deep and understand
what the client is most worried about. Are they
concerned about a customer -facing chatbot? Or
is their concern more about critical infrastructure
that's powered by AI? Each scenario demands its
own very specific approach. So when you're trying
to assess AI risk, you can just look at how the
AI behaves. You also need to look at the security
of the entire infrastructure, all those data
pipelines that surround it. It's a two -pronged
challenge. That distinction between safety and
security is really useful. Now you mentioned
red teaming. What kind of vulnerabilities are
security experts actually finding when they're
poking and prodding these AI models? I keep hearing
about this thing called data poisoning. Yeah,
data poisoning is a big one. And it's the legitimate
concern. Imagine this. Someone intentionally
injecting bad data, malicious data, into the
massive data sets that are used to train these
very large AI models, the ones that we call foundation
models. These foundation models are trained on
information that's scraped from huge portions
of the internet, including things like online
forums and code repositories. Wow. So if an attacker
is able to slip in some of this poison data,
and if they know that it's there, well, they
might be able to retrieve that data later. Or
even worse, they could manipulate the model's
behavior by using really carefully crafted prompts.
And that's a big problem. You can see how that
could be exploited, right? Yeah, that's scary.
Yeah. And here's the thing. The integrity of
an AI's knowledge base is directly tied to the
security of its training data supply chain. And
that's a rest that we don't really see in the
same way with traditional software. There's like
a whole new level of vulnerability then. And
what about this indirect... prompt injection.
Yeah, so indirect prompt injection. That's another
sneaky one. Imagine using AI, let's say a large
language model, to create these super convincing
phishing campaigns. Think back to all those old
phishing emails we used to get. The ones with
all the obvious mistakes. You know, the bad grammar,
the weird formatting. AI can make all of that
go away. It can make phishing attacks way harder
to spot. But here's where it gets even more worrying.
If an attacker can actually get their hands on
someone's login credentials, and those credentials
belong to someone who works at a company that
uses internal AI models, well, that attacker
now has access to this super powerful assistant.
It's like having a sidekick that can dig into
all of the company's proprietary data. So it's
like they've basically given a hacker an AI assistant.
Exactly. And if that AI has what we call agentic
capabilities, meaning it can actually use tools
and do things within the company's systems, well,
then that's a huge risk, an immediate risk. That
paints a pretty alarming picture. It does. And
it's really important to grasp the difference
between, let's say, a researcher demonstrating
a data poisoning trick in a lab setting and the
very real threat this poses to organizations.
What might seem like just a fun experiment could
actually be setting the stage for much more complex
attacks. You know, these nested attacks where
if an attacker gets even a tiny foothold, well,
they might be able to completely take over an
organization's systems. It's that serious. Now
you've mentioned agentic AI and you've talked
about these persona prompt patterns. Can you
explain what those are? Sure. So when we talk
about persona prompt patterns, we're basically
describing a technique where you give an AI a
specific role to play. You set up some boundaries
within your prompt. OK. For example, you might
tell an AI to act as a seasoned cybersecurity
expert. and explain this vulnerability in detail.
You're giving it a persona to adopt. People are
even using coding personas, where they give the
AI really strict rules about what kind of code
it's allowed to generate. Like they might say,
you can only write code that's well -documented,
verified, and comes from trusted sources. It's
like giving the AI a hat to wear during that
particular interaction. So it's like giving the
AI a very detailed job description. Exactly.
You're setting expectations. But, and this is
crucial, Even when you use these fancy personas,
you have to remember that AI systems are still
what we call non -deterministic. That means that
you might get slightly different answers or results
each time. even with the same prompt. Oh, interesting.
So while personas can definitely make things
more predictable, they're not a foolproof solution.
OK, good to know. And just to clarify, when I
talk about agents, I'm talking about something
more advanced. I'm talking about AI systems that
can plan and carry out tasks over time. They
might use tools, interact with different systems.
That's a whole other ballgame. And that technology
is still very much under development. So this
whole idea of controlling the information that
these AI models learn from and then generate,
That raises some important questions. It does.
And what I find really fascinating is the parallel
between this and the idea that if you control
information, you can control what people think.
We've seen that explored in literature. We've
seen it in history. And today, we see accusations
being thrown around about how search engines
and social media platforms might be manipulating
the information we see. It makes you think. It
does. And AI can be influenced in the same way.
Whoever controls and prepares the data that's
used to train an AI, they have a lot of power.
And that has big implications for what we consider
to be true, what we consider to be accurate.
That's why it's so important to have human oversight
of AI outputs. It doesn't matter what you call
it, human in the loop or human at the helm. The
point is, you can't just blindly trust everything
that an AI spits out. Right. You've got to have
someone checking its work. Absolutely. Even if
the AI is using your own internal data, There's
this technique called Retrieval Augmented Generation,
RHG for short, where the AI pulls information
from your knowledge base to answer questions.
Even then, you still need to double check. You
can't assume it's always going to be 100 % right
or unbiased. Humans are still essential. So with
all of these potential vulnerabilities, how do
you actually go about finding and fixing security
issues in AI systems? Do you have any real -world
examples of what this looks like? Yeah, absolutely.
I was actually involved in a red teaming project
for a really big organization. And what was interesting
about this particular project was that it was
highly confidential. We had zero direct communication
with our developers or their security teams.
But even without talking to them, we could tell
that they were addressing the vulnerabilities
we were finding. We could see that the model's
behavior was changing every day as they implemented
fixes based on what we were uncovering. So you're
basically playing this cat and mouse game with
their security team. Kinda, yeah. And it was
really cool to see that our work, even though
it was behind the scenes, was having a real impact.
It just goes to show you how important proactive
AI security work is. It sounds like a real detective
story. Now how does this AI red teaming compare
to the traditional kind of red teaming that people
might be more familiar with? Yeah, there are
definitely some overlaps. In both cases, you're
trying to find weaknesses in a system's defenses.
That's the core idea. But with AI red teaming,
there's often more of an emphasis on trying to
understand how the model behaves. You know, can
it generate harmful content? Can it be tricked
into saying things that are biased? How does
it respond to weird inputs? Traditional red teaming,
on the other hand, might have a broader scope.
They might be looking at things like network
vulnerabilities, physical security, even social
engineering tactics. It really depends on what
the goals are for that particular engagement.
So let's go back to that safety versus security
distinction. It seems like they're tackling different
aspects of the overall risk picture when it comes
to AI. Exactly. Think about it this way, AI safety
is all about the model file itself, how it works,
whether it spits out bad stuff. Okay. AI security
though, that's about the bigger picture. It's
about the entire ecosystem that the AI lives
in. Got it. So that includes all the networks
it connects to, all the applications that are
built using the AI, and even the entire process
of developing that AI. That's where MLCCOPS comes
in. MLCCOPS. It stands for Machine Learning Security
Operations. It's basically about making sure
that security best practices are baked into how
machine learning models are developed, kind of
like how DevSecOps works for traditional software.
OK, makes sense. So if you're a company and you're
building a chatbot or an AI -powered agent, well,
your software engineers need to be thinking about
security right from the very beginning. And it's
not just about the code that a company writes
themselves, right? They could also be introducing
risks by using AI models or data that come from
other places. That's a really important point.
It's all about the source. Where did you get
your AI models from? Where's your training data
coming from? If you just download a pre -trained
model from a site like Hugging Face, well, you
don't know for sure if it's secure. That's kind
of worrying. It is. And it's actually only very
recently that Hugging Phase teamed up with a
company called Protect AI to start scanning their
models for malicious code. That tells you something,
right? People are starting to realize just how
risky this AI supply chain can be. But here's
the thing. Even with all this scanning going
on, there's still no guarantee that a model is
completely safe. So it sounds like we need a
totally new way of thinking about application
security in this age of AI. Absolutely. We're
talking about a whole new field here. AI application
security. Traditional app security that was designed
for deterministic software. The kind of software
where you put in the same input and you get the
same output every time. But AI models, especially
the ones that can create stuff, the generative
ones, they're often non -deterministic. You can
give them the same prompt over and over and you
get different results. So we need new ways of
figuring out where the weaknesses are and how
to fix them. I see. And here's another wrinkle.
AI applications. They can keep learning. They
can change even after you've launched them. Oh,
wow. So you might end up with vulnerabilities
or weird behaviors that you'd never see in traditional
software. It's a whole different ballgame. So
when we talk about AI application security, what
are some of the vulnerabilities that we're concerned
about? What are we actually looking for? Well,
the list is long and it keeps getting longer.
On the safety side, we're worried about AI that
might generate toxic content, you know, stuff
that's offensive or inappropriate. We're worried
about AI that might leak sensitive information.
And of course, there's the whole issue of malicious
use. Could someone use AI to do bad things? We're
also concerned about deception and manipulation.
Could AI be used to trick people? And then there
are the broader social implications. Could AI
be used in ways that harm society? That's a lot
to think about. It is. And then on the security
side, we have things like prompt injection, where
someone tries to hijack the AI by feeding it
carefully crafted prompts. This could be direct
or indirect. We have data -praising, where someone
tries to corrupt the AI's training data. And
then there's jailbreaking, where someone tries
to break out of the AI's safety controls. And
then there's the whole issue of privacy attacks.
Could someone use AI to spy on people or steal
their data? So it's a complex landscape to say
the least. Given all of these potential threats,
how do organizations even begin to approach securing
their AI systems? Where do they even start? The
key is to shift left. That means that security
needs to be part of the conversation right from
the very beginning of the AI development process,
not just an afterthought. So that means you need
to be really careful about where you're getting
your training data from, where you're getting
your pre -trained models from. Are these sources
trustworthy? Makes sense. And you should be scanning
these models for any signs of malicious code,
any signs of unsafe behavior. And here's something
that might surprise you, even if you're taking
a good foundation model. you know, one that's
been carefully vetted, and you're tweaking it
for your specific needs, you still need to check
it. You still need to validate it for both security
and safety. Why is that? Because sometimes, when
you tweak a model, you can accidentally break
the safety features that were built into the
original model. Oh, wow. Yeah. And to do this
validation, you often have to use an API that's
basically a way of interacting with the model
without actually seeing its guts. It's like a
black box. I see. And you should be doing this
validation every single time you update the model.
It's an ongoing process. OK. So you've built
your AI. You've checked it for vulnerabilities.
And now it's out in the wild. How do you keep
it secure once it's actually being used? Constant
vigilance. You have to be on the lookout for
any new vulnerabilities that might pop up. And
you have to keep up with the latest threat intelligence.
What are the bad guys doing these days? Right.
And here's the thing. With traditional software,
You can often just patch a vulnerability, you
know, release an update. But with AI, it's not
that simple. You can't just slap a Band -Aid
on it. You have to think about controls. Controls.
Yeah, like an AI application firewall, for example.
This is something that can block bad requests
from getting to the AI, and it can stop the AI
from generating responses that are unsafe. And
all of the logs from this firewall that can be
fed into your other security systems so that
you can track what's going on and respond to
incidents. It sounds like AI itself is also becoming
a valuable tool in the cybersecurity world. Oh,
absolutely. AI is being used more and more to
help organizations improve their security posture.
We're seeing things like machine learning and
deep learning being used to analyze huge amounts
of data. Things like network traffic, how applications
are being used, even browsing habits. Right.
And by looking for patterns in this data, AI
systems can spot anything unusual that might
be a sign of a security threat. We're even seeing
generative AI being used to take all of this
complicated security data and translate it into
plain English. So instead of having to decipher
all of these technical logs and reports, security
teams can get clear, actionable recommendations.
And studies have shown that companies that really
embrace AI and automation, well, they can detect
threats and respond to incidents much faster.
It sounds like AI is becoming a real game changer
in cybersecurity. It is. But it's not a silver
bullet. Like any powerful tool, it can be used
for good or for bad. So it's a double -edged
sword. Exactly. While AI can make us more secure,
it can also be used by attackers to make their
attacks more sophisticated, more targeted. You
know, things like highly personalized phishing
campaigns. And, of course, the AI models themselves
are new targets. Attackers are looking for ways
to exploit them, to manipulate them. So it's
like a constant arms race. It is. Now, shifting
gears a bit, let's talk about data privacy. How
does the rise of AI impact our privacy? The ERASP
AI Security and Privacy Guide does a great job
of highlighting the growing concerns in this
area. One really important principle is use limitation
and purpose specification. Basically what that
means is that if you collect data for one specific
reason, you shouldn't use it for something completely
different. So if you're collecting biometric
data for multi -factor authentication, you shouldn't
be using that same data to build a marketing
profile. Exactly. That would be a violation of
that principle. And a lot of data privacy regulations,
like GDPR, they require you to have a clear legal
basis for processing personal data. Makes sense.
And there are some AI practices that are considered
so risky that they're being restricted. Like
in the EU, they're working on the AI Act. And
one of the things that it does is ban the use
of AI for individual criminal profiling. Oh,
wow. Yeah. And it's not just about personal data.
Even how you use non -personal data can be problematic.
It can lead to unfair or harmful outcomes if
you're not careful. That's why we're seeing new
techniques being developed, things like data
enclaves and federated learning, which are designed
to give you more control over how data is used.
Fairness and AI is another big topic these days.
It is. And fairness, when we're talking about
AI, means that you're handling data in a way
that people would expect, you know, in a way
that's reasonable. And it means that you're not
discriminating against certain groups of people.
OK. And here's an interesting point. Even if
your AI model is accurate, it can still lead
to privacy problems. How so? Well, imagine you
have an AI system that's designed to detect fraud
and is really good at its job. It's very accurate.
But what if it makes a mistake? What if it flags
someone as a fraudster when they're not? that
could have a huge impact on that person's life.
It could affect their credit score, their ability
to get a loan, all sorts of things. That's a
good point. So accuracy is important, but it's
not the only thing that matters. You also need
to think about fairness. And there are lots of
different ways to measure fairness in AI, but
there's no one -size -fits -all solution. It's
often a balancing act. You're trying to make
your AI as accurate as possible, but you also
want to make sure that it's not discriminating
against anyone. And sometimes you might have
to accept that you can achieve both goals at
the same time. Exactly. Sometimes the most responsible
thing to do is to say, you know what, this AI
model is too risky. We're not going to deploy
it. So given the huge amount of data that AI
models are trained on, what does that mean for
our privacy? That brings us to data minimization
and storage limitation. Basically, the idea is
that you should only collect the data that you
absolutely need, and you should only keep it
for as long as you need it. OK, less is more.
Exactly. And whenever possible, you should anonymize
the data. And that doesn't just mean removing
someone's name and address. You might also need
to remove other identifying information. And
sometimes you need to reduce the level of detail
in the data. You might not need to know someone's
exact age. Maybe you just need to know their
age range. I see. And once you've finished using
the data, you should delete it. And you should
restrict access to the data. Not everyone in
your organization needs to be able to see everything.
Right. Need to know basis. Exactly. And there
are also some really cool new technologies being
developed, like distributed data analysis and
secure multi -party computation. These allow
you to analyze data without actually having to
see the raw data itself. That sounds pretty amazing.
It is. And it has huge implications for privacy.
Transparency is another big issue when we talk
about AI. Absolutely. Transparency is all about
building trust. If people don't understand how
AI is being used, they're not going to trust
it. Makes sense. And a lot of data privacy regulations,
like GDPR, they have specific requirements for
transparency. You know, you have to tell people
how you're using their data. You have to give
them a copy of their data if they ask for it.
And you have to tell them if you make any major
changes to how you're processing their data.
Right, keep them in the loop. Exactly. But transparency
isn't just about the end user. It's also about
internal transparency. Your own employees need
to understand how AI is being used. Okay, so
good documentation is important. It is. And you
need to be able to track how decisions are being
made. You know, if an AI system makes a decision,
you need to be able to explain why it made that
decision. And that explanation needs to be understandable
to humans, not just to other AI systems. So it's
about accountability as well. Exactly. Now, individuals
have certain rights when it comes to their personal
data. How does AI affect those rights? Yeah,
so these are often called privacy rights. And
they give people more control over their own
data. Like what? Well, for example, you have
the right to access your data. You can ask a
company, hey, what data do you have about me?
And they have to tell you. You also have the
right to data portability. That means you can
ask a company to give you a copy of your data
in a format that you can use. So you can take
your data with you if you switch to a different
service. Exactly. And then there's the right
to erasure, also known as the right to be forgotten.
You can ask a company to delete your data. Interesting.
And you have the right to correct any inaccurate
data. If they have your age wrong, your address
wrong, you can ask them to fix it. Makes sense.
And you also have the right to object to the
processing of your data for certain purposes.
So if you don't want them using your data for
marketing, you can tell them to stop. Exactly.
And this is all really important in the context
of AI because AI systems are often trained on
huge amounts of personal data. So you have to
think about how you're going to respect these
privacy rights. And that might mean retraining
your AI models if someone asks you to delete
their data. It could, yeah. Data accuracy is
another big issue when we talk about AI and privacy.
Absolutely. If your AI systems are making decisions
based on inaccurate data, well, that can have
serious consequences for people. Right. Wrong
data, wrong decisions. Exactly. And those decisions
could be about anything. You know, it could be
about whether someone gets a loan, whether they
get hired for a job, whether they get approved
for insurance. So it's really important to make
sure that the data you're using is accurate and
up to date. And that means having processes in
place to check the data, to correct any errors,
and to make sure that you're getting your data
from reliable sources. Exactly. Garbage in, garbage
out, right? Right. And finally, there's the issue
of consent. How does consent play into all of
this? Consent is all about giving people a choice.
Do they want you to use their data or not? And
a lot of data privacy regulations, like GDPR,
they say that you need to get consent before
you can use someone's data. But that consent
has to be freely given. It can't be coerced.
And it has to be specific. You can't just say,
hey, can we use your data for anything we want?
Right. You have to be clear about what you're
going to use the data for. Exactly. And the person
has to understand what they're agreeing to. So
no burying it in a long, complicated privacy
policy that nobody reads. Right. And the consent
has to be easy to withdraw. If someone changes
their mind, they should be able to say, hey,
I don't want you to use my data anymore. And
then you have to delete their data. Right. Ideally,
you should also retrain your AI models so that
they're not using that data anymore. Now, we
talked earlier about all of those security threats
that AI systems face. How do those threats relate
to privacy? So this brings us to model attacks.
Basically, if someone can hack into your AI system,
they might be able to steal the data that the
AI is trained on. And if that data is personal
data, well, that's a privacy violation. Right.
It's a data breach. Exactly. And there are all
sorts of ways. that attackers can try to steal
data from AI systems. They might try to figure
out if a particular data point was used to train
the model. They might try to reconstruct the
training data from the model itself. It sounds
pretty sophisticated. It is. And it's a real
concern. So how can companies protect themselves
against these model attacks? It all comes down
to good security practices. You know, you need
to have strong encryption in place. You need
to control who has access to your AI systems.
And you need to be constantly monitoring for
any signs of intrusion. And you need to be aware
of the different types of model attacks that
are out there so that you can defend against
them. Exactly. Now, switching gears again, you
mentioned earlier that how AI models are stored
and shared can introduce vulnerabilities. I've
heard about this thing called Pickle. Can you
explain what that is? Yeah, so Pickle is basically
the default way that Python programmers save
their AI models. It's like putting the model
in a jar so that you can store it or share it
with others. Okay. But Pickle has a bit of a
security problem. Yeah, the way that Pickle works,
it allows you to embed code inside of the model
file. And when someone loads that model file,
that code gets executed. So if an attacker can
sneak some malicious code into a pickle file,
they can potentially take over someone's computer.
Exactly. It's called remote code execution. That
sounds bad. It is. And it's a real vulnerability.
Now, to be fair to Pickle, this isn't necessarily
a problem if you're only ever loading Pickle
files that you've created yourself or that you've
gotten from a trusted source. OK, so it's about
trust. Exactly. Yeah. But if you're downloading
models from the internet, you need to be very
careful. You don't know who created those models.
You don't know if they've been tampered with.
So what can you do to protect yourself? Well,
the best solution is to use a different format
for saving your models. There's a format called
safe tensors that's designed to be more secure.
OK. But if you have to use pickle, then you need
to be extra vigilant. Only load files from trusted
sources. And if you come across a model that
you think might be compromised, report it to
the creator or the maintainer of that model.
Makes sense. So considering all of these complexities,
all of these potential threats, what does a good
AI security solution look like? What should organizations
be doing to protect themselves? Well, ideally,
you want a comprehensive AI security platform,
something that can give you visibility into all
of your AI systems, that can help you find and
fix vulnerabilities, and that can help you set
up good governance policies. So it's not just
about technology, it's also about processes and
policies. Exactly. And you need to start thinking
about security from the very beginning of the
development process. Secure by design, that should
be your mantra. Secure by design, I like that.
And you need to have good testing procedures
in place. You need tools that can specifically
test AI applications. Right, because traditional
security tools might not be enough. Exactly.
And you need to be thinking about your supply
chain. Where are you getting your components
from? Are they secure? Right. It's all connected.
It is. And OWASP, they have a really helpful
resource called the Top 10 for Large Language
Model Applications. It's a great starting point
for understanding the most common vulnerabilities.
So organizations need a strategy, a plan for
how they're going to secure their AI systems.
Absolutely. They need to figure out what their
risks are, what their needs are, and then they
need to put the right solutions in place. And
that includes training their employees. Absolutely.
You can have the best technology in the world,
but if your employees don't know how to use it
securely, well, it's not going to do you much
good. It's the human factor. It is. And here's
an interesting development. AI itself is being
used to make traditional security tools better.
Oh, really? Yeah. So you're seeing things like
AI -powered static analysis tools, AI -powered
software composition analysis tools. These tools
can provide more intelligent analysis. They can
give developers real -time feedback. They can
even suggest safer ways to write code. So AI
is making security tools smarter. Exactly. But
we also need to be careful about bias. Bias in
the AI tools themselves. Yeah. If the AI tools
are trained on bias data, Well, they might perpetuate
those biases. So that's something that we need
to be mindful of. That's a good point. And we're
moving towards this idea of adaptive application
security. That's basically about having a unified
view of your entire IT environment. You know,
all of your code, all of your cloud infrastructure.
And it's about integrating security throughout
the entire development lifecycle. So it's a more
holistic approach to security. Exactly. And some
people believe that will eventually be able to
use AI to automate a lot of the security casts
that are currently done manually. Oh, wow. So
AI could potentially fix code vulnerabilities
automatically. That's the idea. That would be
pretty amazing. It would. And, of course, you
also need to secure the AI systems themselves.
Right. So it's not just about securing the applications
that use AI. It's also about securing the AI
infrastructure. Exactly. You need to have strong
encryption in place. You need to have access
controls. You need to be monitoring for threats.
And you need to be constantly evaluating your
AI systems to make sure that they're performing
as expected, that they're compliant with regulations,
and that they're not introducing any new risks.
So it's a multifaceted challenge. It is. Now,
what about all of those companies that provide
managed security services, you know, the MSPs,
how do they fit into this AI -driven world? That's
a good question. And it's one that a lot of people
in the industry are asking right now. We're seeing
some new companies pop up that are basically
saying, hey, we can automate all of your security
operations using AI. You don't need an MSP anymore.