Welcome to the Azure Security Podcast, where
we discuss topics relating to security, privacy,
reliability, and compliance on the Microsoft
Cloud Platform. Hey everybody, welcome to episode
124. This week it's just me, Michael. Everyone
else is busy this week. And our guest this week
is Raji, who's here to talk to us about the Microsoft
Security Response Center AI, safety and security.
This is certainly an area that's going to be
very interesting. But before we get to Raji,
I want to just talk about one little news item
that I have, and I literally only have one news
item. And that is, in fact, I really do wish
Sarah was here because this is right up in her
wheelhouse. That is, we now have in general availability
deployment safeguards for pod security in AKS,
or more accurately, the pod security security
standard. So it turns out, I didn't know this,
but now I do. There are some pod security standards
upon the Kubernetes website that have different
profiles. There's privileged baseline and restricted.
And there's now abilities built into AKS that
allow you to deploy these in a consistent way
and to enforce pod security configurations across
clusters. I understand the words. I kind of understand
the security implications of it. But again, I
really wish Sarah was here to go into a bit more
detail because it's her thing. All right, that's
all I have in the news. So now let's turn our
attention to our guest. As I mentioned, our guest
this week is Raji, who's here to talk to us about
MSRC AI safety and security. So Raji, welcome
to the episode. We'd like to take a moment and
introduce yourself to our listeners. Absolutely.
Thank you so much, Michael, for having me. Exciting
to be here. I've always been a big fan of the
podcast, so really excited to be here, specifically
talking about something really near and dear
to my heart, the MSRC AI safety and security
team. So basically, my role at MSRC is that I'm
a principal engineering manager, and I have been
with Microsoft completing three years in Feb
of this year. It feels like a long time, but
I've been here for three years. My team mainly
is responsible for driving the end -to -end lifecycle
for vulnerabilities response for Microsoft AI
services and products. So this is anything to
do with Copilot, Copilot Studio, Consumer Copilot,
M365 Copilot, all the different AI interfaces
you deal with as you are working with Microsoft
products. So my team actually, we also lead the
engagement with a global AI research community.
This spans both academia and industry. And prior
to Microsoft, I have held other security leadership
roles and worked in multiple domains of security
at Apple and Adobe. So yeah, it's fascinating
for me because I've seen the transition to the
cloud, multi -cloud, and now AI and AI agents.
I just want to end by saying with great connectivity
comes great risk. I would say with the ambiguity
of an LLM comes great risk, but let's just leave
it at that for the moment. I think it's great
to see a specialized branch of the Microsoft
Security Response Center coming into focus. Obviously,
it is an area that needs to be treated in its
own way. So maybe worthwhile just spending a
couple of minutes and just talking about the
difference between the Microsoft Security Response
Center, the classic MSRC, and MSRC AI. Has anybody
been here three years? I've been here just a
little bit longer than that. Yeah, 33 years.
But anyway, that's another discussion. But yeah,
so I was there at the beginning with the MSRC.
I remember the MSRC first forming. People like
Steve Lipner and Scott Culp will remember some
of those names who started the MSRC back in the
day. And obviously the big focus back then was
on literally security vulnerabilities in products
like Windows and Office and SQL Server and Exchange
and Visual Studio and so on and so forth. So,
I mean, is this like an adjunct to the MSRC?
Is it part of the MSRC? How does that all fit?
Yeah, great question. And it always fascinates
me, you know, people I meet either at Microsoft
or across the industry. There's like so many
people just across different companies right
now who all started their careers at MSRC. So
it's really fascinating to hear the stories and
kind of, you know, the... period for which this
specific part of Microsoft has been, I would
say, continuing to grow and bring together the
nuance of coordinated vulnerability disclosure,
specifically when it comes to external researchers.
Just fascinated by that. Specifically to your
question about, you know, AI versus MSRC AI versus
MSRC. Yes, MSRC AI is part of the MSRC. We have
two... big groups I would say. One is the security
assurance engineers who are basically working
on triaging the bugs which come in. We also look
at ensuring that it is transferred to the right
teams within Microsoft and get the right level
of visibility. Then we also obviously go through
and ensure that this is in line with our AI bug
bar. We can talk a little bit about that in just
a little bit. But then we assess those bugs to
ensure what kind of severity it is in the space.
And so that's basically what this team has been
doing. And we've also built a relationship with
external researchers, specifically who are doing
research. in AI safety and security. Some of
the names are Johan Rehberger. He has published
the article on Embrace the Red. He has a great
blog on that. And Zenity, we worked quite closely
with Michael Bargeri and his team. So those are
some of the well -known researchers, but we also
have others we've worked on with them. Yeah,
I think the thing that makes it interesting is
that it's such a... brand new area, relatively
speaking. I mean, security vulnerabilities, SQL
injection, memory corruption, et cetera, et cetera,
have been around forever. AI -specific vulnerabilities
are relatively new in the overall scheme of things.
However, in many cases, what is old is new again.
I mean, if you look at prompt injection, you
could argue it's just input validation. Just
on steroids, though. Yeah, exactly. Well, you
know, as you and I were talking about earlier,
you know, there's a security mantra, which is,
you know, thou shalt not mix the data plane with
the control plane. And yet with prompts, that
is precisely what we're doing, right? Yeah. Absolutely.
And that leads to the different types of attack
surfaces, as well as almost sneaky attacks in
some ways. I mean, we all know, everybody who's
listening knows about AI and generative AI and
the impact this has had on our lives from 2023,
at least, since ChatGPT has been released. There's
a complete era booming. Everybody, businesses
are wanting to adapt and obviously it's the right
thing to do because it brings so much additional
goodness and efficiencies. But like I said before,
as the goodness comes along, there's always some
risk involved. And here we have expanded the
threat surface and the threat landscape in completely
new areas. Yeah. I mean, at the end of the day,
I mean, let's be brutally honest. At the end
of the day, you know, for every use case, there's
probably N abuse cases where something can be
used for good. And then all of a sudden, you
know, well, if you do it this way, it can actually
be used for bad. Yeah. So just going back to
the MSRC versus MSRC AI. I mean, so the MSRC
handles classic air quotes, classic security
vulnerabilities and the MSRC AI handles. safety
-related vulnerabilities. And security, AI security
vulnerabilities as well, yeah. Yeah. So a classic
bug is a security bug, but it might not be a
safety bug necessarily, right? Yes, it may not
be. So the word safety, if you think about where
it actually came from, right, it actually came
from ML. When you go back to before generative
AI, when a model was inputting something which
was not in line with what they wanted, then that's
when you would call it a safety. It's a safety
issue. It's not aligning. It was like an alignment
problem. But in those cases, the model developer
and the model user was the same person. But that's
not the case anymore with generative AI. The
model creator is completely different and the
user. obviously uses it in different ways. And
so that's why this has become a bigger challenge
right now. It's not so much misalignment because
the misalignment ultimately ends up with socio
-technical issues, which just makes the category
of AI risks just a bigger blob. It's not just
confidentiality, integrity, availability. It's
also some other... Some other things, like we've
actually published that on the AI Bug Bar. I
can share the link with you, Michael, so you
can add it to when you publish the podcast. It's
basically, we have content -related issues, and
that actually spans a myriad of different types
of safety issues. Some of them are information
integrity, where AI has generated content which
is false or misleading. Then it could be inappropriate
language or information integrity. It could be
things about malicious users. It could also be
CBRN, a harmful use of CBRN to demonstrate information
which shows information which is not readily
available. By the way, CBRN actually stands for
chemical, biological, radiological, or nuclear.
So this is a category of when the model produces
outputs, which are in these categories of chemical,
biological, radiological, nuclear related. category
then we say hey it's it's the category cbrn but
there is more nuance to specifically categorizing
at cbrn that it should not be readily available
publicly like if someone can get the same information
in a google search or a bing search then that
is not brand new we are actually looking for
uplift what uplift means is that what is the
model providing completely new which the user
will not be able to get it otherwise. Yeah. But
then you've got other topics, right? You've got
like sexual content, physical harm. Yeah, self
-harm. Yeah. So these are all like the new categories,
what we have seen with AI safety. This is new
overall in the industry, right? And we are looking
into this because to ensure that we are thinking
about risks more holistically when it comes to
AI. So what are some other examples? I mean,
that's image manipulation. What about any other
examples? I think this is an area where a lot
of people, it's brand new to a lot of people,
and it's an example of stuff you don't even know
you don't know. So any more sort of context you
can give around that I think will be really useful
to our listeners. Yeah, some of the safety issues
we have seen is folks sending us Actually, even
with indirect prompt injection, the way we categorize
something as a security issue is as long as it
has a security impact. So if the specific security
impact could be something like a data exfiltration.
If you can showcase in your bug, as you're sending
us a bug, if you showcase that, hey, it has a
security impact, that means I'm able to exfiltrate
data, then we categorize it as a security issue.
And in a safety issue is anything where you can
show indirect prompt injection, but you're not
able to demonstrate. data exfiltration, for example,
then we call that a safety issue. So that's one
way, one of the things we have seen. The other
stuff we've also, other kind of issues we have
seen is people just manipulating pictures and
sending us you know, like different types of
images with some political figures and things
like that. And those have been safety issues
as well. And most of the time it is because a
classifier, we need to go update a classifier.
So you mentioned very early on that there is
an MSRC AI bug bar. Can you give us examples
of, actually before we do that, why don't I explain
what a bug bar is at Microsoft and then you can
sort of chime in with. what the AI bug bar kind
of looks like. So the bug bar at Microsoft has
been around for a long, long, long time, and
it's a way of categorizing overall risk slash
severity of a vulnerability. So for example,
a remote code execution exploit that is easy
to pull off remotely and gives you system on
a Windows box. That's more than likely critical.
Now, there may be a lot of extenuating circumstances
that reduce the potential damage. Perhaps it's
only a local -only attack. Perhaps it doesn't
give you a system. Perhaps it's on a feature
that's disabled by default. All these other things
and other defenses come into play, and that may
be rated a low or a moderate. I'm just making
that up. So the bug bar has been around at Microsoft
for a long, long, long time. So what does the
bug bar look like for MSRC AI? Yeah, it's a great
question. And it's actually exactly what you
talked about, Michael. It is basically a way
of categorizing the different vulnerabilities
we are seeing in the space of AI, both security
as well as safety, and then talking about severity,
specifically when it comes to... safety, there
is no severity. It's more focused on in -scope,
out -of -scope, because this is basically, we
are trying to identify whether it's, what is
the safety issue? And there is nothing about
in -scope or out -of -scope. Sorry, nothing about
specific severity because of this emerging space
we are talking about. But when it comes to security,
it is very similar to other MSRC bug bars, specifically
inference manipulation. We talk here, some of
the things are prompt injection. The indirect
prompt injection has been the largest category
of bugs we have seen. and if it's a zero click
attack like basically if uh without a user performing
any any action uh then you know uh and the data
is exfiltrated through uh through a prompt then
we call that a critical a severity bug and if
we need a specific interaction that means somebody
needs to click a link and then that allows to
exfiltrate data then we call that an important
severity bug so that's kind of how we've been
categorizing the the ai vulnerabilities We've
also called out input perturbation, which is
basically any time a specific classification
model is misclassified or, you know, those are
what we call input perturbation. But we haven't
really seen these kind of bugs submitted to us.
Mostly it is in the space of indirect prompt
injection. And indirect prompt injection is where
you embed essentially a prompt injection -like
attack in a document that is read by the LLM.
Yes, yes, that is indirect prompt injection,
or it's basically the top LLM vulnerability,
even in OWASP. top 10. This is basically where
the attacker is able to say, you know, summarize
an email, and then the email actually has the
prompt injection embedded in it. So when the
user who's interacting with the LLM says, hey,
summarize my email, the prompt injection automatically
gets triggered. So I just realized while we're
talking about prompt injection and indirect prompt
injection, not everyone will probably understand
what a prompt injection attack is. Do you want
to just give a quick example? Yeah. So, for example,
let's assume somebody sends you an email. The
attacker can send you an email with just a normal
email in your inbox. And as part of the email,
there is a line that says, send this data, send
sensitive data to attacker. website or attacker
email right and the user who is just on their
emails uh using copilot might say hey summarize
my email and when they do summarize my email
in our llm it behaves differently right it interprets
a prompt or an instruction inside an email like
a prompt, and LLM, as useful as it wants it to
be, basically takes that prompt and summarizes
it and basically sets forward that action. What
it does is that the user is basically just summarizing
the email, but in the background, the LLM is
already sent. some sensitive information to the
attacker email or to whoever is basically sent
that prompt injection to them so that's why it's
indirect prompt injection the llm doesn't differentiate
between the instructions coming from the user
directly interacting with the llm or or instructions
embedded in documents it is leveraging to summarize
your instructions hopefully that's helpful I
think if people listening to this podcast don't
know what prompt injection is and indirect prompt
injection is, this is as classic. It will end
up being as classic as SQL injection, memory
safety issues, cross -site scripting. Absolutely.
So everyone needs to be aware of what they are,
especially if you're building your own tools
on top of perhaps your own LLMs or third -party
LLMs or something. I think it's just critically
important that you understand what these... these
vulnerability classes are. In fact, most importantly,
what the mitigations are. And I think it's fair
to say that this whole area is constantly evolving.
The thing that sort of concerns me is if we take
SQL injection as an example, we know how to fix
that. We know precisely how to fix that. You
just use parameterized queries, basically, for
any untrusted data. The problem solved. The problem
with LLM prompt injection is there's no... single
mitigation that fixes the problem. Because as
you noted, we're sort of mixing control constructs
with data constructs, and we don't know which
is which. So it becomes very, very difficult.
Yeah, absolutely. And I think ultimately, I think
security basics... apply whether it is ai or
non -ai so i would say like continue to really
think about defense and depth and defense and
depth is no longer optional it's actually really
important and crucial for uh specifically mitigating
something like indirect prompt injection you
need to be thinking about hey what is my content
security policy And you need to have multiple
layers of defense. So it's almost like the Swiss
cheese analogy of like one, okay, it doesn't
catch here, does it catch the next one? And that
is the best way to mitigate these. Yeah, don't
forget least privilege. Like if you're running
AI agents, if you have an agent that runs with
godlike permissions over your environment. and
the LLM decides to start doing its own thing,
then you can start doing all sorts of damage.
So yeah, least privilege is storm. It's just
interesting to me that we talked about least
privilege since the 60s and 70s, and here we
are again talking about least privilege with
a brand new type of technology, which is generative
AI. Yes. And I think there's another aspect of
this as well, which is basically, you know, we
have to be prepared for what can go wrong. I
mean, and this is like classic security mindset
of threat modeling, right? Like, well, what can
go wrong? I mean, like, this is how we should
be thinking about our systems. So really, really
digging deep into secure by design and threat
modeling right in the beginning, right, as you're
getting this. as you're thinking about ideas
and kind of coming together as a security team,
product team, you know, like and working together
to ensure that we are meeting customer needs
from a customer perspective, but also really
thinking about what is the how can we continue
to keep provide that safety and security aspect
of things as well as, you know, really focusing
on or like almost like. assuming things will
go wrong and uh preparing for what should you
do when things go wrong because it's too late
to figure out things at that point so at least
if you have like um here is my team i will be
who will be looking into these or um you know
how do you pull into incident response or what
are the new terminologies in incident response
Some of those things is kind of keeping in mind
as you're thinking about AI systems. Yeah, and
that's why I think it's so cool that Microsoft
has now set up a Microsoft Security Response
Center division or part of the MSRC just to focus
on AI. This is a constantly evolving area and
response is a critical part of responding to
AI vulnerability. So this is really good to see.
Yeah, absolutely. And it also helps to continue
to think about it from a... you know we have
a lot of processes internally when it comes to
pre -deployment right like before deployment
like a specific uh you know there are a lot of
different controls including a deployment safety
board which actually stamps the approval before
something actually goes out the door from microsoft
from an ai perspective and then on the other
side msrc is working with these researchers who
are seeing the bugs in production right and that
kind of provides that feedback loop to continue
to better our defenses and as well as think about
things in angles which we had not thought about
ourselves so different perspectives bringing
in completely new different perspectives yeah
and i think that's something that's really important
as well as you know serious issues, critical
issues go through a complete root cause analysis
to find out what happened and what we can learn
from this. And I think that virtuous cycle is
critically important. So again, it's good to
see that we're doing this in the AI space as
well. All right, Raji. So one question we always
ask our guests is, what does a typical day in
the life look like for Raji? I feel like the
answer is... for most of your guests must be
there is no typical day in security that doesn't,
it doesn't really, it's very different every
day. And I think that keeps me kind of. excited
and motivated but if i have to really think about
hey what does my typical day look like it looks
uh it basically is working with uh both external
folks as well as internal teams and continuing
to uh kind of bring together some of these perspectives
of new vulnerabilities coming in and helping
our internal teams really think about it from
uh you know how does that fit into their threat
model and how do they mitigate that, as well
as how do they take the feedback from these different
research to basically better their mitigations
in the future, as well as working with our...
bug bounty teams uh because we also you know
some of the some of the all of the work we are
working here uh in ai security specifically have
uh have uh rewards attached to it so we've uh
and recently in black hat eu Microsoft basically,
MSRC basically announced a standard award policy,
which basically says that you don't have to be
in scope anymore. There is no in scope. But as
long as you have a critical or an important severity
bug, then you get paid a standard award, which
is really cool. Yeah, that is cool. So those
are some of the things I do work with. Very cool.
All right. So let's start to bring this episode
to an end. So again, another question for you.
If you had one piece of advice to leave our listeners
with, what would it be? One final thought. Yeah,
one final thought is really, you know, the Microsoft
AI security and safety team is always looking
for AI security researchers or safety researchers,
you know, to work with. And if you find any bugs
in Microsoft AI security products or services,
please reach out to the MSRC at the researcher
portal. And you can submit the box to us. And
we would love to continue to engage with you
and collaborate with you to ensure that we can
make Microsoft AI products and services better
for our customers. Very cool. All right. So,
Raji, thank you for joining us this week. I know
you're very busy, and that's a very interesting
area of research. So it's great to see we have
people like you working in this area. And I always
learn something new. Again, I'm a classic security
person, but over the last couple of years, I've
also learned a great deal about AI security and
safety. And again, it's amazing. The things you
don't know, you don't know. So again, thank you
for joining us this week. And to all our listeners
out there, we hope you found this episode useful.
If you're in the AI space, you really need to
understand AI safety and security, especially
if you're building products. on top of AI. So
stay safe, and we'll see you next time.