Welcome to the Azure Security Podcast, where
we discuss topics relating to security, privacy,
reliability, and compliance on the Microsoft
Cloud Platform. Hey everybody, welcome to episode
131. This week it's myself, Michael, with Mark
and Sarah. And our guest this week is Tae -Soo
Kim, who's here to talk to us about M -Dash.
If any of you watched Microsoft Build, you would
have seen Sarah actually at the keynote with
Satya, demonstrating M -Dash. But before we get
to our guest, let's take a little lap around
the news. I'll kick things off. Funnily enough,
all my news this week is post -quantum crypto.
What a surprise. So first of all, unless you've
been living under a rock, there was actually
a blog post that came out last week by the Azure
CTO, Mark Rasinovich, called Accelerating the
Quantum Safe Timeline. Basically, Microsoft and
others in the industry, we have moved up the
date for air quotes Q Day. to 2029, which has
been a big change since 2040, which was like
the original kind of Q day back in the day. But
yeah, we're up to 2029 now. Perhaps that may
accelerate. Further, I don't know, but 2029 it
is. On that topic, there was also a executive
order signed by the White House requiring key
encapsulation by 2030 to be quantum secure and
digital signatures to be quantum secure by 2031.
So again, we've got some really interesting dates
that are all sort of coalescing around that 2029,
2031 timeframe. Next one is I wrote a blog post.
The first of many, I hope. It's a technical blog
on post -quantum crypto. And the first one out
the gate was on crypto agility. One of the problems
with crypto agility is every person and their
dog says, thou shalt do crypto agility. And everyone
leaves out one little tiny little part of it.
And that is how. So this particular blog post
actually goes through examples of how Microsoft
Office does it with the OpenOffice XML file format.
Also how Azure SQL and SQL Server do always encrypted.
And also how Azure Storage does client -side
crypto and how they all maintain crypto agility.
So everything is derived from that. There's lots
of code in there as well, as well as anti -patterns.
And the last one is a collection of links in
the show notes. Active Directory Certificate
Services, which, by the way, I worked on back
in the day. I was actually interviewed to be
the original PM for this little product called
Microsoft Certificate Server back in the day.
I ended up working on IIS instead. Anyway, so
Active Directory Certificate Services now actually
has support for MLDSA. So if you're not familiar,
MLDSA is the quantum resistant. digital signature
algorithm that has been blessed by NIST and Azure
Directory Certificate Services supports it out
of the box now. And that is GA. It is a version
that you can run today. So fantastic to see.
I've been experimenting with it and kicking the
tires and it works really, really nicely. Certainly
brought back a lot of memories from when I first
worked on it back in the day. All right, Mark,
what do you got? Thanks. Yeah, that's definitely
bringing back some memories because I was on
the support team helping support certificate
services back in the day. So I just got back
from a long vacation. Most of my news is just
some stuff that I launched just before I left.
There is now a Zero Trust Playbook site kind
of describing the Zero Trust Playbook and all
that and the contents of it and how it's working
and how we're approaching the new security operations
playbook as well is on there. And then I also
launched a YouTube channel. And so I'm going
to be kind of showing my ugly face as I talk
about things on YouTube. So look for more to
come. I just ordered up some new cameras and
lights to do my best to make it more pleasant
for you all. So that's my news for now. Okay,
so now we've done the news, we'll move on to
our guest, Teisu, who is our VP of Security Research.
Welcome, Teisu. Now, I know who you are because...
I've been annoying you on Teams for the last
couple of months about various things. But would
you like to introduce yourself to our listeners?
Yeah, thank you for having me. My name is Taesu
Kim. I have multiple hands. I'm VP at Microsoft
and also I'm professor at Georgia Tech. I actually
recently joined Microsoft, maybe January, I believe.
And today's my... half year anniversary at Microsoft.
So we've been working very hard to launch M -Dash
project. Right. Yes. So there we can launch straight
into it. Congratulations on six months at Microsoft.
And also, I guess some of our listeners will
know, but let's go back to the beginning. What
is M -Dash for anyone who has not heard of it
and been living under a rock? So M -Dash. I recall
it stands for multi -modal agent scanning tool,
I believe. You can come up with a letter between,
you can make an em dash. But the idea there is
that by leveraging AI, particularly large language
model, we're going to analyze the repository
and achieve professional hacking level security
auditing service. By purely leveraging LLM with
the combination of all these traditional tools
together, we're going to push the boundary of
what we call AI scanning tool. What is, let's
go back one. What is a harness, just in case?
Because I have to say that before... I read your
first blog post on M -Dash and I discovered what
it was. I hadn't heard the phrase harness before.
So maybe we should just explain what that is
before I quiz you more on M -Dash specifically.
So large language model, when we say AI tool
is backed by the model itself, there is a scaffolding
around. It means that not just model at the end,
it's just a token by token completion at the
end, right? But it is not enough. for most of
the daily tasks like particularly when you're
analyzing source code token completion at the
end is not enough because you have to look for
another symbols what is definition of the symbol
what is a call graph all these tools around work
together to achieve the goal that you want when
we say hardness it's a driver of the model behind
so you can think of in the simple term co -pilot
CLI is one of the great harnesses generally proposed
is built for writing source code and like for
the software development etc but when we say
harness in the context of security auditing we
provide all this workflow behind the scene in
a way that they can interact with with the model
but we provide all the tools necessary for source
code analysis that's what we mean by harness
So if I could put it another way, just to make
sure I'm fully understanding. So if you like
to ask a model to do something, it's like asking
one person to do the job of a team. And they
can do one task really well, but it's really
hard for them to do like 10 or 12 specialized
things. And so the harness kind of allows the
agents to work together almost like a team with
like a project manager or manager. Is that a
good analogy? I think harness is kind of an overloaded
term. Staffing agent is another way to look at
it. the harness anything beyond the model itself
and get most out of the model. What is this?
You can create the thinking logics in between.
This is another way to leverage the model itself.
Again, at the end, the model just provides the
token completion at the end. But when you say
harness, we're going to maximize the utility
of the model. What are those? It depends on the
tasks. For the coding tasks, People figured it
out, coding agent or CLI work the best. MDash
is a harness, the beyond model, that provide
a way to achieve the security tasks, particularly
source code auditing. So I guess coming back
to MDash specifically then, Taesu, what is, well,
I know you've released a couple of blog posts,
which we will link to in the show notes, but
what is so exciting about MDash? What is it?
do? And why is it better than other similar tools?
Mdash leverages multiple models, what we call
ensembling techniques. Depending on the tasks,
we figure out what's the best way to utilize
the model in that context. We are not liable
to use the best performing model like, say, Optus
and GPT 5 .4, 5 .5 even. we select the best model
for the job. That's one of the big innovations
that we did. And for a particular scanning purpose,
we have hundreds of specialized agents running
currently, but we are not activating them everything
at once because it costs like ten hundred times
more than what we provide with a single agent.
But we have a particular dispatcher routine that
figured it out. Given these functions context,
we just need these four specialized agents. They
look for the particular vulnerability they're
good at. I think the way we construct everything,
we unload many of the cognitive load of the agent
in a way that they're really good at one single
job, particularly finding a specific type of
vulnerability. I think at the end, we actually
achieve really professional -grade auditing services,
what I call pen testing. One of the things that
we demonstrated as part of the blog posting,
and then we actually won MDash against Windows
TCP IP stack. When we say TCP IP stack, it's
an extremely stable codebase you can think of
in the Windows. And they really get regular pentesting
from the external vendors and internal team.
And when we actually won MDash, they were very
skeptical about we find anything with AI -based
tools. They're the professionals. They've been
auditing every year, every quarter. But in fact,
when we run them, we find a number of vulnerabilities
in these stores. Out of the box, I mean, does
the product have a set of air quotes, prompts,
or vulnerabilities to look for? Is that kind
of how it works? It's like, here's sort of an
anti -pattern. Here's a prompt to say, hey, if
you see this kind of thing. then you may have
these kinds of issues in your code. Is that kind
of how it works? And it comes to like a whole
bunch of different prompts to run over the code?
That's great questions. We break many of the
security auditing practices into multiple phases.
Not just one simple way to do the pen testing
these days, giving the coding agent, hey, find
the bugs. That's one way. And agent will figure
it out what to mean by bugs. and explore the
code bases and co -pilot CLI might identify some
of the bugs and provide, hey, this is probably
the report and POC. While we approach, we break
down the details of what pen testers are doing,
how security professionals are approaching this
problem. We start with the repository structure.
We review past the history of the repository.
For example, we are scanning all this Git history.
and figure it out what are the potential CVE
around, what are the security patch that they
create, what are the root cause of them, and
shape the thread model around first. So that,
hey, these are potential attack factors that
developer care about, and this is another scope
of the vulnerability or scope of the repository
that we have to tackle separately. We do the
recomp process even before starting the scanning.
And then once we construct the thread model,
threat boundary, what type of asset we care,
and then we construct the repository structure
so that we can make the rest of the agent easily
explore the code bases by using those guidelines
that we provide. In terms of scanning, again,
we have hundreds of specialized agents looking
for a particular vulnerability that include some
of the variant checker based on the past vulnerability
existing in the previous repository. We are looking
for a similar type of vulnerability. And we also
have a spec checker. We have very specialized
integer all four checker for certain languages.
And then we construct those specialized agent
by using deep research that we are creating inside
our research team. So I have two follow -on questions.
The first one is... If you look at classic static
analysis, like CodeQL, for example, which is
up on GitHub, it does data flow analysis or can
do data flow analysis. You don't have to use
data flow analysis, but you can. And when you
do data flow analysis, you end up getting much
higher quality results because I can say, hey,
this data came from an untrusted source, for
example. So that's question number one. Does
MDash support data flow analysis? That's number
one. And number two is, can I add my own, My
own rules. I mean, can I add my own things that
I care about? I mean, I'll give you an example.
It's not a vulnerability per se, but let's say
I want to migrate a large old code base over
to be post -quantum secure. For example, I want
to support TLS 1 .3 with hybrid crypto. So for
example, MLChem, which is the post -quantum algorithm.
But let's say I... I've got a whole bunch of
old code and I've got S -channel in there, which
is the Windows DLL for handling TLS. And there's
certain settings. If I have those settings set,
there's no way I'm going to be able to run TLS
1 .3 with post -quantum crypto, even if it's
available. So I can define like an anti -pattern.
You know, hey, if you see this kind of thing
in S -channel, these flags being set in S -channel,
then hey, I've got a problem. Can you do that
kind of thing as well? So one, does it do data
flow analysis? And two, can I write my own rules?
That's a great question. In terms of data flow,
in fact, we are not just removing any traditional
scanning tool. For example, CodeQL. If there
is a CodeQL database ready for the repository,
we actually leverage them. For example, if there
is a code graph that is already constructed out
of the CodeQL database, we actually have a...
particular tool that we provide to the agent.
In a way, the agent can navigate the source code
based on the code graph. Not only that, we actually
annotate our source code bases by using data
flow, not just a data flow, or we even resolve
the indirect call in memory -unsafe languages
like C, for example. Many of them are extremely
hard, almost impossible problem to solve in larger
scale, like even with the code pair. For those
cases, we leverage the LLN in a way that, hey,
these are the probabilistic list of the indirect
call target, for example. And this is probably
the data flow that human observe beyond how static
analyzer can provide. So we consider all those
together and create a nicely annotated source
code. And most of the agents are running on top
of those source code that we annotate. So annotate
here is not just a taint analysis, data flow.
and beyond that. And in terms of customization,
we actually provide lots of different ways to
extend our system that include customized thread
model and some of the deployment setting, some
of the things that you care or, hey, these are
the scope that I really want to tackle. I don't
care about the rest of the system. We provide
lots of extension points to the MDash project.
So the developer who really want to scan the
source code, we actually meet their taste. what
type of vulnerability they care, what the places
in the source code they have to audit. I have
one other follow -on question. I realize I'm
hogging all the time here. But you said threat
model. So if I have two API endpoints, and one
is anonymous and remotely accessible to the internet,
as per the threat model, and the other one is
locked behind a firewall with strong authentication,
strong authorization, restricted set of IP addresses,
and so on. So it's got a really small attack
surface. Can M -Dash take that into consideration
as well? Yes. We have multiple sources that we
construct a threat model. For example, if there
is a security guideline in the repository or
documentation, our agent deep into it and summarize
the scope of the threat model. We extract the
list of the public -facing entry point to the
source code. If you're analyzing web service,
these are the starting point. But if you're analyzing
external library... we construct a different
threat model. Typically in the external library,
we consider all the external facing API are entry
point for the attack. So we have all these scope
out even before starting the scanning. I don't
know if people realize just how important and
incredible this is. I think we've just met the
static analysis singularity. The fact that you're
taking all of this data into consideration, taking
output potentially, output from CoQL. you know,
sort of exposure information through a threat
model, as well as the code vulnerabilities. That's
everything, like all basically piled together
into one universal tool. That's, for anyone listening,
if they didn't get how important the last couple
of sentences was, just rewind and just listen
to that again. That's actually really, really
incredible. I think that's great to hear. It's
very exciting, incredibly exciting. Do you want
to hear more exciting news? We run mdash against
TCP IP and we backtest it. You can think of TCP
IP stack on the Windows, not just a complex,
never been used for the training in the model.
This is the first time model and our harness
actually saw the proprietary source code basis.
We have all this history of what type of vulnerability
that we discover in TCP IP stack at Microsoft.
We backtest every single one of them. Discovered
by human, discovered fuzzing, all these previous
tools and stuff. Amdash discovered 100 % of them.
We didn't miss any single vulnerability from
those tools. Let me repeat what you just said
to make sure I understood precisely what you
just said. By the way, the reason I'm hogging
all this is I'm a huge CoQL fan. I'm a huge static
analysis fan. I'm a huge fan of fuzzing. For
those who are not aware, back in the day, I actually
owned all the unmanaged requirements in the Microsoft
Security Development lifecycle. So I'm well aware
of all these things. We use them in anger. So
what you just said, make sure I got this right,
is you looked at a whole bunch of old bugs in
the Microsoft TCP IP stack and running MDash,
you found all those bugs and more. So you took
a vulnerable version, like an older version of
the TCP IP stack. Whether those bugs were found
through code review, static analysis, dynamic
analysis, fuzz testing, someone just found, you
found 100%. This is amazing. It doesn't mean
that we are going to find all of them in the
future, right? But still, it was surprising that
we actually discovered all of them with 100%,
near 100 % accuracy. That is incredible. So what
other languages do you focus on? What other languages
can you look at, like C Sharp, Rust, Python?
So we are a professional hacker. And we care
about the complex infrastructure software such
as memory -unsafe languages. So we start with
those memory -unsafe languages like C and C++
and browsers and whatnot. That was our initial
direction. In fact, we actually support officially
many other languages. You mentioned Rust, TypeScript,
and whatnot. We support all of them. We actually
run M -Dash against Rust standard library. Meaning
we actually found many of bugs. But our tool,
one nice thing about our tool, we are not just
looking for exploitable vulnerability. We actually
find we were pushing our MDash very closer to
the developer lifecycle. So while you're developing,
we're running MDash on the background. We are
running MDash in the latest modification that
you have. We found many of the bugs in the standard
library even, which is well -ordered. So were
the bugs in the Rust standard library in Rust,
or were they calling out to unsafe code from
Rust? It's an unsafe code in the Rust, meaning
that we can crash. Okay. Even if Rust compiler
says it's safe from any memory safety issue,
because we found the bugs in, whenever we found
the bugs in unsafe dialect of the Rust, you can
crash the program. Well, not just crash, right?
Not just crash. I'm going to step in. I'm going
to step in here, Michael. Michael, do we have
to nerd about Rust that much? Yes, we need to
nerd about this whole topic. So Sarah set this
meeting up with Teisu to talk about this stuff.
And I did not realize that I would actually be
so excited about this. I knew, I knew, Michael.
I didn't think you realized because you're too
busy with your post -quantum crypto nowadays.
But I knew that when we set up this call and
do this recording, you were going to completely
nerd out and that I wasn't going to say very
much. Do you want to hear more about the story
behind Rust? Our group at Georgia Tech, about
2022, 2023. we start reviewing all the vulnerability
in the Rust ecosystems all the libraries in Rust
we discover 50 % of the entire Rust unsafe given
all these Rust packages and including standard
libraries and there's a crate inside the Rust
we discover 50 % of all memory unsafe issue in
Rust ecosystems at the moment That includes like
200, 300 CVE from our group. Hey, so what about
deserialization bugs, like in C -sharp? There
is a deserialization agent in our system that
focuses on C -sharp and Java, and these are typical
sources. So if anyone's not aware, deserialization
is actually a really serious problem in air quotes,
memory -safe languages. Huge problem. Yeah? Yeah.
And PHP and any XML -related... serialization
issue even like some serializations in python
like whenever you pick up like there are tons
of issues you can see and we have a very specialized
agent that's just looking for this serialization
serialization issue um i wanted to ask like so
like i'm thinking about myself as head of an
appsec program right putting myself in that persona
like if i've already got sassed tools integrated
in my CICD pipeline and generating bugs and work
and all that kind of stuff. Is this complementing
it? Is it side by side? Is it replacing it? Help
me sort of understand how we think this tool
should be used to actually protect the stuff
that I have source code control over. I think
traditional SaaS tools tend to suffer from high
false positives, meaning that many of the reports
are warning. or informative guideline in a very
simple way. Even though CodeCare pushed the boundaries
significantly, right? You can do all this crazy
data for analysis and so that ultimately you
can reduce down for its positive. But Amdash
tool or most of the AI -based tool have much
deeper understanding. In fact, we talked about
these AI hallucinations maybe five months ago
for a very short period of time. If you see the
bug report from M -Dash these days, we rarely
see almost 0 % hallucinations in the code bases.
Because of all these safeguards and pipelines
that we created, whenever we say bug, it's an
extremely likely bug. But at the same time, we
are not simply saying it's a bug. We provide
a guideline. These are the assumptions that we
make. As far as you enable this compilation flag,
that this is a bug. As far as this precondition
met, because we don't know anything about how
it thinks deployed, hey, your reverse proxy has
some of the gate control and firewall and stuff,
probably not a bug. But as far as attacker can
do A and B and C, this is likely a bug. Because
of all these preconditions that we explicitly
specified, we can drastically reduce down the
force project. And so I would assume like most
customers are very conservative. They're going
to keep the SAST in it for now. They'll do a
comparison. They'll also put the MDash in there.
Now, does MDash, you know, does it actually generate
like a proposed fix or anything like that? Or
like, can you talk about like the workflow integration
part? So one nice feature of MDash is that we
actually reason about each of the findings in
multi -dimensional, meaning that... We are aggregating
multiple bugs as a single source, meaning there
can be multiple symptoms can happen because of
the bug that we found, but likely they can be
just one fix that can address all these symptoms
together. And we aggregate together and generate
a fix or a suggestion so that the user of mDash
is part of the Defender portal. They can even
create a PR, the pull request to the GitHub repository
they connect to. to the Defender portal. I think
they're not just getting the suggested fix, they
can create the PR for you, not just from the
M -Dash, with the tight integration with the
autofix features of GitHub as well. Very cool.
Now, Taesu, you mentioned you'd only been at
Microsoft for about six months, but you and your
team, you have a very interesting... How can
I put it? Origin story. So do you want to tell
us how you came to be? Well, at Microsoft, but
also what you did to end up at Microsoft. That's
great. DARPA has been organizing grand challenges.
You probably heard self -driving car challenge
or urban challenge at the time. And there was
a humanoid competition to build a humanoid robot
for rescue missions. Three years ago, Dalpa announced
a huge challenge. Hey, by leveraging the AI frontier
model, what are the things that we can do in
security? Particularly, how to find and fix those
vulnerabilities in the context of infrastructure
source code, open source project, like Linux
kernel and Nginx and whatnot. Given those complex
VOR source code, how can you find bugs? There
was competition organized by the Alpha and Alpha
H three years ago. It was multi -year journey.
Together with Microsoft and all these frontier
AI companies, including Google, Anthropian, OpenAI,
they sponsor those competition together. They
organize, and our job is just to find the vulnerability.
And we organize team from Georgia Tech side so
that we have a bunch of people working for this
competition. And we won at the end last year.
And we talked about where to join at the end.
But Microsoft suggested a very interesting proposal.
Say, what about you just enable this AICC artifact
into the world of GitHub? We have tons of customers
you can approach. You can think of you enable
this AI -based tool in the context of GitHub.
All these customers, all these users of the GitHub
get benefit from it. what we call Real World
AI Cycle Challenge. So we love this opportunity.
So majority of team members join Microsoft together
with me. So Taesu, that is a very cool story
and we are so glad you picked Microsoft because
I completely agree that it's a great way to impact
the world at large. So say I'm a customer I'm
interested. How would I go about this? Is this
like a private preview thing? How does this work?
So we have a link that you guys can request access
to M -Dash. If you are already a customer of
Microsoft under our Defender product, our customer
engagement team will reach out to you after you
file the application. Awesome. Thank you. Nice.
And we'll put a link in the show notes to the
blogs where you can get the links for that. Well,
Tezu, before we finish, we always ask our guests
a couple of questions, which is, firstly, what
does a day in the life of Tezu look like? Apart
from working on M -Dash, working very hard. I
think my... My day life starts at night, meaning
that before I go to sleep, I make sure my agent
can work for the next eight hours. So I think
about, hey, what is the right way to drive this
agent working while I'm sleeping? And then I
make sure it has a continuous number of work
that they have to do. When I wake up, I check
the completed tasks that I... signed to the agent.
And many of my daily work at Microsoft is how
to handle all these completed tasks and nicely
push to the M -Dash. We're also thinking about
what is the next potential direction that we
can take as a team. We're going to announce a
new interesting team in the coming months. So
we're preparing for those big announcements as
well. And one final thing, following on from
Sarah, is if you had one final thought to leave
our listeners with, what would it be? I think
because of the AI velocity and volume that it
enables, I think it's a really good time to think
about what your attackers and what are your powerful
attackers who have access to AI. I think we are
not just advertising M -Dash itself. It's more
like... is everything changed because of the
recent capability of the LOM. And this is much
more significant than we think because attackers
now have this powerful tool and high intelligence
entity that what you call AGI or not. And once
you release your software, it doesn't have to
be the source code, just binary or website, attacker
can just attack immediately after. All these
powerful attackers can attack your systems. I
think this is an alarming moment for security
researchers and everyone in the world because
of this high intelligence entity now become an
attacker if they want. And it's a good time to
think about how to protect your systems starting
from source code. I truly believe that Defender
has much benefit at the end after these chaotic
periods for a couple of years because we have
now access to those tools as well as the source
code. and as a defender we can adjust the release
day and before putting those source code to the
outside you have opportunity to check and address
all these problem in larger scale thanks to AI.
I think I truly believe this moment that defender
will ultimately win this cat and mouse game and
then we have a more opportunity to protect our
system and have a better life up at the end.
Man I could have been here another couple of
hours I think just talking about it but We don't
have that time. Anyway, look, Tacey, thank you
so much for joining us this week. As with every
episode, I always learn something. But this has
been definitely something that is, I don't just
learn a lot, I'm incredibly excited by it. This
is really cool. And I made a comment somewhat
jokingly about the singularity, but we may be
there. So this is really, really cool to see.
So again, thank you so much for joining us this
week. I know all of us really appreciate you
taking the time. And I know you're incredibly
busy, especially with the product, you know,
essentially just getting started. So all the
best of luck with it. And I hope it goes incredibly
well. I know it will. And to all our listeners
out there, hopefully you're as excited as I am
for this thing. If not, doesn't matter. Stay
safe. And we'll see you next time.