Welcome to the Azure Security Podcast where we discuss topics relating to security, privacy,
reliability and compliance on the Microsoft Cloud Platform.
Hey everybody, welcome to episode 80.
This week it's just myself, Michael and Sarah and our guest this week is Matt Zorich who's
here to talk to us about a team at Microsoft called DART and Microsoft Instant Response.
Before we get into the discussion with Matt, let's take a little lap around the news.
Sarah, why don't you kick things off?
Okay, well, you know me and my news, I always love to talk about something related to my
baby, Microsoft Sentinel.
This is tangentially related, which is Azure Monitor logs now have better table level RBAC.
So you may remember if you've tried to use table level RBAC in Azure Monitor, it hasn't
been the most flexible and it's had some restrictions on how you could use it.
But we now have public preview for much better and much more granular RBAC.
And it will also allow you to do it for custom tables, which is something we couldn't do
before.
So we'll have a link in the show notes.
If that's something that you looked at in the past and it didn't quite work for you,
go and have another look because it's definitely much improved on what it was.
And that's it for me this time, Michael.
So over to you.
Yeah, I've got three items.
They're actually all sort of somewhat related.
They're all to do with web application firewalls at some level.
So the first one is in public preview, we now have sensitive data protection for app
gateway web application firewall logs.
So you can enable this and certain things that may be deemed a little bit sensitive.
So for example, JSON argument names, request IP addresses, request header names, they will
be sort of scrubbed with a whole series of asterisks.
So again, things like IP addresses, what could be potentially passwords, those kinds of things,
they can be all sort of erased.
So that way there's loads of chance of PII or other kinds of sensitive data appearing
in the web application firewall logs.
Next one is also in public preview.
We have the default rule set 2.1 for regional WAF with application gateway.
So there's a whole bunch of rules that we have that are available for the web application
firewall and the different versions.
So now what we've done is just sort of standardized on a whole bunch of different types of rules.
And this one is version 2.1, so includes things like cross-site scripting, includes things
like well-known JavaScript attacks, remote code execution, session fixation, SQL injection
right in my wheelhouse, those kinds of things.
So these are the default rules.
I mean, obviously you can go and change them.
We want to sort of continue sort of raising the bar in terms of core security.
Otherwise if I said there was three items, that's only two items allied.
So there we go.
So with that, let's turn our attention to our guest.
As I mentioned at the beginning, our guest this week is Matt Zorich, who's here to talk
to us about the team at Microsoft called Dart and its role in incident response.
Matt, welcome to the podcast.
We'd like to take a moment and introduce yourself to our listeners.
Yeah, thanks for having me.
Actually, yeah, to clarify, just like a lot of things at Microsoft, we have also gone
under a rebranding.
So we are now Microsoft Incident Response, previously Dart.
And yes, I'm a consultant on the Microsoft Incident Response team and I'm based out of
Australia with Sarah and basically we're Microsoft's customer facing incident response team.
So if one of our customers is compromised and they feel the need for a full investigation,
then we can be brought in to do that.
And we'll do the investigation and some hardening work and kind of uncover the story of the
attack and try to give the customers some good advice to hopefully not be a return customer.
We always say we're the one team at Microsoft that can say, we don't really want to get
your call ever again.
I find the stuff that Dart does fascinating.
And I know you're not able to talk about specifics or mention specific customers for obvious
reasons.
Tell us about, obviously, this is very fluid and changes all the time, but tell us a bit
about some of the trends that you're seeing in your incident response land at the moment.
What are the call outs from customers that you're seeing a lot of at the moment?
Because I'm sure if I were a listener, this, well, I like to hear this from Dart and I'm
going to keep calling you Dart.
I know you're not Dart.
You're Microsoft incident response.
But I am always fascinated by the kind of things that you folks do and what you see.
So yeah, what are the trends?
What are the things you're getting called out for a lot at the moment?
Yeah, you're right.
It's kind of fluid and I think trying to understand why that's the case.
Certainly there's no reason why it changes or moves over time.
It just so happens to be the engagements that land with our team or things going on.
Certainly in the last couple of months, we've been engaged in a lot more kind of cloud engagement.
Certainly I have.
So when we do investigations, they may be on-premises focused or they may be cloud,
so kind of Azure and Azure Active Directory and Microsoft 365.
From my last few months, they've certainly skewed towards Microsoft 365 and Azure.
So I'm seeing a lot of those kind of come across my way.
What the reason is, I don't really know.
It could be any number of things.
But I think we're, you know, to kind of guess, I guess I would say or try and kind of
hypothesize about it is I think, you know, maybe post-COVID, now we're in this hybrid
work kind of work style again.
We still have a lot of people working from home.
So kind of the first entry point they might have to your data is like a Microsoft 365.
So maybe ATT&CK is a kind of understanding that people are working at home, maybe not
protected in the same way that they would be in a traditional corporate network.
So they're kind of attempting to fish or compromise like a cloud user and then kind of pivot from
there.
But, you know, we see all kinds of ATT&CK still.
So we still see a lot of ransomware.
We see a lot of like data exfiltration and things like that.
But they certainly appear to have skewed a little bit more towards the cloud in the last
few months for whatever reason.
So you said that you're seeing a lot of incidents in cloud.
Can you talk a little bit more about the kind of incidents you're seeing in cloud and M365?
Yeah, yeah, certainly we can talk to kind of the broad tactics that we see.
So look, I think a lot of people know if you read kind of any incident reports is that
often like large scale compromise begins with a regular user.
And that's certainly what we see reflected.
So it might be like a non-privilege, just a regular user that gets fished or they get
kind of the MFA bombing that you hear about or, you know, they click on a dodgy link or
they have a weak password.
However, kind of the attacker gets that foothold.
And then what we're seeing is them kind of use the M365 kind of platform and just search
within a user's mailbox.
So they might search for other credentials.
They might search for VPN information and things like that to kind of progress their
attack.
I think ultimately towards the end of the attacks in the cloud, we're seeing kind of
a mixture of things.
So it can be very financially motivated.
So, you know, if an attacker can take over an Azure Active Directory tenant, they might
then look to pivot into Azure and spin up kind of crypto mining infrastructure and mine
crypto on the customer's dollar.
Obviously the customer gets left with the bill so that can be very quick and very kind
of expensive.
The other one we're seeing is kind of the data exfiltration and extortion type of attacks
where an attacker might get full tenant level access to everything that's in SharePoint
and OneDrive and Teams and everything else and kind of take copies of that data and then
try to extort the company saying, you know, I've got X amount of data, pay us this amount
before we leak it.
So they're kind of the two strains I would guess we see that the crypto mining and the
extortion in, you know, with that data theft.
For those two strains, are you able to tell us a bit about kind of how you would deal
with them?
Yeah, certainly.
So look, the crypto mining side of things is usually quite easy to detect because generally
the customer gets an alert or something saying you might have a huge bill that's been rung
up because I think the crypto mining kind of the people that are skewed towards crypto
mining know that they're on a very short time frame that they're going to be detected and
they're going to get kicked back out.
So really it's kind of like a smash and grab.
You know, we might only have access to this this tenant for a few days.
So we're just going to spin up as many VMs as we can to mine crypto before we're back
out the door with the extortion and those kind of more sneaky tactics.
What we try to do when we are first engaged with a customer that's kind of in the middle
of that is try to take back control of the tenant from kind of a global administrator
level.
And we call that kind of positive identity control.
So we want to make sure that all the global admins, they're under our control, that there's
no persistence mechanisms left behind so the attacker can get back in.
Once we're kind of happy that the environment's stable and it's in our control, then that's
when we start doing kind of the deep dive and you know what did the access, sorry, what
did the actor access?
Did they access data?
You know, have they created accounts to get back in later and things like that.
But yeah, the first step in these kind of compromises is always try to, we call it positive
identity control because someone's always in control of your tenant.
We just, we prefer it's you that's in control of your tenant, not the bad guys.
And we kind of pivot from there and really with these investigations, it's where the
breadcrumbs lead us because no two investigations are the same.
And look, we certainly see the same tactics and the same motivations, but they're constantly
evolving.
There's always novel ways and novel things that they're trying.
So we just try to get all the data.
We talk to the customer.
We try to get logs.
We try to get information and it's a bit like a detective story, right?
You're just trying to piece together what happened.
You're also trying to, it's always hard to distinguish between sometimes threat actor
activity and users doing silly things activity as well.
So often we have a lot of questions for the customers to clarify those things.
It's interesting you just say, you know, data X fill and Bitcoin mining or some kind of
digital currency mining.
I mean, that's the result, you know, the end result, right?
Someone obviously got into the system and so on, but it's interesting you just say things
like fishing or clicking on a bad link.
It's almost like what's old is new again.
I mean, how long have we had these kinds of attacks?
And we're still seeing them today as the way the bad guys get into the environment, right?
I mean, so I mean, am I kind of right in that or are we seeing updated variations of these
kinds of attacks or is it really just the absolute basics?
It's a bit of both.
Yeah.
Look, you're certainly seeing some of the, yeah, look, people still click on fishing
links.
That's the reality of it.
But we are seeing some more kind of novel and modern attacks, so certainly things like
social engineering are becoming more popular.
So you know, an attacker might email the help desk of a company and pretend to be a, you
know, a user of that company and say, I've got a new mobile because I lost my old one
or my old one broke and here's my new number.
So can you update it in this system?
And then the attacker can use the self-service password reset function to kind of take control
of that account.
We're also seeing, you know, you see things, not just traditional fishing, but you might
see smishing.
So, you know, SMS is sent to users and credentials lost that way.
So it's still true.
It's still fishing, I guess, but there are some modern kind of takes on it and things
like that.
But for social engineering, that's the hard one to kind of protect as well because you're
relying on your help desk and your IT teams to kind of, it's a very soft skill to understand
whether something's not quite right and kind of challenge users and things like that.
So it's, I appreciate it's very hard to stop as well.
So when you said smishing, so using SMS as an attack vector.
Yeah, so is this because I'm thinking of multi-factor stuff, right?
Something about multi-factual authentication using just SMS messages.
Is that the attack we're talking about or are we just talking about in general, hey,
this is Bob from support.
Please log in and click this link and change your password.
Because the big thing in multi-factual authentication is not to use SMS, right?
It's to use an app instead or something like that.
Yeah.
So it tends to be the latter of that, which is, yeah, send some kind of message out, which
is, you know, update your payroll information or this is IT security and can you confirm
your details or whatever they look like.
And then to combine with that, you might get some kind of social engineering to get through
the MFA process.
So they might call them up and say, you know, this is Bob from the help desk or it could
be they often do like what we call like kind of spamming different MFA methods.
So they might send a message and then they might do a prompt and then they might do a
phone call and et cetera, et cetera, until one kind of gets through.
And once they've taken control of that account, they can put their own MFA method in as a
persistence mechanism.
So they only really need to get through just once.
And then, you know, they register their own phone or their own authenticator app.
It's the law of large numbers, right?
It's like if you hit enough people, someone's going to click on the link, right?
A very small percentage of a very large number is still a large number.
So that's what's working in the attackers advantage, right?
They don't have everyone.
Not everyone has to click on the link or respond to the text message or respond to the email.
All it takes is one person ultimately.
So it's the law of large numbers.
You know, again, a small percentage of a very large number is still a large number of potential
victims.
Yeah, definitely.
That's what I always say.
I always say like being a defender or a blue teamer is really the hardest job in cybersecurity.
Like I personally believe because you need to be perfect every day.
The attackers and the bad guys really get as many shots as they want.
It doesn't matter if they miss.
They just like you say, more messages, more emails.
Eventually someone's going to click.
The law of averages says they'll get someone eventually.
And that's like, you know, you laugh at some of these emails and you'd be like, oh, why
would anyone ever click on them?
But it doesn't cost anything to send emails.
You know, someone will click.
I think I should bring that up about, you know, essentially being the defender.
The very first book that I ever wrote when I was at Microsoft was called Designing Secure
Web-based Applications for Windows 2000 back in the day.
And in there I actually wrote a small section called The Attacker's Advantage and the Defender's
Dilemma.
And that's exactly the problem, right?
Is the attacker just has to get lucky once.
The defender has to protect 100% of the time and be correct 100% of the time.
The attacker just has to sneak in once, right?
There's one little chink in the armor and there you go.
Yeah, exactly.
And the defenders often have to deal with the reality of business, which might be, you
know, they don't want to change.
They don't want to deploy MFA.
They don't want to do things that inhibit productivity.
And they have these, like the defenders have all these corporate rules to follow and budgets
and things like that.
Attackers don't have any of those.
There's no scope of work they can really do whatever they like.
So that makes it doubly hard.
I often hear people say, well, I'm not a really big, well-known, you know, Fortune 500 company.
Why would someone try and attack me?
Why would someone try and hack me?
And I just wondered if you had any thoughts on that.
You know, the types of customers that you actually end up helping, are they much bigger,
high profile, well-known brands?
Or do you go to a lot of, you know, smaller customers that might be the type that would
say, hey, why does anyone care about me?
So I think, yeah, no one's definitely no one's immune from cyber attack.
For sure.
I can definitely tell you that.
Like whether it's small business, medium business or massive enterprises might be, you know,
the attackers might be motivated by different things depending on the size of the organization.
Our team probably skews towards bigger customers, but that's probably more because, you know,
we're Microsoft and we're a big IR team and we kind of play in that market.
But there's plenty of other cybersecurity incident response firms that kind of are engaged
with small, medium businesses as well.
I don't think, like I said, I don't think anyone's immune.
And a lot of the attacks are opportunistic.
So if they, like you said, they send out all these phishing emails or smishing and they
get credentials for a particular company and they're able to access it, that might be the
reason to go after that company rather than being particularly targeted.
So yeah, every range.
And I think the ones you read in the news and things skewed towards the bigger companies
as well, but you're likely not hearing about the smaller and the medium sized enterprises
that are getting attacked.
It's just kind of not as newsworthy, I guess, unfortunately.
But you look at the stats for things like business email compromise and business email
compromise is not flashy.
It's been around forever and that's simply compromising someone's user account and logging
onto their email and then you might change the invoice on an email or something and to
your bank account details, which sounds like they might be $20,000, like a $20,000 invoice,
which doesn't sound like much in the scheme of things, but the stats for things like business
email compromise, if you look at them, it's wild.
It's billions of dollars that costs every year and $20,000 to a massive company is a
drop in the ocean.
And that could be devastating to a small or medium business that's kind of month to month.
So yeah, like I say, no one's immune.
What things would you recommend based on what you've seen that people should focus on in
terms of security controls to reduce the likelihood of a breach?
I mean, I know that in a compromise, I mean, I know that's what we all do every day, but
I'd be keen to hear from an incident responder.
What do you think the things are that people should really focus on?
Yeah, definitely.
It's going to be a very, very boring answer.
Honestly, it's the basics.
And like I always say, don't get the basics confused with it being easy because they're
two different things, especially in massive, massive organizations.
I'm not sure if you've ever seen the Microsoft Cybersecurity bell curve.
I'm a big fan of that.
You know what?
I have that nowadays, no joke, in pretty much every presentation I do.
We'll link to it in the show notes.
It's in the digital defense report.
Sorry, Matt, go on.
The reality is, is like being an incident responder is you see that it is actually true.
It's the basics.
So I think it's like 98% of attacks will be stopped by, you know, MFA, least privilege,
attaching vulnerable servers and kind of applying zero trust and having like a modern anti-malware,
like a defender for endpoint or something like that.
And I remember seeing a phrase once, it was like, basically what we're after is brilliance
in the basics.
And that's like a term that's always stuck with me and I've really enjoyed it.
And I use it occasionally is, and that's what it is.
It's doing those basics, doing them across a whole organization.
And like I say, like I've been on the other side and worked as like a blue teamer in big
organizations and it is hard, like hundreds of thousands of users or millions of users
and deploying MFA to them.
Like we understand it's difficult, but there's no secret source.
If you do do it, it'll reduce your risk.
And that's what we always say is there's no 100% secure.
It's risk mitigation.
And if you've got 100,000 users and you can deploy MFA to 90% of them, that's a huge win.
Even if you can never get the last 10,000 or whatever it is, it's all reducing risk.
What are the common mistakes you see that people make that subsequently end up being
breaches?
I realize this is kind of sort of very closely linked with what I've just asked you.
Yeah, I think that the constant ones we see in this is both in on premises and kind of
in Azure Active Directory is probably too much privilege for users that don't really
require it and kind of not understanding what that privilege can grant users to.
So you might have like a service account that's in a domain admins group.
We see that all the time or a service account that's in global admins in Azure Active Directory.
Those kind of service accounts and things like that are often not secured the same way
like a human identity would be.
And that's kind of the nature of them.
You can't really put MFA and things on service accounts.
And often credentials for those things are left kind of just hanging around in clear
text, whether that's in scripts or that's the old passwords.txt sitting in your One
Drive or sitting in your SharePoint because they're very if you can find them, if you
search for, you know, I've got my passwords list that I go to.
And so I search for passwords, then an attacker can also search for passwords and uncover
them as well.
So that's definitely the most common one.
And I think the other one is basically not enforcing kind of stricter controls on your
tier zero and your very, very high privileged account accounts.
So like we said earlier, like we appreciate that it's very hard to deploy all these security
controls across massive organizations uniformly.
But your privileged accounts and your tier zeros and your domain admins and your global
admins, that should be a very small subset of your users.
And those users should be held to much stricter controls.
Look, I think everyone on this call and probably everyone listening is you'll never stop regular
users being compromised.
It's just going to happen.
Like we said, whether it's phishing, whether it's smishing, whether it's downloading cracked
software, that's okay.
Like that can be inconvenient for that one user.
What ultimately we're trying to do is stop the kind of the compromise of a single user
turning into a compromise of all our users.
And that's where we need to, if we harden our tier zero, we can deal with the user installing
malware.
We can give them a wrap on the knuckles and send them a new laptop.
That's no big deal.
But we don't want that to be like the first domino that ends up in a huge incident.
Just last couple of questions for you, Matt.
Hopefully none of our listeners will have to do this.
But if anyone out there did need the services of Microsoft IR, what is the process to get
in touch if a customer thinks they need help?
Yeah.
So there's a few ways.
So you can log a severity A case through your unified support.
And we kind of on the Microsoft IR side, we kind of keep an eye on those.
They don't always require full incident response.
Sometimes the cert team can handle it or it's not actually a security thing.
It might be an operational thing.
And we'll kind of keep an eye on any that may come our way.
Or we believe that the customer may need full incident response because they don't maybe
we don't appreciate kind of the situation they're in.
We also were available under retainer.
So if you're interested in having Microsoft IR just available, it's like, you know, pick
up the phone, something's happened.
We can kind of mobilize very quickly.
There's also just like on a link on our website, and I'll find you the link just basically,
you know, we're suffering a security incident at the moment.
And that comes through to our team and we reach out and kind of see what's happening.
And again, if you if we feel you need full incident response, and we can try mobilize
a team and get going.
So lots of ways.
What, in your opinion, what is the value of some kind of protected workstation like a
protected access workstation or privilege access workstation, I should say, or, you
know, secured access workstation?
In other words, people shouldn't, you know, not going into production, like not administering
production from their from their normal work machines, rather they're going in from, you
know, machines that are designed to do a specific task, which is administration.
And that's it.
But you can't do your email, you can't go and look on Facebook or any of your socials.
Is that something that that sits well with customers?
Or is it a bit of a hard sell?
It's one of those things it's we always laugh about it is like the saying is like, you know,
never waste a crisis.
So post incident or when we're engaged, things like the privileged access workstations and
MFA and things like that, they all they all become a really good idea.
It's very it's very easy to kind of invoke change during the incident or in the brief
period after.
So often customers are very open to that idea because they've seen the ramifications of
not not having a privileged access workstation.
And it is like a fantastic control.
And it lends on to what I said earlier about protecting those tier zero accounts as well
as you can.
So, you know, privilege access workstation is ultimately trying to stop the kind of spread
of privilege credentials.
Because if you think about like if you've got 10,000 workstations and you've got a domain
admin and if a user has a problem with their workstation, they log on to the workstation
as their domain admin account and kind of help the user out.
What you've done is actually left your credentials on that end user workstation and an end user
workstation is probably not going to be secured as well as a domain controller or things like
that.
So if an attacker was to kind of compromise that end user device, there's a chance that
they could compromise the domain admin credentials.
So a poor or, you know, a secure access workstation or whatever you'd like to call it basically
says if you're going to access a tier zero service like a domain controller, you can
only access it from this particular device and it's got its own controls.
And like you say, it doesn't browse the Internet.
It doesn't have a mailbox so it can't be phished.
It's hardened.
It's whatever controls you want to put around it and that way we're not leaving domain admin
credentials kind of scattered around the environment in lower tiers.
So yeah, it's a very, very strong control.
Definitely worth doing and I think it's gotten easier as well in time.
I think you've got things like you can have like virtual secured workstations now running
in Azure and things like that.
So it doesn't always have to be, you know, another laptop or another device and things
like that.
Yeah, I did some work a few years ago with a healthcare company and they were moving
some of their workloads to Azure.
And basically it was, if you access, even try to access production in Azure without
using a privileged access workstation, it was actually a potentially fireable offense
because they needed to make sure that, you know, access to the environment was through
a secured environment, not from Bob's laptop, you know.
Yeah, that's right.
Yeah, that's exactly right.
Yeah.
And with, you know, things like conditional access in Azure, I do really help that now
because it's good to have, you know, the policy of course saying that you can be fired if
you do that, but you really, what you really want is like a hard technical control because
people will be lazy and eventually they'll probably kind of, you know, if you have an
incident and you put a paw and you put all these cool rules in and that's awesome and
kind of 12 months later, two years later, the security, you know, the compromise you
had is kind of a distant memory and then the bad habits sneak back in.
That's just human nature.
Yeah, it has to be a strong preventative control, right?
Like not a responsive control.
It has to be something that prevents someone from accessing the environment unless they're
using a privileged access workstation or similar.
Yeah.
It's good to know that you're seeing people actually adopt it, but only after that you're
like, you know, the incidents happened, but I guess that's better than better than never.
So Matt, one thing you may be aware of is that every time we have a guest, we always
ask them for a final thought.
So if there's one final thought you'd like to leave our listeners with, what would it
be?
Yeah, I think my final thought kind of speaks to what we were just talking about actually
in that, look, in our experience, we see the don't waste a crisis mentality and like it's
great that companies go through and make the changes after the fact, but would really like
to have people to do it beforehand.
And really it's easier, it's cheaper, it's less stressful to make those changes before
an incident than when you're in the heat of it.
And kind of leading on with that is that a saying we have is that don't, you know, don't
let perfection be the enemy of good.
It doesn't need, your security controls don't need to be perfect.
You don't need to be a hundred percent coverage of MFA or it doesn't need to be that.
It can be just iterative.
So you can deploy MFA to your admins, to your privileged account, to your board members,
things like that.
And it's all risk reduction.
So trying to kind of architect solutions for a hundred thousand users, there's always going
to be exclusions and that's not a problem.
But don't let those exclusions kind of get the way in the way of rolling out these things
to the majority of users.
I think that would be my final thought is like I say, it's all risk reduction.
If you can reduce risk a little bit, that's a really good win.
Excellent.
Thanks for that.
So it makes absolute sense.
And with that, let's bring this episode to an end.
So Matt, thank you so much for joining us this week.
I always learn something on these episodes and this was absolutely no exception.
And to all our listeners out there, thank you so much for listening.
Stay safe and we'll see you next time.
Thanks for listening to the Azure Security Podcast.
You can find show notes and other resources at our website, azsecuritypodcast.net.
If you have any questions, please find us on Twitter at Azure Setpod.
background music is from ccmixtor.com and licensed under the Creative Commons license.