This is the AmpicCyber Critical Assets Podcast. Each episode we cover important OT and ICS
security topics with an eye towards standards and regulation to keep you ahead of your adversaries
and your auditors.
Hey everyone, welcome back to the Critical Assets Podcast. I am Patrick Miller and with
me today I have Leslie Carhartt, also known as Hacks for Pancakes and I am just thrilled
to have you on. I've got so many questions, I'm so interested in what you're doing. And
let's just start by telling us a little bit about yourself for the audience that doesn't
already know who you are.
Hi everybody, my name is Leslie Carhartt and thank you for having me, Patrick. It's nice
to be here. I am an OT cybersecurity professional. I've been doing that for quite a long time.
Specifically, I work for a company called Dragos and at Dragos we specialize in industrial
and critical infrastructure cybersecurity and I work specifically there in incident response
and digital forensics for systems like power plants and mines and transportation that have
been compromised by various types of adversary groups.
Awesome. So keeping the people safe from the big machines.
I'm certainly trying to do that. That's important to me.
Fantastic. Well, on that same thread, these systems are definitely different than IT,
right? They're very different. What are some of the unique challenges for incident response
and forensics in OT generally? Like, can you do the same things that you would do in IT
in the OT world and if not, why?
Incident response is stunningly different in OT. Yes, we use some of the same forensic
tools to do investigations of malware and compromises and computers and things like
that, but there's some things that just make it a completely different way of life. The
first one, of course, is process consequences. Everything. When you're talking about physical
kinetic systems like industrial control systems, you're talking about things that can kill
people. They can cause critical services to fail if they aren't there, like water, power,
things like that. They can also harm operators. They can cause environmental damage, facility
damage. They do physical stuff in the real world.
So everything that we do in cybersecurity in OT, we have to do with these process consequences
in mind. And so when you're talking about a system that's been compromised, there's
a whole bunch of extra considerations that go into the response and recovery efforts
like how long can the system be down without people dying? How long can we operate on manual
operations without any of the computers before things start failing? And then there's a side
of the investigation too. What tools can I safely run in these computers that aren't
going to cause a worse consequence? You're not going to find EDR or XDR on a bunch of
industrial hosts. You just can't do that on a lot of vendor and older hosts. Even on newer
hosts, they have to operate in a very structured, vendor approved way to operate safely. So
limited security tooling and oftentimes very old systems with very, very legacy security
tooling and the forensics tools we have to use in those hosts are very old as well. If
we're talking about Windows XP or older Windows 95, I see on a routine basis, Windows NT.
So old school forensics, sometimes very manual forensics, and then very judicious risk decisions
about what we can and can't do and how much time we actually have to do those forensics
efforts.
What do you mean by time? So talk to me about that.
Sure. So everything in a process, when you're talking about business continuity and process
operations, everything comes down to time. Time of how fast the process functions, how
many widgets you produce, how much power you put out per minute, per hour, per day, et
cetera. And that ties into your bottom line of whether you can stay in business or not.
And then time in terms of how long you can safely run without your computers. The computers
that I'm talking about, the industrial control digital systems I'm talking about are things
that provide telemetry. So making sure systems are operating safely and effectively and efficiently.
They provide process efficiency. And a lot of human beings have been removed from the
mix and a lot of process environments and their roles of keeping things running smoothly
have been replaced by computers. So switching back to manual operations is a big deal. And
then even more beyond that, some things like modern micro grids and smart grids are so
complicated and mathematically complex that they have to be run long-term by computers.
So there's the timeframe there of how long can we have these systems down, pulling images
from them, investigating who's compromised them, figuring out what they did, how long
before we just have to start swapping things out with an adversary still in the system
to keep people alive. Everything comes down to these time and risk decisions in our investigations.
Okay. And in some cases, I assume there are even situations where you just can't take
them down, right? You're going to have to do this in a live operating environment.
Yeah. I mean, there's a lot of it depends in what I tell customers when they call me
in on what's usually their worst day ever. And that comes down to what do you need? Do
you need to be up and running in two hours or somebody dies or you go out of business?
Can we spend 24 hours doing an investigation? Things like that. Those are tough conversations.
And in talking about these immature environments where there's less logging and very old systems
that have less forensic detail, all those things play into how much I can actually say
about what the adversary did. Oftentimes these incidents and these more legacy and less defended
environments are detected very, very late. I'm talking about months or years after the
initial intrusion. So I do my best. I have to use a lot of manual, very advanced old
school forensics to pull things out of the ether to try to tell people what the root
cause was in the initial point of intrusion. And sometimes I can't. Sometimes it was too
long ago. Sometimes we just have to get things up and running. And I collect the forensic
evidence I can to do a very manual investigation on after the system's been restored. And
that's a very hard decision to make because you start restoring things on a system that
say a state adversary or criminal adversary group is still in. They're going to respond
to that. They see restoring the systems. If they still have access when the internet connectivity
is restored, they're just going to destroy things again. So very complicated decisions
to make with the leadership in those organizations, with the safety personnel, with the external
experts on that industrial system, and with my colleagues and peers in forensics and intelligence.
Okay. This sounds incredibly challenging. So these organizations, I know that they vary
in terms of maturity and skill sets and technologies, but generally speaking, if you had to pick
like, you know, top three, top five, what are the things that you think they could do
better to enable you to do your job easier? Like when you walk in, what are the things
you wish or you hope they have ready for you or the capabilities that they have for you
to use to actually make your life easier to help them?
Start with the basics. There's a white paper out through SANS called the five critical
controls for industrial cybersecurity. And they name things like basic architectural
controls, vulnerability management, remote access control, having some fundamental incident
response plan and some fundamental security monitoring and logging. And those are things
that you build onto over time. You start with the very basic elements of those five capacities
and start with something. Give me something to work with. Give me anything. Give me some
logging. Give me some architectural diagram, some asset inventory, some idea of how your
environment is connected to remote access. Anything helps. If I don't have those foundational
details at the beginning of one of these very critical response efforts, I have to go find
that stuff. And that's time in these. I was talking about how important time is when you're
talking about restoring a perhaps crippled industrial environment. Time is crucial and
you don't want me at incident response rates having to build you a network map. And so
think about those very foundational things like understanding your environment, building
some segmentation in it, doing some logging, some monitoring, having some plan of what
you'll do if you do see something nefarious. Anything is better than nothing there.
Excellent. Okay. That's fantastic advice. Totally agree. Well, in general, I mean, the
critical infrastructure organizations, OT organizations, and I'm going to generalize
this just globally because I know I both work on a global scale. Which sectors do you think
are leading or lagging in general? Kind of curious to get your take.
Yeah. So it usually has to directly equate to the funding of the organizations. So it's
not necessarily related to the size of the organizations either. It's how much money
they spend on IT and cybersecurity. So the ones that have the most money for cybersecurity,
the best resources are usually like the large oil and gas firms and some of the very large
electric utilities. There's some legislative pressure there to do cybersecurity and they
also have money and they're willing to invest that money, that margin on cybersecurity,
on preventative controls. So they generally in the range of things are doing better. And
then you have like municipal utilities that are also doing very important things like
making sure we have safe drinking water and our toilets flush and garbage goes away and
transportation works, our traffic lights work. And a lot of those are very underfunded organizations
in general. They're small municipal organizations. They maybe have one or two IT people and things
are much, much more challenging there. Things are prioritized in terms of keeping the water
running instead of spending money on margins of things for cybersecurity. So that's one
of the scariest spaces for people who are professionally in this field. If you're talking
about like manufacturing, even like big manufacturing firms oftentimes have big problems with industrial
cybersecurity because of the low margins, even though they can make a lot of profit
and be very large companies, they don't tend to invest as much in IT because that's not
green dollars coming in. That's an expense. That's a cost center. So we see problems there
as well. This is generalization. So I mean, we see variation in organization size and
global locations in those various industries. And the thing I do want to express is that
spending a bunch of money and having a bunch of people in your cybersecurity program is
good, but it's not necessarily a complete defense against intrusions. I respond to incidents
in those organizations too. They have their own challenges. But in general, the industries
that concern most of us, and I don't know if you agree with this, municipal utilities,
things like water, sewage, any power that's handled by a small co-op, things like that.
Those are scary spaces that do really important stuff and don't have many resources for cybersecurity.
Yeah, I would generally agree. And I would also say the same where I've seen very large
organizations that have spent lots of money on a wide range of tools and they've got even
multiple divisions within a security department. And they still suffer from, in some cases,
fundamental issues where I've seen some smaller organizations, even in some of these co-ops,
for example, that they've got just a superstar one or two people that can do an amazing job
with limited tool sets. So people that are there and the investment from senior leadership
in how much they care about cybersecurity and stand behind those people. And in general,
sometimes you can be more agile as a small organization. So don't feel defeated if you
are that small organization. You have potential to do really interesting things too.
Yeah, absolutely. Now in these sectors, like I agree, I've seen a lot in the electric sector,
oil and gas space as well. They do have generally deeper pockets. I would see them definitely
leading, especially in front of local water utilities, even in some cases like rail and
manufacturing and shipping, mining. Yeah. Now, just globally, I wanted to bring a little
bit of a global aspect. Which areas do you see are more advanced or further along, shall
I say? I hate using the word mature or advanced, but I guess further along on their security
journey than other areas.
Well, I mentioned like oil and gas tends to be very, very well-resourced. They have a
lot to protect and they've invested a lot in their IT programs and in their cybersecurity
programs and they're willing to have larger margins in these things that aren't direct
income. So yeah, so that's usually one where cybersecurity is quite strong. Again, there's
still problems there when you have that large of an organization and oil and gas tends to
be very distributed operations with a lot of different facilities, remote facilities
that make cybersecurity very challenging. When you're talking about like platforms out
in the ocean, that's a tough one. That's a really hard thing to secure, but they tend
to do quite a bit better. When we're talking about like electric utilities, that depends
on how they're regulated and organized in different parts of the world, whether they're
centralized, whether they're a bunch of different companies, whether they're small co-ops, if
there's a lot of like independent generations, smart grids, et cetera, that changes things.
But in general, beyond those verticals, it tends to be the organizations that have good
management buy-in and a willingness to have that cost center of cybersecurity that put
organizations ahead of other organizations. Okay. Yeah. And I'm even thinking in terms
of a global perspective, like in some cases it's been regulatory driven and in other places
it's been market driven. So I've seen situations where, oh, let's say higher threat locations
obviously go for a higher security profile, right? And then there's other areas where
just the regulation in that region or that country have caused the whatever sector to
do more. In the U.S. of course we have NERC SIP for the electric sector. We have the TSA
pipeline security directives for our gas space. There's been talk about the water sector.
In Europe they have more of a general approach overall with the NIS2. And then of course
each individual member state has their own quasi, that's a quasi that I almost direct
in some cases specific to state things like German DSI or UK has a framework like the
CAF, Australia has their own CSF. Any of those that you see are really kind of, I guess becoming
more of the standard to which others should look to, I guess is the best way to ask the
question. I think that there's a lot of good things coming out of Europe and the U.S. and
Australia. I'm always a very cautious person on critical infrastructure cybersecurity regulation.
And I'll tell you why. It's because oftentimes I see two problems. The first one is that
it's very strategic and it doesn't actually provide tactical details. So strategic in
terms of who owns what, who's in charge of what and very, very high level directives
that can be interpreted in a lot of different ways. And I understand why they do that because
there's so much variety and variation and debate about how to do a lot of these things.
But I do wonder about the efficacy in some of those cases where it's just very, very
high level and strategic. But the bigger concern to me is when things are regulated without
providing resources to do them. Again, thinking back to like the small municipal utilities,
those water utilities, they got one IT person, maybe half an IT person, maybe they're an
operator the other half of their time. And you're going to tell them to do a bunch of
installing EDR patching or installing elaborate network controls. That's going to take away
from something else. Have you thought about what that's going to take away from? Is that
going to stop them from doing monitoring? Is it going to stop them from responding to
trouble tickets from the operators who see stuff happening? What is that going to take
away from if you don't give them tools and actual functional resources to do that cybersecurity?
So anytime I see regulation like that come into play, I also want to see at least good
educational resources being provided to those utilities. More preferential would be giving
them monetary resources, incident response resources, tooling, licenses for tooling to
do things that you're legislating. This stuff is expensive. It's hard to do. OT cybersecurity
is hard and it costs money. And again, if you're regulating stuff, that money and those
people are coming from somewhere. So be very cautious about that. But in general, I mean,
to answer your question, there's a lot of good ideas in that legislation. They are reaching
out to industry. They're reaching out to specialists in the field in most of those countries. And
they're thinking about, again, those kinds of critical control things like, gosh, we
really need to control remote access. We really need to make sure that we understand vulnerabilities
in these environments. We really need to segment. Yeah, that's all really important stuff. But
again, doing that is not easy. You think it's hard in enterprise. It's really hard in a
legacy industrial environment that you can't shut down. So a lot of considerations to go
into that. And I wish there was a little bit more tactical recommendations and resources
coming out from those countries too. Yeah. I have the joy of having written regulation
and then they forced it. So I've seen what it looks like on the other side too. It's
really difficult to write regulation in a tactical way because technology moves so fast
and threat actors change their tactics so quickly. So you're basically forced to write
it strategically. It's a difficult balance. But one thing I am seeing on a regular basis,
I would say commonly through much of the regulation that leans into the critical infrastructure
spaces in an OT and our world is definitely a push for good incident response. One of
the things I've debated with Danielle Czaponski on a previous podcast was if you had to write
one regulation, if you just had the capability to write one, what would it be? And we both
kind of circled around the concepts of just incident response or just threat hunting.
I see how they kind of go both hand in hand, but I have seen this as such a common thread
through a lot of these different regulations across multiple countries and regions. And
when I ask those that are writing it or influencing it, it seems to come down to the fact that
there, of course, many companies will not report an incident on the street, right? To
any federal agency or to the news or to anywhere, basically other than even to their shareholders
in some cases. It just, if it only goes up and out to the places, it absolutely must.
So given that we've got a really bad and patchwork and misleading set of actuarial data. So we
know that if Leslie eats bacon cheeseburgers every day and never exercises, watches TV
and is extremely sedentary, there's a chance that you're going to die by 40. But we know
this because of the actuarial data we have on humans and healthcare for like, you know,
arguably a thousand years or more. We have, you know, in some cases, not only, you know,
none, but misleading data in this space. And this is what they're trying to do is what's
working and what's not. What's actually being effective and what isn't. So these regulations,
when I ask about them, this is what they're trying to get to is how do we get this information?
So barring, you know, asking and getting inconsistent, well, as a euphemism, inconsistent answers,
regulation and forcing it out of them is pretty much the only way. So that's where we're seeing
a lot of this incident response regulation or standardization in some cases around it.
So do you think this is even a good idea given what you do? This is your world and you do
this every day. So is this something they can do? Is this a good idea? Is this useful
for the regulatory bodies?
I'll start with for the audience. I do this every day. I am an incident responder for
industrial systems. I am busy enough and I have been for the last seven years at Dragos
that this is what I do every day. There is a lot more going on in the industrial incident
space than people see on the news. And all the things that you described are true. Nobody
wants to have a big eye incident. They don't want to report it to their board. They don't
want to have to report it to regulators. They don't want to get dinged for whatever they
think they were responsible for in the intrusion, even if they weren't. And so that really is
a thing. There is a resistance to reporting incidents when there's no regulatory requirement.
We do track those incidents. We have our own numbers, not with customers named, obviously,
but just general to what types of incidents are happening and how many per year. And yeah,
I mean, that's important information. That's important information to track adversary activity,
to understand what threat groups are doing, what criminal groups are doing, and how people
are adapting their techniques to attack, especially lower level industrial systems, which is happening
too. Now, the other side of that, though, is a lot of stuff is not being detected in
time. And there's a few different reasons for that in industrial environments. First
of all, the security maturity is so low. There is a lack of detection tooling. There is a
lack of host tools, network tools, antivirus, EDR, smart firewalls, things that we are very
dependent on in our security operations centers are only trained on these days don't exist
in a lot of levels of the industrial environments. So picking up things is hard. Oftentimes it
takes somebody at a system seeing something weird happening or something actually breaking
or something visible, like a ransomware pop-up, something like that, to detect things. And
that can happen quite late. And there's a built-in temptation because of the poor relationship
oftentimes between cybersecurity and these OT environments that do their own thing for
a multitude of reasons to just swap out boxes when something weird happens, troubleshoot
it like it's a equipment failure and not a cybersecurity issue. So we don't know how
many things are just being swept under the rug because systems aren't monitored or when
they have problems when they're infected past the point of no return or an adversary destroys
them. They just get swapped out by the vendor or the people on site because that's what
they do. And you can talk about incident reporting, but if nobody ever detects anything, things
aren't going to get reported anyway. And if they are massively de-incentivized to detect
anything, they aren't going to ever detect anything because they're not going to install
any more detection controls. So you mentioned having a debate previously about detection
versus having incident response as the primary capability. And they're both important. They
go hand in hand. I don't have a horse in that race. You have to have something to do if
you detect something. It can be very minimal. And to be able to do something, do incident
response and report anything, you have to have some kind of detection. They're both
essential. They're both critical controls in the five critical controls for industrial
cybersecurity. You have to start a little bit on both of those and slowly build both
of them up over time. But yeah, if you just build an incident response plan, there's a
lot of organizations out there with some kind of OT incident response plan. Maybe it's never
been tested that have zero detection and they're never going to detect anything because when
somebody perhaps gets horribly injured in a critical incident that was perhaps caused
by somebody tampering with their environment, they're just going to treat it as an equipment
failure. The vendor is going to swap out all the equipment and nobody's ever going to know
that it was a cyber attack. So culture is really important there. Building a strong,
healthy culture between OT operators, vendors, OEMs and cybersecurity firms, agencies, et
cetera, is really important. And then making sure as we're building that legislation, we're
putting a focus on both of those things, both having some capability to actually do investigations,
make cybersecurity part of root cause analysis, do detection in those environments, even in
a very rudimentary way, and have some plan to do incident response so you know there's
an incident going on and you can report it. All that stuff, it's cyclical, it's dependent
on one another. You can't pull one of those elements out, unfortunately, but the trick
is just starting with the very basics in each of those things.
Excellent. Yep, you summed it up right there. That was fantastic. So if they're going to
mandate incident response reporting, and this in some cases, even when I talk to the regulators
who are writing this stuff, they assume that there is the capability for detection. They
assume that there is the capability for all the things that feed into the incident response
capacity because you can't respond effectively if you don't have this other information.
So they try to, I would say, imply that you need these other capacities in order to do
incident response. So they do this in a lot of ways by saying you must have these minimum
components that you report, and that kind of passively or indirectly says that you must
have the capability to detect these things in order to provide this information in your
incident response report. So they kind of get to it through that roundabout waivers
as directly mandating that you have detection capabilities, for example. If they were going
to do that, what do you think would be the most useful minimum set of things that they
need to know?
You have to understand your environment, which is more challenging than people expect. I
know that, again, these cybersecurity people coming from the enterprise side of things
are like, yeah, sure, I just go into my asset inventory software and it tells me based on
agents what computers I have. It's really easy. That's not the case in OT. Like, what
computers do you have? What do they do? Who owns them? Where do their logs go? A lot of
that stuff has to be figured out on paper in OT environments, and it's really, really
tough. So I highly recommend starting there, having a foundational knowledge of your environment,
your logging, remote access. That's another really tough one. How are things connected?
Those are all tough things to understand. And then what things mean in your OT environment.
That's the hardest bit. A lot of organizations get to the point of actually being able to
do security monitoring and they have some knowledge of the environment. They stick their
sock on it. They get their security personnel to start looking at logs from that environment,
and they have no idea what anything means. Because they don't have a good relationship,
again, with the OT personnel, and they don't have anybody to ask, and weird stuff comes
out of those devices. It's a totally different set of protocols, and the domain is different.
The devices are different. There's older systems, and you don't know what any of it means. You
can establish something of baseline, but something's change doesn't tell you a lot. So I recommend
job shadowing programs. I think that's a really, really important thing. If you're going to
build a healthy cybersecurity program for OT, I highly recommend that you implement
some kind of job shadowing for your cybersecurity personnel to spend time in the OT environment,
and how people work there, and how the system works, and just build a good relationship
to ask those types of questions. What does this mean? What's going on here? Is this bad?
That kind of stuff is going to help you a lot at those beginning stages.
Okay. That's super useful. Yeah. When I look at, for example, NERC-ZIP or Circea, and even
the NIST 2, it doesn't have a lot of specifics that you have to report. But I look at what
they're asking for, and I'm just wondering, is that enough for them to really... It sounds
like they're just really grasping... Oh, like incident details? Like what happened
in the case? Yeah. Yeah. I mean, anything is helpful there. It depends on the level
of investigation. Yeah. What vertical was it? How many systems... I mean, how many systems
is going to depend a lot on how flat your network is in a lot of cases?
And how much you know about how many of them were actually hit?
And what security controls you have in place? Whether you have some security controls that
prevented lateral movement. But more relevantly, there is fundamental threat intelligence data
there. Do you know which adversary group it was potentially by any name, by any organization's
identifier? What were their tactics across MITRE ATT&CK? That's useful. What parts of
the kill chain did you forensically analyze and identify? So we can track that in somebody
else's environment. IOCs are important. All of that information can be useful to peers
and to tracking these adversary groups at a national level. If you're talking about regulation,
you're talking about what's useful for intelligence agencies to understand what other countries
are doing to our critical infrastructure or to other friendly nations' critical infrastructure.
So all of that tactical detail is useful. But I wouldn't make all of it obligatory because
people don't always have the maturity to get all that information. You might see one little
piece of MITRE ATT&CK. You might see one little piece of the cyber kill chain. You might just
know that a system was tampered with and have no other details. And that's still important
to know.
Yeah, I've been advising that less is more. Just notification of incident in some cases
would even be enough that they're having a problem. Because while I would love to have
that rich data set you just described, some of them I've never even heard of the MITRE
ATT&CK.
Yeah, you can't really legislate that. You can't require it. They don't have the capability
to provide it. You could provide suggestions. Again, when I'm talking about more tactical
details in some of this documentation, I'm talking about cross-references to supplemental
documentation, not necessarily obligating people to do it as part of the regulation.
But again, those resources, the supplemental resources to help people do stuff, when things
are very high level and vague, they can feel very overwhelming for a less mature organization.
So if you're going to say you have to have an incident response plan, then gosh darn
it, maybe we should start having government sanction templates out there for various types
of playbooks and incident response plans in OT. Because the enterprise ones don't work.
So yeah.
That would be tremendously useful. Maybe one of our agencies like CISN or even MITRE can
produce something like that for us.
Maybe.
Yeah. We can certainly ask. We know some people. Okay. And then the next piece on these is
not just what is reported, but the timeframe. So what, I mean, you've done so many of these.
What do you think is realistic for any, again, generally speaking, and I know I understand
it widely varies, but generally speaking, is 72 hours enough time? I mean, because NERC
sets as an hour after you notice an impact to your operation.
After you notice some things going on.
Yeah.
The sad reality is, okay, so let's say an adversary uses a system as a testbed. Maybe
they pick a little co-op and they want to test one of their future attacks against a
big city. So they pick somewhere they think is less defended and this happens, this is
the real thing. And they test their attack on them. So there is a impact. It's successful.
It does something bad. It has a process impact. What are the things that happen early on in
the first day of that? Well, first of all, the operator and the engineers try to figure
out, troubleshoot what's going on. They try to fix it themselves. And then when they can't
do that, they call in their OEM. And maybe then the vendor has to get called in by the
OEM because the OEM can't figure out what's going on. And then we've got the iffy situation
where maybe they just swap out the box because they can't figure out what's going on, or
maybe they call their cybersecurity team because they think it's so weird and they want them
to be part of root cause analysis. So how long has that taken? I mean, that could have
taken days of troubleshooting, of getting the vendor to go out to the site. They have
to get it, dispatch somebody, they have to get in their car. Then they decide that it's,
you know, first they called out an engineer and the engineer couldn't fix it and then
they called out the OEM. The OEM couldn't fix it. Then they called out the vendor. So
I mean, I don't want to let people take more time than is necessary, but understand that
there is time involved in troubleshooting OT stuff. Now if it's ransomware, if it's
like, you know, you see some mouse moving on the screen, that's quite visible, although
there is going to be troubleshooting involved in that too. Somebody's going to, an engineer
is going to have to call somebody and they're going to have to call somebody, but it's not
going to happen fast. I'm getting called in six months after incidents. I'm getting called
in six years after incidents. Like it's a slow process right now because of that lack
of detection and incident response maturity I talked about. It's a really slow process
for people to figure out it's a cyber thing and get somebody involved who knows what to
do with that. So I don't have an answer to the question. Like, you know, 72 hours, if
it's ransomware, yeah. If it's something that's visibly cyber, yeah, but don't expect that's
going to happen when it's actually the big bad doing something to a process. It's not
going to happen that fast. I've often said the more advanced the adversary, the longer
it's going to take for them to notice it's an actual incident. So those are the ones
you're actually trying to solve. Yeah. Yeah. And that's the really scary stuff because
they're making the same thing happen as a dangerously failed piece of equipment. So
it's going to look like a dangerously failed piece of equipment. Yeah. Well, it's good
to know that even someone who does this every day doesn't have a good answer. Maybe an actuary
does or something. Maybe looking at the incident metrics, you could come up with a number,
but it takes a long time and that's a big problem that we have to solve first. It's
super challenging. No, I couldn't agree more, but that's, I didn't expect anything. Definitely
be honest. Now the last one I wanted to argue about, argue about, discuss is the definition
of incident because rarely is this well defined. Do you have something you use or is this just
as squishy for you as it is everybody else? So definition of how you, how you qualify
one, what an incident means? Yeah. Yeah. So, um, yeah, I mean, I would still use the standard
cybersecurity definition of incident. You know, it's, it's a major event, but what,
how you quantify a major of event, a cyber event in your environment is going to depend
on what matters to you in your industrial environment. And this is something that everybody
misses in OT as well. Um, they usually copy their severity matrix over from enterprise
and it's absolutely goofy and ineffective for OT. Um, your severity matrix in OT, what
you quantify as a high, medium, low severity audit level incident or event in OT is going
to depend on what makes people get hurt. That's, that's the worst day ever is how I usually
explain this to people is what you really need to do is you need to sit down with your
leadership and say, what is our worst day ever? What is our worst case scenario as a
industrial business or an organization that relies on industrial systems? Is it somebody
dying? Is our process failing for 24 hours? A worst case scenario is our, our really expensive
equipment being utterly destroyed, a worst case scenario. And then you work from down
from there and figure out, well, what could practically cause that and what mitigations
for those hazards are in place because you don't want that to happen because an operator
screws up. So there's safety controls in place. They're not all digital safety controls.
Some of them are physical, some of them are human. So you have to do that kind of risk
mapping down from actual consequences you actually care about what would really cause
the worst case scenarios. And those are your high and critical level incidents. And that's
going to vary depending on your industrial environment, your process, your vertical,
your location, et cetera, your redundancy, all of those things are going to play into
that calculation. So you've got to do that for your industrial environment. And that's
how you name what an incident is, what a high severity incident is versus a audit level
event that isn't actually considered an incident to a moderate severity incident, incident,
et cetera. You can't just say, oh, a web server compromise. That can't be in your severity
matrix for OT. It can't be, oh, there's 40 computers infected. Does that actually matter?
If they're infected with Conficker or Sality? No. Shutting down the process to reimagine
those computers is going to have a much worse impact from a consequence perspective. If
it's ransomware and there's a loss of control, that's a very, very different situation. And
you should be calling out a loss of control, loss of visibility situation, not, oh my God,
you've got 40 computers infected with malware. I've been called into environments that have
been infected for 10 years and they finally have a big enough maintenance outage and upgrade
period to get rid of it. So they're calling me in 10 years later. Like consequences are
king there. Yeah. I would say it's, I've been to generation plants and they're just rife
with Conficker and no one cares. It's just not impacting anything.
But they're all infected. Yeah. You can't think about things like that.
Yeah. There are some that argue that it doesn't qualify as an incident unless there is malicious
intent. What are your thoughts on this? Oh my God. I don't care. I don't care.
Design incidents however the hell you want. Like, you know, I see them defined differently
in every industrial org because they define words differently between IT and OT. Sometimes
OT defines incident as, you know, something independent of malicious activity. Yes. I
understand in enterprise environments, we usually clearly define cybersecurity incidents
as things that have malicious intent involved. But in OT, sometimes you've already defined
the big eye incident as anything that impacts the process. So I don't care. I'm sorry. That's
my answer. My old curmudgeonly answer to that is figure out your definition, share it with
OT, make sure you're on the same page with definitions and make sure you've quantified
how you're going to define those incidents at different severities very, very clearly
in a way that actually makes industrial and process and safety sense.
Yeah. No, and I totally agree. I understand. The reason I was asking was it matters when
you're trying to report both upstream within the organization for incident management metrics
or KPIs in addition to outwardly to insurance firms or whether you're going through M&A
or regulatory agency, for example. So yeah. Figure it the hell out. Decide whether that's
going to be a differentiator for that and how that figures into your regulation. That's
the only answer I'll give there is make sure you have a clear, common, shared definition
that everybody knows and make sense for making people stay alive.
Yeah. I'm just still waiting to see what that clear definition is for any of them.
Yeah. I don't think that they're going to include malicious intent in regulation for
incidents. I haven't seen it in a lot of places globally for that type of legislation and
regulation for reporting. I haven't seen that requirement for it to be purposefully malicious
activity. Yeah. It's not in like NIST 2, for example,
but if you get to the NERC SIP standards, they actually have a clause in there that
says attempt to compromise. And that one seems to lean into the malicious intent space a
little bit. To compromise. Is the worm that's just headless
and spreading in your environment and doing bad stuff unintentionally, is it attempting
to compromise systems? Gosh. That's like, is sality getting on something today? Attempting
to compromise. I don't know. I don't know. It's ambiguity again.
Yep. That seems to be the crux of the argument when you have those situations that really
do, they look like they would qualify as an event or a cyber security incident, but you
don't have that malicious intent or that attempt to compromise. So it's one of those things
I just thought it would be interesting to ask. It's an interesting thing to think about.
Again, make sure that you have a common standard definition. And if you are a regulator, make
sure that your definition for other people who need to report to you is clearly defined.
And based on what you want to know about, if you're the regulator, do you want to know
every time somebody deals with Conficker? Do you want to know about that? Is that useful
to you? It has real impacts and environments. It can eat up low resource machines and cause
really weird, bad things to happen in industrial environments. But like we discussed earlier,
pretty much everybody has old headless worms in their critical infrastructure environment.
So do you want to know about everyone? Then you're going to need to define that. You're
going to need to say that in your regulation. Yeah. And then you're going to have to house
all that data and then see if this is trying to make things worse.
That's a them problem. That's definitely a them problem there. Figure that out. Define
it clearly. Don't make these ambiguous definitions. You have to make things clear for people.
Yeah. Do the hard work to actually get it defined right.
Yep. Awesome. Okay. Well, I'm going to wrap up with asking you what is next for pancakes?
What's going on? No. I mean, I guess personally, I am currently
me and making an endeavor to move to Australia with Drago and start working on a new frontier
of industrial operations and incident response there in that region of the world. So there's
a lot of development and growth and incident response there. And I hope that that works
out. I'm very excited to start working in Australia and the nearby nations in defending
their critical infrastructure as well. So that's what's next for me, hopefully.
Okay. Fantastic. I really hope that works out because I know it would be a valuable
resource anywhere you go. So I certainly hope that happens.
Thank you. You're so kind. This was so fun.
Yeah. Now, are you going to any of the conferences that we usually see you at coming up? So in
case anybody's listening, they can catch you.
I will be at Shmoocon. I'll be at ChiBurcon in Chicago. I will be speaking in Canada a
little bit later in the winter time. And I don't know beyond that. Right now, my plans
are a little bit up in the air based on when I'm going to move. But I'm always trying to
find opportunities to speak and participate in conferences. So I'm sure I will, whatever
continent I'm on that month.
Fantastic. Okay. And I will put all of your contact info in the show notes so that people
can find you and contact with you and get much, much more of your valuable information.
Oh, you're so kind.
Alyssa, I really appreciate your time. Thank you so much.
It was such a pleasure. Thank you for the really thought provoking questions.
Thanks for listening to the Ampex Cyber Critical Assets podcast. You can find us on all your
favorite podcast sources. So please like, subscribe and share with your colleagues.
Check out our other content such as blogs, news and more at AmpexCyber.com. It's A-M-P-Y-X-C-Y-B-E-R.com.
Ampex Cyber, securing your world.