Welcome to the Talking Security podcast.
We will talk about items related to Microsoft Security.
So hi, welcome.
There are we again with the Talking Security DevOps podcast.
And again, Pujan and Sander are with me.
So welcome guys.
How are you?
What do we have today?
After last time we did a recording about the test phase within DevSecOps.
What do we have on the menu today?
Pujan, it was all yours, I guess.
That is true.
Yeah, we had some planning in the last episode.
So yeah, indeed, from the last episode where we talked about testing and what you can do
in your test phase and what the vulnerabilities you are, it's about time to spend some time
about release.
In my opinion, it's a very important stage in the whole DevSecOps process, because this
is the moment that you go from your code towards your running environment.
It will be production or acceptance.
So it's a pivotal point where you need to think about the process because you don't
want to, you want to have that process as fluid as possible, but also in control as
possible.
But you don't want to stop the developers like Sander too much by creating all kinds
of issue or performance for them.
So yeah, today we are going to talk about release.
Why it is important?
Which features in there are important together with you guys?
We will also take a look what kind of, what you need to do and what kind of to align this
process as well.
What is your view on the release?
My view on the release?
I find, how do you say that?
And we're talking about like DevSecOps and release.
I think it's a really, really important part.
Like Poojan already mentioned it before.
When I started learning DevOps and CI, CD a few years ago and stuff, like right after
school, it was really cool to me.
I could just deploy my applications immediately and release them.
But I think with DevSecOps, it's maybe even more important because the entire focus is
on the security part.
Like in the previous episodes, we talked about security.
And now if you have found some security issues and you need to fix them, you need to also
be able to release them really, really quickly.
And especially like what Poojan then says is right, if we want to release something,
we should not have that many barriers in place, but the barriers would still ensure that you
are secure.
So you need to find a bit of a balance in that.
But I think it's a really, it's a fun step of being able to have your software.
It's done, it's built, it's secure, it's tested.
And now we're actually going to finally release and do something with it.
And yeah, it's always a fun thing to do.
Yeah, if we want to kick off, what should be the first question you should ask Sander
in this area?
The first question you should ask?
Yeah.
What question should we answer to our listeners within this phase?
What is the topic that you want to cover within this phase?
And probably if you have the question, Poojan can maybe answer that.
I think maybe that's a good thing to ask.
Maybe it's a bit open.
So if we're releasing our code, how do we know that what we are releasing is going to
be released to a secure environment?
Right?
Like we've tested and we've built our code, we did all of our security thingies there,
but now we're going to release it.
So how can we ensure that what we release will be running securely?
So if you look at, if it runs security, it's of course more on the operation side, right?
So then we are more going, for example, if you take our code, because this is the pivotal
point, right?
I think what you're saying is like, release is integrating the code, security and operation
together.
And for us to see how the code is functioning in operation is building on solutions like
Defender for Cloud, right?
For example, we are doing container Kubernetes, or we are doing some database changes by monitoring
our solutions in operations, but solutions in the Defender for Cloud family gives us
a good overview.
Definitely when you integrate it with Defender for Devils, but yeah, we have told that a
lot of times.
That gives a good feel like, okay, what were the changes and how are they functioning in
the production side?
So that would be a key.
And I think of course there's a process on top of that.
It's like after your operations center, your security operations center, which is monitoring
and responding based on those security alerts or findings or full abilities that are coming
from that environment.
Yeah.
So that would be definitely a way forward, in my opinion.
Yeah.
That's a good point.
If you go back, that's a question to both of you guys.
In my opinion, if you take a look at release, it's mainly having defined how we are going
to release and what steps are included in there, right?
That's part of the DevOps, but it's also part of the DevSecOps because things we discussed
last time like code scanning, those are also, in my opinion, part of the release stage,
those are things that you define in your workflows often as well.
So that brings us to a terminology, of course, that's often used like the CI CD, right?
The continuous integration, continuous deployment.
And how do you guys see the uses of that in your environments?
I'm not really sure if I understand what you mean.
The uses of that in my environment?
Can you explain that a bit differently?
Yeah.
Do you have a really, if you build an application, do you have a CI CD in place?
Like when the code change hits certain branches that the certain steps happen, or if your
main branch is updated, certain workflows get triggered to do certain deployments or
packagings or even updating your ETSM or environment?
Yeah, definitely.
So in my case, normally, when main gets updated, we build our code, you can do your security
steps right there that we talked about last episode, which you should definitely listen
to.
And then there's of course, when that is built, it produces an artifact.
At least that's what it's often called in Azure DevOps, which is what I mainly use,
which is then released.
And then I think it's a really good way, would you say, for example, when we have a new Docker
image that we want to, we have scanned, we can upload that to an Azure Container Registry
or an AWS registry or whatever kind of registry it is, where also more security steps can
take place, which I think is a good thing to do in your release steps.
And then of course, we need to publish some more stuff to the cloud or to wherever you're
running.
Which I think kind of flows into a very nice next step is how do you connect securely in
your release step to whatever you're deploying?
I hope people are not losing a username and password when they're deploying something
to Azure.
You should be using the most modern techniques for that.
Maybe you can talk a little bit about that, Poojan, because I think you talked about that
before.
Yeah, definitely.
I think the usage of a username or what we see a lot, I think we all, three of us, is
the usage of search principle and passwords, for example.
Well, there are great tools that can help you.
I mean, go for GitHub and security has already code or password detection pre-commit, so
it doesn't even get to your commit history.
But yeah, we also have like the defender for DevOps, you can integrate that in your workflow
that you do a scan to see if there's any code and if there's not, then you will continue
with that.
So we need and the same is possible with GitHub advanced security, with fair code, for example,
what we discussed last time.
So the usage of security like passwords is really one of the important things.
But one of the things that's also important to check when it comes definitely towards
your if you are doing things on cloud like Azure, definitely important to do like, okay,
the environment which I'm deploying or maybe I'm creating because that's part of your configuration
as well.
Are there any full ability there as well?
So we see different levels when it comes to security in the release page.
But I have like, for example, also customers when it comes to the release is like, okay,
we need to have after each deployment to our production, we need to have a change record
created for that.
So that we are for our compliance purposes.
We see that also a lot in this release workflows coming back.
I don't know if you guys see that as well.
I'm not too familiar with it.
And if we look at compliance in policies, we're talking about secure code, scanning codes
and so on before releasing it.
So we have a secure piece of software that has deploying.
But on the other hand, what about policies?
If you are, let me give you an example, if you do a pull request and to merge that into
a new branch or a new master before deploying.
You can't say that anymore.
You can't say a monster anymore.
I know, but it's an example.
How many approvals do you need?
What steps do you need to take before something gets a final approval and gets to watch a
release?
That's a good point.
I kind of overlooked that.
That's a good point.
So yeah, you're talking about, for example, when code is merged, like, you know, who needs
to, your code can contain things, but people need to, of course, look at the code as well.
But then there's, of course, the really important thing, like, what I usually do is, well, my
code gets merged, then it goes to automatically gets deployed to the test environment and
then just automatically no one has to look at that.
But when we want to deploy the production, someone has to click on the approve button.
And in my case, in my teams, my teams are usually pretty small that I work in.
So I usually just do any member of the team can do it.
So if I want to, I can also just deploy to prod whenever I merged.
It's a small team and it doesn't really matter all that much.
But for bigger companies and for enterprises, that's probably a good thing to have.
Like, hey, maybe multiple members of your team need to look at what you're deploying
to production.
Maybe if you have multiple really big teams and you are affecting some other team's code,
might someone else to also click on the approve button to kind of take a look when stuff is
going to be deployed.
I think that's a good thing to mention.
Yeah.
And Poojan, how does that affect the security teams in that?
Yeah, that's a good.
Yeah, I think it's a great question because but also to add on what Sander mentioned is
the number of people that need to approve things.
I think it all comes down to the level of workflow that you have created in the sense
of how advanced is it like are the code testing, are the functional testing, user testing,
all kinds of tests already in your workflow.
Like building up your change log, building up your documentation, creating your change
requests in your ETSM platforms.
So it depends on the security side as well.
We also say, OK, for example, if you have your CI part of the deployment, the continuous
integration part, pretty well framework and you have your tests, your validations in place,
then your deployment checks will decrease, of course, the number of checks because you
can rely on your tests more.
And the same is for security when it comes if you have done definitely active testing
on your code.
Let's say, for example, you have multiple solutions and you deploy your code changes
to an acceptance environment before you deploy to production.
And there you have your code validation part in place and you have that environment also
connected to something like Defender for cloud.
Then you can both find no vulnerabilities or any misconfiguration or code violations.
Then it's much easier to say, OK, the deployment to production needs one approval.
That's a good point.
So to summarize, I think what you're saying is if you have a lot of active monitoring,
for example, in place, the code is running 24-7 somewhere in the cloud.
And there's also some monitoring solution that is checking all of that.
If there's loads of stuff already before or during the runtime of your application, you
don't need to add 20 checks and approvals and ceremonies to the release process.
You need to think, especially for your team and for the product you're working on, what
part needs to be secured?
If I'm deploying some really tricky software that needs to be very well secured during
the build process, but doesn't really accept any user inputs or cannot really be talked
to, maybe then it's more important to do the build security stuff and let go of the release
process, do some basic checks, some basic approvals.
And that's it.
Yeah.
Because you need to ask yourself, I think, the following questions.
If you have three people approving your deployment, what are they actually approving?
And are they aware of what they are?
Yeah.
Usually it's nothing.
And usually when I work in a team or even when I set up the policies and stuff, I usually
always click on the person that created the release is also allowed to merge it.
And if you've got to be very puristic about it, you could say, yeah, but if that person
has bad intentions, they could release something to production.
I never assume people have bad intentions.
And I've had multiple occasions where I had to release something in the evening and someone
had enabled that I was not allowed to release to production my own code.
So I had to call someone to turn on their computer so they could click on a blue button
for me.
So it's really important.
Yeah.
Personally, I'm never a big fan of the like, other people have to look at it.
I think it's something you talk to with your team.
You need to communicate with your team about this kind of stuff.
But yeah, it's more of a sidetrack where I'm going right now.
Sorry.
Well, to point out the first point you made, I never assume anybody.
Well, that's not really in line with assume bridge.
So you need to always assume someone is trying to because that person doesn't need to be
that person.
So that's, I think, the whole reason why you have the DevSecond.
That's a good point.
So on the second part, I think that's one of the biggest challenges for organizations
because you don't want to call up your colleague at 7 p.m. in the evening or later even in
the hope that they answer to click about that.
In my opinion, I think definitely when it comes to security and also developers, developers
want always to have speed, right?
You want to get things done as soon as possible.
And security wants to be in charge and see what has been done and what's going to happen.
And that's why the release is so important because if you bring all those requirements
into one workflow, like what is security asking us to do and what is developers asking us
to do and what's operation asking us to do?
If we have that process in one overview and we change that into our CI CD workflows, then
in my opinion, you don't need to call someone at 7 p.m.
There should be that control should be in place to give you the possibility to do your
thing while the monitoring on security is high alert.
From the operations side, because we are talking about security and we are also talking about
the developers, but I think it has a really huge impact on the operations as well, right?
Because they need to be aware what has been changed because at the end, often they will
get the calls from the customers or from the employees that something is not working.
How do you guys integrate that part as well?
That's actually a good point.
I haven't really thought about it in that way.
Yeah, usually what the team lead or the product owner does is just send an email to the organization.
Hey, this is going to be changed.
But I guess that works for small companies, for bigger enterprises.
I've seen systems where they have like build change logs and then you can just kind of
go to an internal page where you can always see what version of a system is running when
it was updated and the change logs and stuff you can see right there.
That's kind of interesting.
On the other hand, I also feel like if you're going to do breaking changes during your release,
that should definitely be something that is communicated beforehand or in fact, you should
never do breaking changes until everyone is ready for them.
But yeah, how do you communicate with it?
Is it also kind of an important part, especially if it's going to be something secure that
you're going to be deploying or you're going to be deploying changes to your configuration?
For example, some biceps, some infrastructure scope, that's going to be deployed with different
firewall rules, different security stuff, something that you definitely need to keep
in mind when you're going to be deploying that in a big enterprise.
Everyone should be aware of that.
Yeah.
And I think in the end, what you see a lot is that security teams are trying to step
in into that developer's ecosystem, right?
To get a better understanding and controls in place, and then the same should be there
for operations.
And I think that's what makes release important is to get a clear overview of all those requirements
from those teams as well.
But what do you see, for example, as challenges when it comes to, for example, let's say we
are having our CI CD in place, we are having our code validation and scanning what we discussed
last time and previous, everything in place.
And once you do a change and that change got blocked because it contains certain security
vulnerabilities.
Yeah, that's tough.
I mean, I guess it's kind of the boring answer if it depends.
I still feel like if there's something happening in the business and something needs to be
deployed now, and suddenly I get like a CVE, right?
Like a critical vulnerability and the E, I don't know what that stands for anymore.
You should, I guess, still be able to deploy it.
Before you were deploying something and you didn't know there was a vulnerability, but
now you're deploying something and you know that there's a vulnerability.
And I think it's just kind of on you, you as the human need to make the decision, am
I going to deploy this?
Right?
So that's kind of where it depends.
If the business is losing money at this moment because something needs to be deployed, you
can't tell the business, no, we have a security, we have a tiny security issue.
You need to still be able to deploy.
Perhaps it depends, right?
Like if it's for a critical banking system, I don't think you want to do that.
I would prefer the bank not to deploy something with a known security availability and I can
wait a bit longer if possible.
But I feel like if there's a little ceremony, the releases should be able to be done quickly.
Like the process of the release itself should be very quick.
The release should always either succeed or fail.
And you should just be able to do it because you want to release something to production.
And maybe you can like set up a workflow, right?
Like, yes, I'm deploying something and this artifact that I'm deploying has a vulnerability.
So it is automatically cataloged somewhere or someone has mentioned in Slack or like
some security officer champion person in the company is notified and says, hey, this needs
to be fixed right after this release.
So I think some strategies like that, I think could be really helpful.
So but that should be already in place, you said, because that's again, something that
you see a lot when it comes with when we try to create and put monitoring points into the
whole deployment.
One of the discussions like often customers say, well, don't do it then or then you need
to have a different path to release.
And that again, what happens often is that the end it will go through, but it will increase
the release time.
And that brings us to the second question that I have for you guys is what is unacceptable?
Because I get a lot like from our internal team to our customers like this workflow takes
too long to validate or to test to deploy because it has certain checks included into
it.
Yeah, yeah.
I like time wise, I mean, how long may an entire pipeline take?
In my opinion, not more than 10 minutes.
And I've had some I've talked to some of my colleagues, you never like 10 minutes, dude,
like my release takes an hour and a half.
I'm like, I don't have the patience.
I do not have the patient.
And that's kind of point in time, I would just go crazy.
It really depends on the product.
Again, like if you're building some small web applications, if they take longer than
five minutes, I'm like, what are you doing in there if it takes that long?
Like if you're doing a lot of security checks, that's fine.
You know, I mean, if that is needed, that is a good point.
And if no one really cares that it takes longer, fine.
But if it's a critical business process, I feel like you and you know that this might
need releasing very, very soon, very quickly.
You might want to think about maybe having a process that allows you to skip some checks
and just release it quicker.
I don't know some critical software that it might be very vulnerable.
You might want to be releasing quickly, but some small status page application doesn't
really matter all that much.
If that goes quick or bad, quick or slow.
I think it really depends on how big the data, the code is and you want to deploy.
If it's a small app, like a small web app, for example, it can be a few minutes.
But if it's a complete application that has multiple things to do, lots of code, and it
could be much, much, much longer than a small one.
Yeah, it depends.
That's the consultancy answer again.
It is.
I think it also just kind of like I've had projects where there were like 2000, 3000
unit tests.
So yeah, those are going to take some time to run.
And that's not even that's kind of still the build step and the release step.
Then it depends on the, well, on what you're deploying.
If you're going to be deploying to E to IIS, I guess is what you're producing in English.
Just some web server that goes quite fast.
But I've had situations where I wanted to deploy something to Kubernetes, to Azure Container
Apps.
And then it was just quite slow because Kubernetes had to get another VM running and everything.
It really depends on what you're deploying.
But I think it might be something that you might want to talk about with your team.
It's like, hey, this critical business component, I want to talk about this.
I don't want to talk about the status page.
That's not waste time on this.
This critical business component, how long is it going to be?
How long do we want it to take to deploy?
And if it takes longer than that, maybe we want to look into this at some point.
Yeah, and another question that may be related.
In the pre-phase of this recording, Poojan, you talk a lot about chaos engineering.
Maybe you can elaborate a little bit on that.
Why should we have that?
Does it relate to what we have here, the release phase?
Chaos engineering, yeah, I think that's something we discussed in our last episode as well,
right?
Very quickly, I think very quickly.
Yeah, chaos engineering, I think one of the most famous, and I think there are even the
starters of that.
That's Netflix.
The chaos engineering is important to the release, in my opinion, because chaos engineering
means that you just shut down some of your systems without pre-notice, and then you see
if your workflows and everything is in place to get that thing running again without any
issues, or that it also doesn't interrupt your production, right?
So I think, in my opinion, if it would bring chaos engineering to release, it would mean
that it shows that, for example, we have a solution and the whole Kubernetes cluster
with all nodes and ports and every configuration containers, everything running there, and
somebody just destroys everything, that we are able to rebuild that from zero to everything
through the configuration application and the whole operation sites and monitoring,
getting that in place fully automated.
But I think it also, Sander, has different meanings from the developer's perspective.
Yeah, I do.
Maybe you can elaborate on that.
I think maybe we want to talk about it very shortly, because I think this is a perfect
topic for the, like, if we're going to talk about from a developer's perspective, maybe
from an ops perspective, I think it's a really good thing for, like, the monitoring episode
kind of thing.
So for my point of view, what it is really useful for is, let's say we have two services,
like a user service and, like, an order service.
And the fact that the order service is down, just because some system in Azure or Kubernetes
or whatever just kills your system, then your system should at least still be able to restart.
And I think that's more of like a thing that should regulate itself.
So then how does your system react to this?
Do people get notified?
Is it allowed?
Is the service allowed to be down for 10 seconds?
And then do people get notified?
How long is the service allowed to be down?
All of this kind of stuff is what you can use Chaos Engineering for.
Just really truly test, like, hey, 20% of my system is down, but I can still process
orders because that's what earns the business money.
So we can really test, like, how those critical flows work.
Yeah.
Werner, I think you were talking more about, like, how is the code released and, like,
set in place again after that happened?
And I'm thinking more about how the system recovers and how you set up a strategy and
how you can write your code in a way so you can test, like, hey, my code is down, but
I can still process orders or something like that.
It just kind of goes to different ways.
Yeah.
I think, indeed, it has an impact on different stages and with different scenarios, right?
Each scenario, somehow it is applicable, Chaos Engineering.
But yeah, guys, so I think that was the release stage.
I don't know if you guys have anything to add to it because I think to sum it up, it
is somewhere a technical solution in the whole process, right?
Because you need to define your workflows.
But I think in the end, it is the most important to highlight how important it is that when
you do that, that you cooperate your security, like code scanning or any other security features
that you want to have in place, right?
Because defining security is a big as well, right?
Code scanning can be one, but having a change record for your compliance purposes in your
ATSM too could be also part of your whole security strategy.
So to bring that together, and I think at the end, it shows also what we are discussing
here how important that is to have that conversation with your operations, with your security,
and with your developers.
When you are building and writing out your release strategy.
Yeah.
Because it's DevOps, right?
So we're going to involve the developers, the security and operations people.
So we want to know that when we're releasing something, everyone should agree.
And that can be a manual process by asking some people or letting people click on the
approve button, but also automated, which you were talking a little bit about, right?
Making sure that there's some steps involved and some final checks that you can build in.
Yeah.
And it depends on how big the company is.
Because if you have a, we talked about in a previous recording, we did already talk
to a bit of it on that.
If you are on a small company, like you are developing Sander and work with some small
teams, who is doing the security staff?
Normally you have your own security team, but is that available in your organization?
Or should someone from the dev team take care of that role to approve and to see if all
the security checks are done?
So I think how many people are involved in the test, but also in the release phase are
depending on how big the company is and how big the fees are.
Yeah, definitely.
I think that is for all the phases, right?
It sounds maybe crazy, but also don't make it over complicated if you have a small team.
Oh, please don't.
Yes, absolutely.
Don't put four approvers if you are a team of five people.
Ten approvers if you are with three people.
Yeah, that's good.
No, don't make it so complicated.
Like it's, yeah, I could rant about this for half an hour.
I'm stopping myself right now.
Yeah.
Yeah, that's it.
That was the release phase, as you mentioned, Poojan.
If you have questions when you are listening to this recording, please reach out.
We can do a listener session later on when we grab all the questions that we got from
participants, from listeners, we can do a specific session on that.
The next one is about monitoring.
You already mentioned it, Sander.
I think.
Yeah, yeah, yeah.
No problem.
Yeah, definitely.
That's going to be an exciting one because I feel like that's part, at least where I
also still have a lot of stuff to learn, but also have had a lot of lessons learned in
the last few years.
So I think I'm really excited to talk to you guys about it.
Let's do that.
And if you like our videos and podcasts, subscribe on our YouTube channel, on our different podcast
platforms.
Please like and give us a review because that helps us to make these recordings better.
And yeah, hopefully see you till the next time.
Thank you, guys.
Thank you.