1
00:00:00,000 --> 00:00:08,800
Welcome to the Talking Security podcast.

2
00:00:08,800 --> 00:00:21,920
We will talk about items related to Microsoft Security.

3
00:00:21,920 --> 00:00:23,680
So hi, welcome.

4
00:00:23,680 --> 00:00:28,640
There are we again with the Talking Security DevOps podcast.

5
00:00:28,640 --> 00:00:32,680
And again, Pujan and Sander are with me.

6
00:00:32,680 --> 00:00:33,680
So welcome guys.

7
00:00:33,680 --> 00:00:34,680
How are you?

8
00:00:34,680 --> 00:00:38,520
What do we have today?

9
00:00:38,520 --> 00:00:45,860
After last time we did a recording about the test phase within DevSecOps.

10
00:00:45,860 --> 00:00:47,800
What do we have on the menu today?

11
00:00:47,800 --> 00:00:52,720
Pujan, it was all yours, I guess.

12
00:00:52,720 --> 00:00:53,720
That is true.

13
00:00:53,720 --> 00:00:57,560
Yeah, we had some planning in the last episode.

14
00:00:57,560 --> 00:01:02,640
So yeah, indeed, from the last episode where we talked about testing and what you can do

15
00:01:02,640 --> 00:01:08,720
in your test phase and what the vulnerabilities you are, it's about time to spend some time

16
00:01:08,720 --> 00:01:10,920
about release.

17
00:01:10,920 --> 00:01:17,720
In my opinion, it's a very important stage in the whole DevSecOps process, because this

18
00:01:17,720 --> 00:01:24,600
is the moment that you go from your code towards your running environment.

19
00:01:24,600 --> 00:01:28,240
It will be production or acceptance.

20
00:01:28,240 --> 00:01:33,320
So it's a pivotal point where you need to think about the process because you don't

21
00:01:33,320 --> 00:01:39,880
want to, you want to have that process as fluid as possible, but also in control as

22
00:01:39,880 --> 00:01:40,880
possible.

23
00:01:40,880 --> 00:01:47,240
But you don't want to stop the developers like Sander too much by creating all kinds

24
00:01:47,240 --> 00:01:49,720
of issue or performance for them.

25
00:01:49,720 --> 00:01:52,560
So yeah, today we are going to talk about release.

26
00:01:52,560 --> 00:01:54,560
Why it is important?

27
00:01:54,560 --> 00:01:58,320
Which features in there are important together with you guys?

28
00:01:58,320 --> 00:02:07,360
We will also take a look what kind of, what you need to do and what kind of to align this

29
00:02:07,360 --> 00:02:12,280
process as well.

30
00:02:12,280 --> 00:02:15,400
What is your view on the release?

31
00:02:15,400 --> 00:02:16,680
My view on the release?

32
00:02:16,680 --> 00:02:22,200
I find, how do you say that?

33
00:02:22,200 --> 00:02:24,280
And we're talking about like DevSecOps and release.

34
00:02:24,280 --> 00:02:25,920
I think it's a really, really important part.

35
00:02:25,920 --> 00:02:28,880
Like Poojan already mentioned it before.

36
00:02:28,880 --> 00:02:33,200
When I started learning DevOps and CI, CD a few years ago and stuff, like right after

37
00:02:33,200 --> 00:02:34,360
school, it was really cool to me.

38
00:02:34,360 --> 00:02:38,400
I could just deploy my applications immediately and release them.

39
00:02:38,400 --> 00:02:42,160
But I think with DevSecOps, it's maybe even more important because the entire focus is

40
00:02:42,160 --> 00:02:43,160
on the security part.

41
00:02:43,160 --> 00:02:47,120
Like in the previous episodes, we talked about security.

42
00:02:47,120 --> 00:02:50,600
And now if you have found some security issues and you need to fix them, you need to also

43
00:02:50,600 --> 00:02:53,180
be able to release them really, really quickly.

44
00:02:53,180 --> 00:02:57,040
And especially like what Poojan then says is right, if we want to release something,

45
00:02:57,040 --> 00:03:03,760
we should not have that many barriers in place, but the barriers would still ensure that you

46
00:03:03,760 --> 00:03:04,760
are secure.

47
00:03:04,760 --> 00:03:06,720
So you need to find a bit of a balance in that.

48
00:03:06,720 --> 00:03:10,920
But I think it's a really, it's a fun step of being able to have your software.

49
00:03:10,920 --> 00:03:13,240
It's done, it's built, it's secure, it's tested.

50
00:03:13,240 --> 00:03:15,880
And now we're actually going to finally release and do something with it.

51
00:03:15,880 --> 00:03:18,160
And yeah, it's always a fun thing to do.

52
00:03:18,160 --> 00:03:26,520
Yeah, if we want to kick off, what should be the first question you should ask Sander

53
00:03:26,520 --> 00:03:30,440
in this area?

54
00:03:30,440 --> 00:03:32,400
The first question you should ask?

55
00:03:32,400 --> 00:03:33,400
Yeah.

56
00:03:33,400 --> 00:03:40,280
What question should we answer to our listeners within this phase?

57
00:03:40,280 --> 00:03:45,560
What is the topic that you want to cover within this phase?

58
00:03:45,560 --> 00:03:51,000
And probably if you have the question, Poojan can maybe answer that.

59
00:03:51,000 --> 00:03:55,320
I think maybe that's a good thing to ask.

60
00:03:55,320 --> 00:03:56,520
Maybe it's a bit open.

61
00:03:56,520 --> 00:04:01,560
So if we're releasing our code, how do we know that what we are releasing is going to

62
00:04:01,560 --> 00:04:04,880
be released to a secure environment?

63
00:04:04,880 --> 00:04:05,880
Right?

64
00:04:05,880 --> 00:04:10,480
Like we've tested and we've built our code, we did all of our security thingies there,

65
00:04:10,480 --> 00:04:11,720
but now we're going to release it.

66
00:04:11,720 --> 00:04:19,280
So how can we ensure that what we release will be running securely?

67
00:04:19,280 --> 00:04:26,200
So if you look at, if it runs security, it's of course more on the operation side, right?

68
00:04:26,200 --> 00:04:33,800
So then we are more going, for example, if you take our code, because this is the pivotal

69
00:04:33,800 --> 00:04:34,800
point, right?

70
00:04:34,800 --> 00:04:39,320
I think what you're saying is like, release is integrating the code, security and operation

71
00:04:39,320 --> 00:04:40,960
together.

72
00:04:40,960 --> 00:04:45,920
And for us to see how the code is functioning in operation is building on solutions like

73
00:04:45,920 --> 00:04:47,440
Defender for Cloud, right?

74
00:04:47,440 --> 00:04:55,000
For example, we are doing container Kubernetes, or we are doing some database changes by monitoring

75
00:04:55,000 --> 00:05:01,040
our solutions in operations, but solutions in the Defender for Cloud family gives us

76
00:05:01,040 --> 00:05:02,040
a good overview.

77
00:05:02,040 --> 00:05:06,520
Definitely when you integrate it with Defender for Devils, but yeah, we have told that a

78
00:05:06,520 --> 00:05:07,920
lot of times.

79
00:05:07,920 --> 00:05:12,240
That gives a good feel like, okay, what were the changes and how are they functioning in

80
00:05:12,240 --> 00:05:14,820
the production side?

81
00:05:14,820 --> 00:05:16,160
So that would be a key.

82
00:05:16,160 --> 00:05:19,120
And I think of course there's a process on top of that.

83
00:05:19,120 --> 00:05:23,560
It's like after your operations center, your security operations center, which is monitoring

84
00:05:23,560 --> 00:05:28,120
and responding based on those security alerts or findings or full abilities that are coming

85
00:05:28,120 --> 00:05:29,680
from that environment.

86
00:05:29,680 --> 00:05:30,680
Yeah.

87
00:05:30,680 --> 00:05:35,480
So that would be definitely a way forward, in my opinion.

88
00:05:35,480 --> 00:05:36,480
Yeah.

89
00:05:36,480 --> 00:05:37,480
That's a good point.

90
00:05:37,480 --> 00:05:43,080
If you go back, that's a question to both of you guys.

91
00:05:43,080 --> 00:05:50,760
In my opinion, if you take a look at release, it's mainly having defined how we are going

92
00:05:50,760 --> 00:05:55,240
to release and what steps are included in there, right?

93
00:05:55,240 --> 00:06:00,800
That's part of the DevOps, but it's also part of the DevSecOps because things we discussed

94
00:06:00,800 --> 00:06:06,560
last time like code scanning, those are also, in my opinion, part of the release stage,

95
00:06:06,560 --> 00:06:13,280
those are things that you define in your workflows often as well.

96
00:06:13,280 --> 00:06:18,600
So that brings us to a terminology, of course, that's often used like the CI CD, right?

97
00:06:18,600 --> 00:06:22,800
The continuous integration, continuous deployment.

98
00:06:22,800 --> 00:06:29,160
And how do you guys see the uses of that in your environments?

99
00:06:29,160 --> 00:06:33,800
I'm not really sure if I understand what you mean.

100
00:06:33,800 --> 00:06:35,720
The uses of that in my environment?

101
00:06:35,720 --> 00:06:37,480
Can you explain that a bit differently?

102
00:06:37,480 --> 00:06:38,480
Yeah.

103
00:06:38,480 --> 00:06:44,000
Do you have a really, if you build an application, do you have a CI CD in place?

104
00:06:44,000 --> 00:06:50,920
Like when the code change hits certain branches that the certain steps happen, or if your

105
00:06:50,920 --> 00:06:56,160
main branch is updated, certain workflows get triggered to do certain deployments or

106
00:06:56,160 --> 00:07:01,240
packagings or even updating your ETSM or environment?

107
00:07:01,240 --> 00:07:03,040
Yeah, definitely.

108
00:07:03,040 --> 00:07:09,960
So in my case, normally, when main gets updated, we build our code, you can do your security

109
00:07:09,960 --> 00:07:13,280
steps right there that we talked about last episode, which you should definitely listen

110
00:07:13,280 --> 00:07:14,880
to.

111
00:07:14,880 --> 00:07:18,120
And then there's of course, when that is built, it produces an artifact.

112
00:07:18,120 --> 00:07:22,520
At least that's what it's often called in Azure DevOps, which is what I mainly use,

113
00:07:22,520 --> 00:07:23,520
which is then released.

114
00:07:23,520 --> 00:07:27,920
And then I think it's a really good way, would you say, for example, when we have a new Docker

115
00:07:27,920 --> 00:07:33,440
image that we want to, we have scanned, we can upload that to an Azure Container Registry

116
00:07:33,440 --> 00:07:38,480
or an AWS registry or whatever kind of registry it is, where also more security steps can

117
00:07:38,480 --> 00:07:43,520
take place, which I think is a good thing to do in your release steps.

118
00:07:43,520 --> 00:07:48,120
And then of course, we need to publish some more stuff to the cloud or to wherever you're

119
00:07:48,120 --> 00:07:49,120
running.

120
00:07:49,120 --> 00:07:55,940
Which I think kind of flows into a very nice next step is how do you connect securely in

121
00:07:55,940 --> 00:07:59,760
your release step to whatever you're deploying?

122
00:07:59,760 --> 00:08:03,880
I hope people are not losing a username and password when they're deploying something

123
00:08:03,880 --> 00:08:04,880
to Azure.

124
00:08:04,880 --> 00:08:08,040
You should be using the most modern techniques for that.

125
00:08:08,040 --> 00:08:11,400
Maybe you can talk a little bit about that, Poojan, because I think you talked about that

126
00:08:11,400 --> 00:08:12,400
before.

127
00:08:12,400 --> 00:08:13,400
Yeah, definitely.

128
00:08:13,400 --> 00:08:23,040
I think the usage of a username or what we see a lot, I think we all, three of us, is

129
00:08:23,040 --> 00:08:26,680
the usage of search principle and passwords, for example.

130
00:08:26,680 --> 00:08:30,160
Well, there are great tools that can help you.

131
00:08:30,160 --> 00:08:39,560
I mean, go for GitHub and security has already code or password detection pre-commit, so

132
00:08:39,560 --> 00:08:43,560
it doesn't even get to your commit history.

133
00:08:43,560 --> 00:08:48,920
But yeah, we also have like the defender for DevOps, you can integrate that in your workflow

134
00:08:48,920 --> 00:08:53,280
that you do a scan to see if there's any code and if there's not, then you will continue

135
00:08:53,280 --> 00:08:54,440
with that.

136
00:08:54,440 --> 00:09:01,520
So we need and the same is possible with GitHub advanced security, with fair code, for example,

137
00:09:01,520 --> 00:09:03,360
what we discussed last time.

138
00:09:03,360 --> 00:09:12,840
So the usage of security like passwords is really one of the important things.

139
00:09:12,840 --> 00:09:19,560
But one of the things that's also important to check when it comes definitely towards

140
00:09:19,560 --> 00:09:26,880
your if you are doing things on cloud like Azure, definitely important to do like, okay,

141
00:09:26,880 --> 00:09:31,240
the environment which I'm deploying or maybe I'm creating because that's part of your configuration

142
00:09:31,240 --> 00:09:32,240
as well.

143
00:09:32,240 --> 00:09:34,520
Are there any full ability there as well?

144
00:09:34,520 --> 00:09:39,880
So we see different levels when it comes to security in the release page.

145
00:09:39,880 --> 00:09:46,360
But I have like, for example, also customers when it comes to the release is like, okay,

146
00:09:46,360 --> 00:09:52,680
we need to have after each deployment to our production, we need to have a change record

147
00:09:52,680 --> 00:09:54,180
created for that.

148
00:09:54,180 --> 00:09:58,800
So that we are for our compliance purposes.

149
00:09:58,800 --> 00:10:04,680
We see that also a lot in this release workflows coming back.

150
00:10:04,680 --> 00:10:06,920
I don't know if you guys see that as well.

151
00:10:06,920 --> 00:10:10,560
I'm not too familiar with it.

152
00:10:10,560 --> 00:10:20,640
And if we look at compliance in policies, we're talking about secure code, scanning codes

153
00:10:20,640 --> 00:10:22,360
and so on before releasing it.

154
00:10:22,360 --> 00:10:27,280
So we have a secure piece of software that has deploying.

155
00:10:27,280 --> 00:10:31,520
But on the other hand, what about policies?

156
00:10:31,520 --> 00:10:42,960
If you are, let me give you an example, if you do a pull request and to merge that into

157
00:10:42,960 --> 00:10:48,720
a new branch or a new master before deploying.

158
00:10:48,720 --> 00:10:50,960
You can't say that anymore.

159
00:10:50,960 --> 00:10:52,960
You can't say a monster anymore.

160
00:10:52,960 --> 00:10:58,320
I know, but it's an example.

161
00:10:58,320 --> 00:11:03,560
How many approvals do you need?

162
00:11:03,560 --> 00:11:11,120
What steps do you need to take before something gets a final approval and gets to watch a

163
00:11:11,120 --> 00:11:12,120
release?

164
00:11:12,120 --> 00:11:13,120
That's a good point.

165
00:11:13,120 --> 00:11:14,120
I kind of overlooked that.

166
00:11:14,120 --> 00:11:15,520
That's a good point.

167
00:11:15,520 --> 00:11:19,520
So yeah, you're talking about, for example, when code is merged, like, you know, who needs

168
00:11:19,520 --> 00:11:25,160
to, your code can contain things, but people need to, of course, look at the code as well.

169
00:11:25,160 --> 00:11:30,240
But then there's, of course, the really important thing, like, what I usually do is, well, my

170
00:11:30,240 --> 00:11:35,760
code gets merged, then it goes to automatically gets deployed to the test environment and

171
00:11:35,760 --> 00:11:38,280
then just automatically no one has to look at that.

172
00:11:38,280 --> 00:11:43,840
But when we want to deploy the production, someone has to click on the approve button.

173
00:11:43,840 --> 00:11:47,720
And in my case, in my teams, my teams are usually pretty small that I work in.

174
00:11:47,720 --> 00:11:50,320
So I usually just do any member of the team can do it.

175
00:11:50,320 --> 00:11:55,720
So if I want to, I can also just deploy to prod whenever I merged.

176
00:11:55,720 --> 00:11:59,480
It's a small team and it doesn't really matter all that much.

177
00:11:59,480 --> 00:12:02,960
But for bigger companies and for enterprises, that's probably a good thing to have.

178
00:12:02,960 --> 00:12:06,000
Like, hey, maybe multiple members of your team need to look at what you're deploying

179
00:12:06,000 --> 00:12:07,680
to production.

180
00:12:07,680 --> 00:12:13,240
Maybe if you have multiple really big teams and you are affecting some other team's code,

181
00:12:13,240 --> 00:12:18,400
might someone else to also click on the approve button to kind of take a look when stuff is

182
00:12:18,400 --> 00:12:19,400
going to be deployed.

183
00:12:19,400 --> 00:12:21,120
I think that's a good thing to mention.

184
00:12:21,120 --> 00:12:22,120
Yeah.

185
00:12:22,120 --> 00:12:25,800
And Poojan, how does that affect the security teams in that?

186
00:12:25,800 --> 00:12:28,800
Yeah, that's a good.

187
00:12:28,800 --> 00:12:36,480
Yeah, I think it's a great question because but also to add on what Sander mentioned is

188
00:12:36,480 --> 00:12:38,360
the number of people that need to approve things.

189
00:12:38,360 --> 00:12:46,760
I think it all comes down to the level of workflow that you have created in the sense

190
00:12:46,760 --> 00:12:54,120
of how advanced is it like are the code testing, are the functional testing, user testing,

191
00:12:54,120 --> 00:12:57,600
all kinds of tests already in your workflow.

192
00:12:57,600 --> 00:13:01,200
Like building up your change log, building up your documentation, creating your change

193
00:13:01,200 --> 00:13:06,560
requests in your ETSM platforms.

194
00:13:06,560 --> 00:13:10,840
So it depends on the security side as well.

195
00:13:10,840 --> 00:13:19,160
We also say, OK, for example, if you have your CI part of the deployment, the continuous

196
00:13:19,160 --> 00:13:25,960
integration part, pretty well framework and you have your tests, your validations in place,

197
00:13:25,960 --> 00:13:30,160
then your deployment checks will decrease, of course, the number of checks because you

198
00:13:30,160 --> 00:13:35,440
can rely on your tests more.

199
00:13:35,440 --> 00:13:42,120
And the same is for security when it comes if you have done definitely active testing

200
00:13:42,120 --> 00:13:46,280
on your code.

201
00:13:46,280 --> 00:13:54,520
Let's say, for example, you have multiple solutions and you deploy your code changes

202
00:13:54,520 --> 00:14:00,160
to an acceptance environment before you deploy to production.

203
00:14:00,160 --> 00:14:05,960
And there you have your code validation part in place and you have that environment also

204
00:14:05,960 --> 00:14:11,000
connected to something like Defender for cloud.

205
00:14:11,000 --> 00:14:18,080
Then you can both find no vulnerabilities or any misconfiguration or code violations.

206
00:14:18,080 --> 00:14:24,080
Then it's much easier to say, OK, the deployment to production needs one approval.

207
00:14:24,080 --> 00:14:25,960
That's a good point.

208
00:14:25,960 --> 00:14:30,400
So to summarize, I think what you're saying is if you have a lot of active monitoring,

209
00:14:30,400 --> 00:14:34,960
for example, in place, the code is running 24-7 somewhere in the cloud.

210
00:14:34,960 --> 00:14:39,240
And there's also some monitoring solution that is checking all of that.

211
00:14:39,240 --> 00:14:44,160
If there's loads of stuff already before or during the runtime of your application, you

212
00:14:44,160 --> 00:14:52,080
don't need to add 20 checks and approvals and ceremonies to the release process.

213
00:14:52,080 --> 00:14:55,840
You need to think, especially for your team and for the product you're working on, what

214
00:14:55,840 --> 00:14:57,840
part needs to be secured?

215
00:14:57,840 --> 00:15:02,640
If I'm deploying some really tricky software that needs to be very well secured during

216
00:15:02,640 --> 00:15:08,200
the build process, but doesn't really accept any user inputs or cannot really be talked

217
00:15:08,200 --> 00:15:14,920
to, maybe then it's more important to do the build security stuff and let go of the release

218
00:15:14,920 --> 00:15:19,520
process, do some basic checks, some basic approvals.

219
00:15:19,520 --> 00:15:20,520
And that's it.

220
00:15:20,520 --> 00:15:21,520
Yeah.

221
00:15:21,520 --> 00:15:24,680
Because you need to ask yourself, I think, the following questions.

222
00:15:24,680 --> 00:15:34,240
If you have three people approving your deployment, what are they actually approving?

223
00:15:34,240 --> 00:15:36,840
And are they aware of what they are?

224
00:15:36,840 --> 00:15:37,840
Yeah.

225
00:15:37,840 --> 00:15:38,840
Usually it's nothing.

226
00:15:38,840 --> 00:15:44,880
And usually when I work in a team or even when I set up the policies and stuff, I usually

227
00:15:44,880 --> 00:15:51,200
always click on the person that created the release is also allowed to merge it.

228
00:15:51,200 --> 00:15:54,000
And if you've got to be very puristic about it, you could say, yeah, but if that person

229
00:15:54,000 --> 00:15:56,920
has bad intentions, they could release something to production.

230
00:15:56,920 --> 00:15:59,280
I never assume people have bad intentions.

231
00:15:59,280 --> 00:16:03,000
And I've had multiple occasions where I had to release something in the evening and someone

232
00:16:03,000 --> 00:16:09,080
had enabled that I was not allowed to release to production my own code.

233
00:16:09,080 --> 00:16:12,720
So I had to call someone to turn on their computer so they could click on a blue button

234
00:16:12,720 --> 00:16:13,720
for me.

235
00:16:13,720 --> 00:16:14,720
So it's really important.

236
00:16:14,720 --> 00:16:15,720
Yeah.

237
00:16:15,720 --> 00:16:19,200
Personally, I'm never a big fan of the like, other people have to look at it.

238
00:16:19,200 --> 00:16:21,360
I think it's something you talk to with your team.

239
00:16:21,360 --> 00:16:24,240
You need to communicate with your team about this kind of stuff.

240
00:16:24,240 --> 00:16:29,000
But yeah, it's more of a sidetrack where I'm going right now.

241
00:16:29,000 --> 00:16:30,000
Sorry.

242
00:16:30,000 --> 00:16:34,920
Well, to point out the first point you made, I never assume anybody.

243
00:16:34,920 --> 00:16:37,640
Well, that's not really in line with assume bridge.

244
00:16:37,640 --> 00:16:42,360
So you need to always assume someone is trying to because that person doesn't need to be

245
00:16:42,360 --> 00:16:43,560
that person.

246
00:16:43,560 --> 00:16:47,200
So that's, I think, the whole reason why you have the DevSecond.

247
00:16:47,200 --> 00:16:48,200
That's a good point.

248
00:16:48,200 --> 00:16:49,200
That's a good point.

249
00:16:49,200 --> 00:16:54,160
So on the second part, I think that's one of the biggest challenges for organizations

250
00:16:54,160 --> 00:17:01,320
because you don't want to call up your colleague at 7 p.m. in the evening or later even in

251
00:17:01,320 --> 00:17:03,800
the hope that they answer to click about that.

252
00:17:03,800 --> 00:17:09,040
In my opinion, I think definitely when it comes to security and also developers, developers

253
00:17:09,040 --> 00:17:11,260
want always to have speed, right?

254
00:17:11,260 --> 00:17:14,280
You want to get things done as soon as possible.

255
00:17:14,280 --> 00:17:22,320
And security wants to be in charge and see what has been done and what's going to happen.

256
00:17:22,320 --> 00:17:28,400
And that's why the release is so important because if you bring all those requirements

257
00:17:28,400 --> 00:17:33,880
into one workflow, like what is security asking us to do and what is developers asking us

258
00:17:33,880 --> 00:17:36,160
to do and what's operation asking us to do?

259
00:17:36,160 --> 00:17:46,280
If we have that process in one overview and we change that into our CI CD workflows, then

260
00:17:46,280 --> 00:17:49,560
in my opinion, you don't need to call someone at 7 p.m.

261
00:17:49,560 --> 00:17:55,440
There should be that control should be in place to give you the possibility to do your

262
00:17:55,440 --> 00:18:05,640
thing while the monitoring on security is high alert.

263
00:18:05,640 --> 00:18:10,560
From the operations side, because we are talking about security and we are also talking about

264
00:18:10,560 --> 00:18:14,720
the developers, but I think it has a really huge impact on the operations as well, right?

265
00:18:14,720 --> 00:18:20,840
Because they need to be aware what has been changed because at the end, often they will

266
00:18:20,840 --> 00:18:28,400
get the calls from the customers or from the employees that something is not working.

267
00:18:28,400 --> 00:18:33,680
How do you guys integrate that part as well?

268
00:18:33,680 --> 00:18:34,680
That's actually a good point.

269
00:18:34,680 --> 00:18:38,240
I haven't really thought about it in that way.

270
00:18:38,240 --> 00:18:44,640
Yeah, usually what the team lead or the product owner does is just send an email to the organization.

271
00:18:44,640 --> 00:18:46,440
Hey, this is going to be changed.

272
00:18:46,440 --> 00:18:49,280
But I guess that works for small companies, for bigger enterprises.

273
00:18:49,280 --> 00:18:54,200
I've seen systems where they have like build change logs and then you can just kind of

274
00:18:54,200 --> 00:18:59,740
go to an internal page where you can always see what version of a system is running when

275
00:18:59,740 --> 00:19:04,240
it was updated and the change logs and stuff you can see right there.

276
00:19:04,240 --> 00:19:05,240
That's kind of interesting.

277
00:19:05,240 --> 00:19:09,600
On the other hand, I also feel like if you're going to do breaking changes during your release,

278
00:19:09,600 --> 00:19:13,520
that should definitely be something that is communicated beforehand or in fact, you should

279
00:19:13,520 --> 00:19:18,960
never do breaking changes until everyone is ready for them.

280
00:19:18,960 --> 00:19:21,960
But yeah, how do you communicate with it?

281
00:19:21,960 --> 00:19:26,400
Is it also kind of an important part, especially if it's going to be something secure that

282
00:19:26,400 --> 00:19:30,440
you're going to be deploying or you're going to be deploying changes to your configuration?

283
00:19:30,440 --> 00:19:34,600
For example, some biceps, some infrastructure scope, that's going to be deployed with different

284
00:19:34,600 --> 00:19:38,600
firewall rules, different security stuff, something that you definitely need to keep

285
00:19:38,600 --> 00:19:42,300
in mind when you're going to be deploying that in a big enterprise.

286
00:19:42,300 --> 00:19:44,320
Everyone should be aware of that.

287
00:19:44,320 --> 00:19:45,320
Yeah.

288
00:19:45,320 --> 00:19:54,140
And I think in the end, what you see a lot is that security teams are trying to step

289
00:19:54,140 --> 00:19:56,600
in into that developer's ecosystem, right?

290
00:19:56,600 --> 00:20:01,160
To get a better understanding and controls in place, and then the same should be there

291
00:20:01,160 --> 00:20:02,160
for operations.

292
00:20:02,160 --> 00:20:11,680
And I think that's what makes release important is to get a clear overview of all those requirements

293
00:20:11,680 --> 00:20:13,560
from those teams as well.

294
00:20:13,560 --> 00:20:23,080
But what do you see, for example, as challenges when it comes to, for example, let's say we

295
00:20:23,080 --> 00:20:30,080
are having our CI CD in place, we are having our code validation and scanning what we discussed

296
00:20:30,080 --> 00:20:33,520
last time and previous, everything in place.

297
00:20:33,520 --> 00:20:40,720
And once you do a change and that change got blocked because it contains certain security

298
00:20:40,720 --> 00:20:41,720
vulnerabilities.

299
00:20:41,720 --> 00:20:43,720
Yeah, that's tough.

300
00:20:43,720 --> 00:20:50,920
I mean, I guess it's kind of the boring answer if it depends.

301
00:20:50,920 --> 00:20:57,960
I still feel like if there's something happening in the business and something needs to be

302
00:20:57,960 --> 00:21:01,160
deployed now, and suddenly I get like a CVE, right?

303
00:21:01,160 --> 00:21:07,320
Like a critical vulnerability and the E, I don't know what that stands for anymore.

304
00:21:07,320 --> 00:21:10,880
You should, I guess, still be able to deploy it.

305
00:21:10,880 --> 00:21:14,480
Before you were deploying something and you didn't know there was a vulnerability, but

306
00:21:14,480 --> 00:21:17,080
now you're deploying something and you know that there's a vulnerability.

307
00:21:17,080 --> 00:21:21,320
And I think it's just kind of on you, you as the human need to make the decision, am

308
00:21:21,320 --> 00:21:22,720
I going to deploy this?

309
00:21:22,720 --> 00:21:23,720
Right?

310
00:21:23,720 --> 00:21:24,720
So that's kind of where it depends.

311
00:21:24,720 --> 00:21:28,760
If the business is losing money at this moment because something needs to be deployed, you

312
00:21:28,760 --> 00:21:32,720
can't tell the business, no, we have a security, we have a tiny security issue.

313
00:21:32,720 --> 00:21:36,040
You need to still be able to deploy.

314
00:21:36,040 --> 00:21:37,520
Perhaps it depends, right?

315
00:21:37,520 --> 00:21:42,000
Like if it's for a critical banking system, I don't think you want to do that.

316
00:21:42,000 --> 00:21:46,240
I would prefer the bank not to deploy something with a known security availability and I can

317
00:21:46,240 --> 00:21:48,320
wait a bit longer if possible.

318
00:21:48,320 --> 00:21:53,720
But I feel like if there's a little ceremony, the releases should be able to be done quickly.

319
00:21:53,720 --> 00:21:57,360
Like the process of the release itself should be very quick.

320
00:21:57,360 --> 00:22:00,920
The release should always either succeed or fail.

321
00:22:00,920 --> 00:22:06,600
And you should just be able to do it because you want to release something to production.

322
00:22:06,600 --> 00:22:09,400
And maybe you can like set up a workflow, right?

323
00:22:09,400 --> 00:22:14,800
Like, yes, I'm deploying something and this artifact that I'm deploying has a vulnerability.

324
00:22:14,800 --> 00:22:19,040
So it is automatically cataloged somewhere or someone has mentioned in Slack or like

325
00:22:19,040 --> 00:22:25,320
some security officer champion person in the company is notified and says, hey, this needs

326
00:22:25,320 --> 00:22:27,680
to be fixed right after this release.

327
00:22:27,680 --> 00:22:32,800
So I think some strategies like that, I think could be really helpful.

328
00:22:32,800 --> 00:22:38,960
So but that should be already in place, you said, because that's again, something that

329
00:22:38,960 --> 00:22:48,120
you see a lot when it comes with when we try to create and put monitoring points into the

330
00:22:48,120 --> 00:22:49,120
whole deployment.

331
00:22:49,120 --> 00:22:54,760
One of the discussions like often customers say, well, don't do it then or then you need

332
00:22:54,760 --> 00:22:58,680
to have a different path to release.

333
00:22:58,680 --> 00:23:04,960
And that again, what happens often is that the end it will go through, but it will increase

334
00:23:04,960 --> 00:23:07,120
the release time.

335
00:23:07,120 --> 00:23:13,960
And that brings us to the second question that I have for you guys is what is unacceptable?

336
00:23:13,960 --> 00:23:19,600
Because I get a lot like from our internal team to our customers like this workflow takes

337
00:23:19,600 --> 00:23:26,520
too long to validate or to test to deploy because it has certain checks included into

338
00:23:26,520 --> 00:23:27,520
it.

339
00:23:27,520 --> 00:23:28,520
Yeah, yeah.

340
00:23:28,520 --> 00:23:33,240
I like time wise, I mean, how long may an entire pipeline take?

341
00:23:33,240 --> 00:23:35,080
In my opinion, not more than 10 minutes.

342
00:23:35,080 --> 00:23:38,840
And I've had some I've talked to some of my colleagues, you never like 10 minutes, dude,

343
00:23:38,840 --> 00:23:40,800
like my release takes an hour and a half.

344
00:23:40,800 --> 00:23:42,560
I'm like, I don't have the patience.

345
00:23:42,560 --> 00:23:43,840
I do not have the patient.

346
00:23:43,840 --> 00:23:47,120
And that's kind of point in time, I would just go crazy.

347
00:23:47,120 --> 00:23:48,440
It really depends on the product.

348
00:23:48,440 --> 00:23:52,360
Again, like if you're building some small web applications, if they take longer than

349
00:23:52,360 --> 00:23:55,880
five minutes, I'm like, what are you doing in there if it takes that long?

350
00:23:55,880 --> 00:23:58,160
Like if you're doing a lot of security checks, that's fine.

351
00:23:58,160 --> 00:24:01,480
You know, I mean, if that is needed, that is a good point.

352
00:24:01,480 --> 00:24:04,640
And if no one really cares that it takes longer, fine.

353
00:24:04,640 --> 00:24:10,000
But if it's a critical business process, I feel like you and you know that this might

354
00:24:10,000 --> 00:24:13,920
need releasing very, very soon, very quickly.

355
00:24:13,920 --> 00:24:18,040
You might want to think about maybe having a process that allows you to skip some checks

356
00:24:18,040 --> 00:24:19,840
and just release it quicker.

357
00:24:19,840 --> 00:24:25,000
I don't know some critical software that it might be very vulnerable.

358
00:24:25,000 --> 00:24:29,920
You might want to be releasing quickly, but some small status page application doesn't

359
00:24:29,920 --> 00:24:31,200
really matter all that much.

360
00:24:31,200 --> 00:24:34,280
If that goes quick or bad, quick or slow.

361
00:24:34,280 --> 00:24:40,360
I think it really depends on how big the data, the code is and you want to deploy.

362
00:24:40,360 --> 00:24:46,160
If it's a small app, like a small web app, for example, it can be a few minutes.

363
00:24:46,160 --> 00:24:56,280
But if it's a complete application that has multiple things to do, lots of code, and it

364
00:24:56,280 --> 00:25:00,600
could be much, much, much longer than a small one.

365
00:25:00,600 --> 00:25:03,320
Yeah, it depends.

366
00:25:03,320 --> 00:25:06,600
That's the consultancy answer again.

367
00:25:06,600 --> 00:25:07,600
It is.

368
00:25:07,600 --> 00:25:12,080
I think it also just kind of like I've had projects where there were like 2000, 3000

369
00:25:12,080 --> 00:25:13,080
unit tests.

370
00:25:13,080 --> 00:25:14,640
So yeah, those are going to take some time to run.

371
00:25:14,640 --> 00:25:18,280
And that's not even that's kind of still the build step and the release step.

372
00:25:18,280 --> 00:25:22,960
Then it depends on the, well, on what you're deploying.

373
00:25:22,960 --> 00:25:27,580
If you're going to be deploying to E to IIS, I guess is what you're producing in English.

374
00:25:27,580 --> 00:25:29,120
Just some web server that goes quite fast.

375
00:25:29,120 --> 00:25:33,280
But I've had situations where I wanted to deploy something to Kubernetes, to Azure Container

376
00:25:33,280 --> 00:25:34,280
Apps.

377
00:25:34,280 --> 00:25:40,600
And then it was just quite slow because Kubernetes had to get another VM running and everything.

378
00:25:40,600 --> 00:25:42,000
It really depends on what you're deploying.

379
00:25:42,000 --> 00:25:45,120
But I think it might be something that you might want to talk about with your team.

380
00:25:45,120 --> 00:25:48,480
It's like, hey, this critical business component, I want to talk about this.

381
00:25:48,480 --> 00:25:50,240
I don't want to talk about the status page.

382
00:25:50,240 --> 00:25:51,680
That's not waste time on this.

383
00:25:51,680 --> 00:25:54,200
This critical business component, how long is it going to be?

384
00:25:54,200 --> 00:25:57,200
How long do we want it to take to deploy?

385
00:25:57,200 --> 00:26:01,800
And if it takes longer than that, maybe we want to look into this at some point.

386
00:26:01,800 --> 00:26:09,840
Yeah, and another question that may be related.

387
00:26:09,840 --> 00:26:16,840
In the pre-phase of this recording, Poojan, you talk a lot about chaos engineering.

388
00:26:16,840 --> 00:26:20,480
Maybe you can elaborate a little bit on that.

389
00:26:20,480 --> 00:26:23,600
Why should we have that?

390
00:26:23,600 --> 00:26:30,160
Does it relate to what we have here, the release phase?

391
00:26:30,160 --> 00:26:36,000
Chaos engineering, yeah, I think that's something we discussed in our last episode as well,

392
00:26:36,000 --> 00:26:37,000
right?

393
00:26:37,000 --> 00:26:40,000
Very quickly, I think very quickly.

394
00:26:40,000 --> 00:26:46,680
Yeah, chaos engineering, I think one of the most famous, and I think there are even the

395
00:26:46,680 --> 00:26:49,320
starters of that.

396
00:26:49,320 --> 00:26:53,480
That's Netflix.

397
00:26:53,480 --> 00:27:01,680
The chaos engineering is important to the release, in my opinion, because chaos engineering

398
00:27:01,680 --> 00:27:08,640
means that you just shut down some of your systems without pre-notice, and then you see

399
00:27:08,640 --> 00:27:13,960
if your workflows and everything is in place to get that thing running again without any

400
00:27:13,960 --> 00:27:21,200
issues, or that it also doesn't interrupt your production, right?

401
00:27:21,200 --> 00:27:27,480
So I think, in my opinion, if it would bring chaos engineering to release, it would mean

402
00:27:27,480 --> 00:27:33,560
that it shows that, for example, we have a solution and the whole Kubernetes cluster

403
00:27:33,560 --> 00:27:38,560
with all nodes and ports and every configuration containers, everything running there, and

404
00:27:38,560 --> 00:27:48,600
somebody just destroys everything, that we are able to rebuild that from zero to everything

405
00:27:48,600 --> 00:27:56,400
through the configuration application and the whole operation sites and monitoring,

406
00:27:56,400 --> 00:28:01,400
getting that in place fully automated.

407
00:28:01,400 --> 00:28:05,560
But I think it also, Sander, has different meanings from the developer's perspective.

408
00:28:05,560 --> 00:28:06,560
Yeah, I do.

409
00:28:06,560 --> 00:28:08,080
Maybe you can elaborate on that.

410
00:28:08,080 --> 00:28:12,680
I think maybe we want to talk about it very shortly, because I think this is a perfect

411
00:28:12,680 --> 00:28:17,160
topic for the, like, if we're going to talk about from a developer's perspective, maybe

412
00:28:17,160 --> 00:28:21,080
from an ops perspective, I think it's a really good thing for, like, the monitoring episode

413
00:28:21,080 --> 00:28:22,080
kind of thing.

414
00:28:22,080 --> 00:28:28,280
So for my point of view, what it is really useful for is, let's say we have two services,

415
00:28:28,280 --> 00:28:31,080
like a user service and, like, an order service.

416
00:28:31,080 --> 00:28:36,040
And the fact that the order service is down, just because some system in Azure or Kubernetes

417
00:28:36,040 --> 00:28:43,360
or whatever just kills your system, then your system should at least still be able to restart.

418
00:28:43,360 --> 00:28:47,760
And I think that's more of like a thing that should regulate itself.

419
00:28:47,760 --> 00:28:50,800
So then how does your system react to this?

420
00:28:50,800 --> 00:28:53,600
Do people get notified?

421
00:28:53,600 --> 00:28:54,600
Is it allowed?

422
00:28:54,600 --> 00:28:56,400
Is the service allowed to be down for 10 seconds?

423
00:28:56,400 --> 00:28:57,740
And then do people get notified?

424
00:28:57,740 --> 00:28:59,520
How long is the service allowed to be down?

425
00:28:59,520 --> 00:29:01,920
All of this kind of stuff is what you can use Chaos Engineering for.

426
00:29:01,920 --> 00:29:06,480
Just really truly test, like, hey, 20% of my system is down, but I can still process

427
00:29:06,480 --> 00:29:08,780
orders because that's what earns the business money.

428
00:29:08,780 --> 00:29:11,960
So we can really test, like, how those critical flows work.

429
00:29:11,960 --> 00:29:13,560
Yeah.

430
00:29:13,560 --> 00:29:18,280
Werner, I think you were talking more about, like, how is the code released and, like,

431
00:29:18,280 --> 00:29:20,880
set in place again after that happened?

432
00:29:20,880 --> 00:29:27,400
And I'm thinking more about how the system recovers and how you set up a strategy and

433
00:29:27,400 --> 00:29:31,640
how you can write your code in a way so you can test, like, hey, my code is down, but

434
00:29:31,640 --> 00:29:33,640
I can still process orders or something like that.

435
00:29:33,640 --> 00:29:35,880
It just kind of goes to different ways.

436
00:29:35,880 --> 00:29:36,880
Yeah.

437
00:29:36,880 --> 00:29:43,280
I think, indeed, it has an impact on different stages and with different scenarios, right?

438
00:29:43,280 --> 00:29:49,520
Each scenario, somehow it is applicable, Chaos Engineering.

439
00:29:49,520 --> 00:29:54,360
But yeah, guys, so I think that was the release stage.

440
00:29:54,360 --> 00:30:00,160
I don't know if you guys have anything to add to it because I think to sum it up, it

441
00:30:00,160 --> 00:30:04,760
is somewhere a technical solution in the whole process, right?

442
00:30:04,760 --> 00:30:07,120
Because you need to define your workflows.

443
00:30:07,120 --> 00:30:14,680
But I think in the end, it is the most important to highlight how important it is that when

444
00:30:14,680 --> 00:30:22,600
you do that, that you cooperate your security, like code scanning or any other security features

445
00:30:22,600 --> 00:30:25,520
that you want to have in place, right?

446
00:30:25,520 --> 00:30:28,440
Because defining security is a big as well, right?

447
00:30:28,440 --> 00:30:33,400
Code scanning can be one, but having a change record for your compliance purposes in your

448
00:30:33,400 --> 00:30:39,480
ATSM too could be also part of your whole security strategy.

449
00:30:39,480 --> 00:30:45,560
So to bring that together, and I think at the end, it shows also what we are discussing

450
00:30:45,560 --> 00:30:50,560
here how important that is to have that conversation with your operations, with your security,

451
00:30:50,560 --> 00:30:52,520
and with your developers.

452
00:30:52,520 --> 00:30:56,480
When you are building and writing out your release strategy.

453
00:30:56,480 --> 00:30:57,480
Yeah.

454
00:30:57,480 --> 00:30:59,920
Because it's DevOps, right?

455
00:30:59,920 --> 00:31:04,460
So we're going to involve the developers, the security and operations people.

456
00:31:04,460 --> 00:31:08,920
So we want to know that when we're releasing something, everyone should agree.

457
00:31:08,920 --> 00:31:12,920
And that can be a manual process by asking some people or letting people click on the

458
00:31:12,920 --> 00:31:16,880
approve button, but also automated, which you were talking a little bit about, right?

459
00:31:16,880 --> 00:31:21,680
Making sure that there's some steps involved and some final checks that you can build in.

460
00:31:21,680 --> 00:31:22,680
Yeah.

461
00:31:22,680 --> 00:31:25,200
And it depends on how big the company is.

462
00:31:25,200 --> 00:31:30,240
Because if you have a, we talked about in a previous recording, we did already talk

463
00:31:30,240 --> 00:31:32,320
to a bit of it on that.

464
00:31:32,320 --> 00:31:38,000
If you are on a small company, like you are developing Sander and work with some small

465
00:31:38,000 --> 00:31:43,200
teams, who is doing the security staff?

466
00:31:43,200 --> 00:31:49,320
Normally you have your own security team, but is that available in your organization?

467
00:31:49,320 --> 00:31:59,640
Or should someone from the dev team take care of that role to approve and to see if all

468
00:31:59,640 --> 00:32:01,440
the security checks are done?

469
00:32:01,440 --> 00:32:08,120
So I think how many people are involved in the test, but also in the release phase are

470
00:32:08,120 --> 00:32:11,480
depending on how big the company is and how big the fees are.

471
00:32:11,480 --> 00:32:14,800
Yeah, definitely.

472
00:32:14,800 --> 00:32:18,880
I think that is for all the phases, right?

473
00:32:18,880 --> 00:32:24,320
It sounds maybe crazy, but also don't make it over complicated if you have a small team.

474
00:32:24,320 --> 00:32:26,000
Oh, please don't.

475
00:32:26,000 --> 00:32:27,000
Yes, absolutely.

476
00:32:27,000 --> 00:32:30,080
Don't put four approvers if you are a team of five people.

477
00:32:30,080 --> 00:32:33,000
Ten approvers if you are with three people.

478
00:32:33,000 --> 00:32:34,000
Yeah, that's good.

479
00:32:34,000 --> 00:32:36,360
No, don't make it so complicated.

480
00:32:36,360 --> 00:32:39,640
Like it's, yeah, I could rant about this for half an hour.

481
00:32:39,640 --> 00:32:41,600
I'm stopping myself right now.

482
00:32:41,600 --> 00:32:42,600
Yeah.

483
00:32:42,600 --> 00:32:46,480
Yeah, that's it.

484
00:32:46,480 --> 00:32:50,240
That was the release phase, as you mentioned, Poojan.

485
00:32:50,240 --> 00:32:56,600
If you have questions when you are listening to this recording, please reach out.

486
00:32:56,600 --> 00:33:05,240
We can do a listener session later on when we grab all the questions that we got from

487
00:33:05,240 --> 00:33:12,040
participants, from listeners, we can do a specific session on that.

488
00:33:12,040 --> 00:33:17,280
The next one is about monitoring.

489
00:33:17,280 --> 00:33:19,280
You already mentioned it, Sander.

490
00:33:19,280 --> 00:33:20,280
I think.

491
00:33:20,280 --> 00:33:21,280
Yeah, yeah, yeah.

492
00:33:21,280 --> 00:33:22,280
No problem.

493
00:33:22,280 --> 00:33:23,280
Yeah, definitely.

494
00:33:23,280 --> 00:33:26,640
That's going to be an exciting one because I feel like that's part, at least where I

495
00:33:26,640 --> 00:33:30,400
also still have a lot of stuff to learn, but also have had a lot of lessons learned in

496
00:33:30,400 --> 00:33:31,400
the last few years.

497
00:33:31,400 --> 00:33:33,840
So I think I'm really excited to talk to you guys about it.

498
00:33:33,840 --> 00:33:35,240
Let's do that.

499
00:33:35,240 --> 00:33:43,520
And if you like our videos and podcasts, subscribe on our YouTube channel, on our different podcast

500
00:33:43,520 --> 00:33:44,680
platforms.

501
00:33:44,680 --> 00:33:50,240
Please like and give us a review because that helps us to make these recordings better.

502
00:33:50,240 --> 00:33:52,360
And yeah, hopefully see you till the next time.

503
00:33:52,360 --> 00:33:53,360
Thank you, guys.

504
00:33:53,360 --> 00:33:54,360
Thank you.

505
00:33:54,360 --> 00:34:13,720
Thank you.