Welcome to the Talking Security Podcast. We will talk about items related to Microsoft
Security.
Hello, welcome again to the new recording of the Talking Security Podcast. This time
again in the Defender for DevSecOps series and we're back with Pouyan and Sander. What
are we going to do today?
Today we're going to talk about the build section of DevSecOps. So last episode we talked
about code and this episode we're going to talk all about the build environment and build
process.
How to build such piece of software or some infrastructure for example to enroll that.
So let's dive in. Do we have, first of all, do we have anything to announce? It's almost
MVP Summit, so we're both, all three of us, we are moving to America for a week. So we
will do some recordings there, I guess.
Hopefully.
We'll take our gear, not this gear, but we'll take the gear with us and then let's see if
we can talk to some people over there. So really looking forward. It will be my first
time so hopefully it will be awesome. First time in Seattle.
Your first time at the Summit as well?
Yes.
Oh really? I thought you were gone there for like a million times already.
I think I'm the youngest MVP in this room.
You're certainly not the youngest person in the room.
No, no, no, no, definitely not. But let's dive into the DevSecOps build process. What
is it? What things do we need to cover?
Well I kind of see this in two different sections. So you have like the build process and you
have the build environment. And I think it's really, really important to talk about security
aspect here because in the normal DevOps flow, when someone talks about DevOps they usually
talk about CI, CD, like building your software when something changes and then releasing
it automatically instead of doing it by hand with FTP or just doing stuff by hand. But
with DevSecOps it's all about the security aspect. So how do we make sure that this building
process is very secure? Because it's very easy to introduce security issues with this.
Yeah, first, first dive in the process and the environment, but first the process. Let's
see what kind of things do we need to cover? Like the secure build, you talk already about
security. What kind of aspects around security do we need to look into?
Well I think it's good before we dive in into the aspects is to define what we mean when
we are talking about build, right? Because when we are doing build on an infrastructure
project, it's completely different than when we are talking about build, when we are about
a.NET application where you need to generate stuff. So I think that's initially because
after that we can start thinking, okay, there are multiple parallels that we can discuss
and see, okay, what are the security challenges on that aspect? Is that a good intro?
Yeah, that sounds good. I think it's a good thing that you're saying, Clique. We're not
just releasing source code onto a machine. Also if we're talking about stuff like infrastructure
as code or like bicep or stuff like this, you're not sending the bicep files to Azure.
You're actually sending the result. So I think the build process could really be summed up
as taking the source of a project and turning it into an artifact, turning it into something
that will eventually run on a machine somewhere. And that can take many forms. Like in.NET
you're compiling an application with a normal website. You're maybe just minifying it, but
the HTML, the CSS and JavaScript stays the same. And with bicep, well, bicep is just
turning it into ARM templates in the end.
Yeah. I think what we also were discussing on before the start is how the build is also
changing, right? Because with the introduction of the infrastructure as code, for example,
what I told you guys earlier, in the early days when we did the infrastructure as code
in Azure DevOps, we needed to do build and then create an artifact which we could use
in the release deployed, which is now not the case anymore as well, right?
Kinda, kinda. So now you use, yeah, you used to like click ops kind of the pipeline together
with your by hand and it worked and it was really nice to begin with, but you didn't
really have any backups. It wasn't in your Git repository or something like that. Nowadays,
exactly what you say, you had like, well, you had, you used to have like a build section
and a release section. And nowadays the modern approach is to use like a YAML file with your
pipeline in it. And in there you define all of the build steps and then you can also define
your release steps in there. Usually you do that in a separate stage or in a separate
file. So it's nicely separated, but now it's a whole more, it's a, it's a different approach
to building and releasing your application.
Yeah. So if you focus on the, let's start with the infrastructure as code, for example,
if you take a look when we are doing infrastructure as code or we are doing conditional access
deployment through DevOps processes, right? Things, configuration as code. How do you
see, how do we see the build stage and do we see any security issues in there as well?
When only having infrastructure as code and deploying workplace related items, identity
related items like conditional access, for example, can we deploy that in a secure way
and how should we do that?
So the authentication part you mean, or is it for example, for example,
you need to connect to a certain Azure actor and try the environment. You need to deploy
within the entire ID. You need to deploy the conditional access policies. And in this case
we need an app rack that app rack needs some permissions, right? How do we secure that
optimally? So nowadays we're using Key Vault. Is that the right way or do we store that
in another way?
So I think what you're going to is like, for this is of course kind of Azure DevOps specific,
but I think you're leaning into like the service connections bit. Like how do we connect our
Azure DevOps environment to our Azure environment to make the changes and how do we create a
secure connection?
It could be that that could be the way, but you can also run a script on a PowerShell
script for example, or whatever. So there needs to be a connection and that connection
can be established in different ways in my opinion.
Yeah, yeah. You can definitely do multiple things. Yeah.
So what you often see is I think at the moment that in Azure DevOps or people, you can create
service connection, which you can use in your pipelines for example. But we had also seen
scenarios where people hijacked the connection or the deployment process and then got the
token out of the connectivity. And then through that they hopped through. So I think that's
what you're meaning with the authentication part. But GitHub it's different. I think you
need to store, you cannot define connections. You need to define credentials, which you
can use to connect.
Not really sure. I think under the hood, it kind of works the same in Azure DevOps where
you create like a service connection to Azure. You can like use like an app registration,
for example, like a service principle under the hood. But more modern approaches are like
using a managed identity or a workload federation, something. Does someone here know the correct
term? Some workload federation.
Federated credential. That's something new. And that's really changing it because it's
like managed identity, but it's so you don't have a secret anymore.
Yeah, something like that. Yeah. And I think using, I mean, you can definitely use a PowerShell
script, right? To like in your pipeline, say I want to connect to Azure. And instead of
using a service connection, I'm just going to do it myself. You could, but I think this
is going to come up often more in this episode is why would you do it yourself if you can
use like a cloud hosted, Microsoft hosted or provider hosted option, like a service
connection.
And if we want, if we want to use that service connection, for example, what do we need to
do to secure that optimally? Because last days I talked to a few customers regarding
phishing resistant MFA that is more user based. From a user perspective, there are multiple
challenges regarding MFA and phishing regard with the MFA token and so on. Is that, is
that the same in the area with the service connection and using a managed, a managed
identity of service principle, whatever, do we face the same challenges as well in, in
creating connections between a DevOps environment, the Git em environment and a backend system,
for example, Azure Active Directory or the, the, the build environment where you are publishing
your, your software.
I think, I think definitely, I think then it's more like, I don't, if you're using
service connections, what I'm trying to say is that then it really depends on how much
access you give that Azure resource, right? Like in the end you're using somewhere in
Azure is defined a service principle, a user, let's just say it's like a user and that user
is the thing that you're using in Azure DevOps to connect to your Azure environment. And
I've seen a lot of cases where then they're like, Oh, it's really difficult to manage
that. So we're just going to give that, give that service principle global admin and there
we go. And then like what you said, like if you get your bill process hijacked, for example,
you're using like a task that you don't know is malicious, then that of course could misuse
that Azure credential and like hijack your systems. I think it's really important that
you then know if I'm going to use service connections, if I'm going to use a PowerShell
script, how much can I do? Like I want a minimal access.
And I thought that we, in the last few months, we saw a few of these, these kinds of cases
that a certain user, a developer is hacked and that the password and the multi-factor
token was stolen and then they are into a DevSecOps pipeline or whatever. And then
they running the software and then the software with malicious code in it is deployed to customers.
So that is, if a user, a developer has access to that managed instance that apprack with
a lot of rights in a certain environment, that could be potential big damage could be
there.
Yeah. Well then definitely in cases like this is of course, how do you say that? Like if,
for example, I was hijacked, right? And I have a lot, I can have administrator rights
somewhere, then that is bad, but hopefully at least my credentials are not used in the
automated build pipeline. That is hopefully exactly. You see something you shouldn't do.
You should definitely have like a separate entity or separate identity that connects
to your Azure environment automatically. Of course, if someone has my token, they could
probably go to the Azure portal. And if I have a lot of administrator, I can still delete
stuff or delete the code, but at least it could not really be automated that much anymore.
And then we're definitely going to have to talk about the, like exactly what you say,
phishing resistant MFA, pass keys, passwordless authentication and that kind of stuff. But
don't know how well that fits in the build process here.
No, no, no. Now we are looking a little bit broader than only the build pipeline, but
let's go back to the build process. What do we need to, we talked to some points. What
do we need to cover next? Because we have, we have a built and then I've tried to deploy
that built. What do we need to do in that, in that kind of steps? What do we need to
do to secure? What do we store the credentials that are using? Are we only using a service
connection or what is more within the build? So I think also pointing a little bit to the
last question, I think it's a great topic. And I think if you zoom out on that topic,
it's the whole reason why we are having this podcast, right? Because it's not in one process
where we can solve that kind of issue. So the whole DevSecOps, it's the idea is to build
that checks and controls in every stage and every phase. So for example, let's say we
have that connection and if we have the right permission, we can use that connection. And
we have, as you mentioned, our workflow instead of just somebody clicking it through the portal,
but code defined stored in our repository. And like we discussed last time, we have our
branch policies and repo protection in place. It minimizes the impact, the chance that somebody
can hijack our workflow to build steps into it without checks being done ahead. So I think
yes, there are possibilities, and I think there are a couple of big challenges when
it comes to authentication side of the build or even release, because you're always working
with service principles or user accounts or API keys, which have permanent permission
basis.
Exactly. Are you talking really a lot like minimizing the chances that your build process
gets corrupted, gets attacked?
Regarding on first time we talked about artifacts that we are not using anymore and we're still
using it, but not for infrastructure scope. If we look at nowadays, should we use DevOps,
GitHub, whatever, and why should we do that in a certain way? What are tools that we can
use to optimally use it in the build process and to see how we can move forward on that?
I got a couple of notes for this. One of the things, the tools, we're talking about toolings
and how to optimize the security for your build and your build pipeline. There's definitely
things that we also talked about in the last episode. We talked about vulnerability scanning.
We talked about container scanning. There's definitely still a mention here for code QL,
which is checking your code quality and stuff like that, maybe also vulnerabilities. You
want to use those tools and those tools are best used in a build environment. Before you
do a deployment to your production environment, there's going to be some vulnerability scanning
happening. They're like, hey, in this version of the SQL that you're using in the package,
we found a vulnerability and the deployment will be stopped.
Stuff like this is really how you can secure your code deployment process. That is definitely
something that with a DevSecOps approach, you want to automate. You don't want someone
to say, oh no, let's just take a look at the vulnerabilities. We'll look at the vulnerabilities
each sprint. No one actually does that. That needs to be automated.
We had a great session on this topic with David, of course, on the DefSecOps, on the
Defender for DevOps, sorry. How Microsoft is also helping to bring visibility into that
process of the build and release. For example, what you exactly said is you want to see which
vulnerabilities are coming in the code maybe, but also in the configuration-wise. Maybe
secret scanning, we know GitHub has it already or the advanced security has it in house.
With tools as Defender for DevOps, you can integrate, for example, with GitHub or with
Azure DevOps. Then you step in into the build process and into the PR process. Then we can
say, okay, if you create a PR to make certain changes, then this pipeline needs to run and
then do some checks before we allow it to be merged. If there are any findings like
there is a static password in your code or you are creating for... If we go back to
the infrastructure scope, you are doing storage account with weak encryption or with a VM
with a public IP address. You see that kind of configuration or at least which can end
up into being vulnerable already into that build process. That will be integrated to
Defender for DevOps as well, so in Defender for Cloud. What I really love about the Defender
for DevOps is it has a feature. They call it PR integration where the feedback goes
back to your PR. As a developer, you see, okay, I have a PR. Oh, on line 13, there was
a static password found. On line 15, I had an adjacent template for a storage account,
but I had a weak encryption model configured. As a developer, you don't need to escape your
environment for that build phase, but they can bring it back to you.
You're talking now about Defender to secure your code and the tools that you're using.
Defender is integrating in GitHub and Azure DevOps. Are they the only tools that we are
still using at the moment or are there any other tools that we are using to create a
build environment?
To create a build environment?
To create your build, to create your code.
I think there's definitely, I mentioned CodeCube a little bit. Things like SonarCube and SonarCloud
that will also do a lot of code scanning for you. That will also make sure that the thing
that you are going to deploy is secure or is at least secure enough according to the
policies that you've set up.
In the other end, the question is more or less, we are still using GitHub at the moment.
I don't know exactly, are you still using GitHub as well?
I use Azure DevOps and Azure DevOps Pipeline for work.
Both are within the Microsoft ecosystem, more or less general products that we are using.
Are there other tools like GitHub, like Azure DevOps that people in the wild are using to
create code?
Sure. To deploy their applications, maybe you mean.
For example?
Yeah, there's things like Jenkins, right? Which used to be GitLab, CircleCI, I believe
GitLab definitely. CircleCI is a really famous one.
GitLab is an on-prem environment.
I don't know, it's also served. It's also hosted for you.
It's hosted, but you can also build it by yourself.
GitHub as well, I believe. I think a part of it.
Not GitHub.
I know Azure DevOps in the past, you could have it on-prem.
Yeah, yeah, yeah.
DevOps Foundation.
Yeah, yeah, yeah, exactly.
But Jenkins is in the cloud service?
Not sure, actually.
Why is that?
I'm happy to not really use it that often.
For me, it's either. But if we are using tools like Defender or SonoCube or whatever to secure
your code, is it possible to have that on the other tools as well?
Because in my opinion, Defender is only available at the moment for GitHub and Azure DevOps.
So within the Microsoft ecosystem, it's okay. But if we extend that, and we want to extend
that to other developers, probably we can make a switch there as well.
Yeah, I think the good thing about GitHub and Azure DevOps and that kind of stuff is
that all of these things like vulnerability scanning, secret scanning, and hopefully in
the future things like automatic package updates, it's just all integrated into GitHub, into
Azure DevOps.
For other platforms like GitLab and what we talked about also, you can definitely still
install SonoCube, but it's more of a manual job. But I think, regardless of the platform
you're using, you should be using these kind of tools to ensure that what you're going
to deploy is secure and meets your expectations.
Yeah, and in the modern world, we're using GitHub, for example, or Azure DevOps. And
we try to, within the build process, we want to do everything within GitHub, like credentials,
like a service connection, like code scanning, everything.
Yeah, definitely use the tools and the options that your provider gives you. Don't start
building stuff yourself. There's nothing really wrong with building stuff yourself. Who's
going to maintain it? Who's going to maintain that PowerShell script that might be using
an old version of the Azure API? Just use Azure DevOps service connections and Microsoft
will host it for you.
Another thing I might want to bring up into this process thing to kind of move away from
the tooling. So as a developer, if I think about securing the build process, for me it's
really important to also make sure that what you are trying to build and trying to publish
is secure. There's a lot of articles online about people that accidentally also commit
and build their Git folder. So they actually deploy their.git folder to a website. And
then if you visit the website, you can just go to slash.git and steal all of the source
code.
So that's also definitely a thing that you, when you're going to build something, really
inspect what is the result of that build. And am I deploying, for example, stuff like
the Git ignore file that's really not necessary? Am I deploying some files that should actually
be in the Git ignore? Where is my Git folder in there? Because it really shouldn't be.
You want to make sure that your build is optimized.
So let's search the internet.
Yeah, there's tools for that. There's tools for that. You can just, you know, it will
just go through tons of websites on the internet and just go to slash.git and bam, you got
all the source code.
You have the whole Git history then from there.
Yeah, you have the entire, you have all the source code. And that's also the thing that
I always tell people, like sometimes they commit a secret into their repo and they're
like, oh no, I'm just going to put another commit on top of it to revert that. But I'm
like, no, if someone has access to your Git history, they can just go back to last month
and they'll find the original thing again.
The same is for PRs. So some people say, okay, I have multiple commits in my PR and that contains
a secret and then, but I merge it and I smash it to one. It's still in the history of your
PR, you can find it.
Just a bit of self-promotion. I have a blog post on my web page. No, but exactly. You
want to, but if you're using...
But how do you control that then?
So what you do is you, if you're using stuff like GitHub or if you're using stuff like
Azure DevOps, there's this feature called GitHub advanced security or GitHub advanced
security for Azure DevOps. And this contains tools that you can just click with one click.
You can enable it and then it will actually like every time you, when you will try to
push code to Azure DevOps or to GitHub, it will scan all of your code. And if there's
a secret in there, which it just learns machine learning, it's just smart. And then it will
just reject your push. And there's it was...
That's what GitHub has, right?
Yeah. But also Azure DevOps.
Yeah. That's pre-commit checking if there's a password.
On the server as well. On the server it checks it. So the server receives a request with
the new code and then before it accepts it, it scans it for secrets. And there's also
really, really interesting where GitHub does is if it, for example, there's services like
SendGrid, right? For email sending. And those API keys, like you said, they are just like
permanently valid, which is a big problem. If that leaks, you have a really big issue.
So a lot of companies are now doing that. The API key starts with like a prefix, like
for SendGrid, I think it's SG and then a colon. And then what GitHub does is it when it detects
a secret, starting with SG and colon, it will automatically send us a request to SendGrid
to invalidate that API key. Which is like, even if you leak the secret, it will be immediately
invalidated.
You can do that as well for your own projects. You can define your own keys, which it will
search in other repositories, I believe.
That is useful, but it is scary because like if the secret just doesn't meet that regular
expression or something, or doesn't meet the filters, then it will leak through. But yeah,
you definitely want to make sure that your code. And also what I do in my pull requests,
I have a pull request template set up. So every time one of my team members has a pull
request, I have like a little checklist that you can just create in GitHub and then should
develop. So like, did you add documentation? Does the project requirements need to be updated?
For example, if you add a connection to a new database, you need to update so people
can use that.
Also in there, it's a bit of a manual thing, but like, did you leak any secrets? And I
hope to make people think twice. Like first of all, I always tell people, don't put them
in your repo and all that kind of stuff. I try to teach them about it, but also make
it part of the PR flow that you actually have to visually check that checkbox.
It's a check mark that you need to check. And it's visible. So you see it, it's visible
and you think twice. That's what you realize with that one.
You definitely want to automate it, but it's also just good to teach people. Just don't,
you know, don't rely on tooling too much. Like then people are just not going to think
about it anymore. They're going to work in an environment where these tools are not in
place and then things are going to go unstuck.
In the normal user environment, we have security awareness to educate people with phishing,
with clicking on URLs and so on and so on. Within the Defender for DevOps, often in the
DevOps, DevSecOps world, we need to create awareness regarding these kinds of stuff.
So do you have a secret stored in your code or do you use other stuff that is malicious,
that you can do stuff that you don't want to know that it does?
I think awareness is one of the key aspects within a DevSecOps world.
Yeah, definitely. That's also the thing that we're trying to integrate, right? Like the
DevOps thing, it kind of exists. Lots of companies are doing it. It's so much better than doing
stuff manually or Lisa's computer is the production release machine. You know, that's just that
you don't want to work like that. You want an automated system and making it secure is
very important.
What I really love that I think I mentioned a couple of times on the PR integration of
the Defender for DevOps. I mean, it doesn't force the developer to come to a security
environment and the phish and phersa. But they talk the same on the same platform that
they need to intend. But if we take a look on the build, right? So on the build side,
there are credentials involved. And I think from our perspective, it's often about some
ARM templates, PowerShell coding and that kind of, but I think when you are building
an application, you have more requirement to that agent where you do the build, right?
So bring us to that. What, if you are doing a build of an application, what are the requirements
often that you are meet? Because I can imagine that organizations say, well, you can do this,
but it's an, it needs a certain version of Java or net or it needs a certain network
connectivity. Do you use like the GitHub runners and Azure DevOps runners or do you use your
own stuff?
Oh, we can talk about that. That's really interesting. One little thing that I wanted,
we talked kind of about this yesterday and you triggered me about something. So one thing
that I want to, I want to talk about really, really, it's really interesting is that there's
also now, instead of like building your own CI CD pipeline and having to think about what
version of Java you're using, are we using Ubuntu 20 or are we using Ubuntu 18? A lot
of people are still using Ubuntu 18. They shouldn't do that anymore. There's also these
approaches in Azure where you can just say, Hey, this is my GitHub link and this is a
react project. Go release it. And again, Azure will just figure it out for you. So there's
not even, you don't even need to think about it anymore. Just when there's a new commit
on main, when some code is finished, Azure will automatically detect it kind of in a
GitOps approach, just push it to the server. And then it's even more secure because you
don't even need to think about the builds anymore.
Yeah, we have the same with Sentinel for example, where it's a little bit different what they
are doing. Sentinel is a Microsoft SIEM solution. So it's where they bring all the alerts and
incidents together to create an overview, which we use often for our SOC and that kind
of services. But what they do is you have also repository integration. And if you go
and you say click and say, yeah, is it on DevOps or on GitHub? You say it's on GitHub,
just as authenticated GitHub. And what they do is they install an extension, which creates
a deployment workflow for you on the show. I think even there, there are multiple version
because it looks like that Sentinel is pulling that. But because you created that connection,
you installed an extension, which is doing...
Yeah, it's pushing it. Yeah, yeah, something like that. Yeah, yeah. But that just, it's
just another thing that could be automated and could be secured by Microsoft. But let's
go back to what you're like, let's go back a step to what you're saying about like as
a developer, what do I need? Because if we're thinking about deploying a bicep or deploying
ARM or deploying Terraform, we usually don't need that much, right? I think you, like I'm
just saying you guys probably need stuff like, you still need, your only thing you need is
probably like some Linux machine with like the...
PowerShell.
With PowerShell.
Azure PowerShell.
I feel like that is all you need. But for like a developer for me, so I'm like deploying
maybe an application with a backend that's written in.NET, a front-end written in React.
So I'm probably, I'm gonna use modern.NET, so I'm running on Linux. But I definitely
need a specific.NET version installed. I need a version of NPM installed, maybe yarn,
maybe a lot of project dependencies that it all needs to pull in. And yeah, there's a
lot more dependencies to build. Luckily with stuff like Docker, which we also kind of mentioned
this, some of this pain goes away because you can just let Docker do all of this. But
yeah, you definitely need more dependencies, which you also hope are secure.
So do you use, even when you need to have certain specific version of software tooling,
the public hosted, the managed versions of GitHub or Azure DevOps, or do you use your
own? Because there has been, I think definitely in the past, there has been a lot of discussions
like sharing and build agent with multiple organization. And there were some scenarios
that people could see things that they shouldn't be able to see.
Oh yeah, yeah, yeah. I remember it. So yeah, yeah, you're right. I always use that's a
true risk, right? If you are talking about build, seeing, being able. So if you're running
your program and I'm running my program on the same build agent and I find a way to escape
and then pull your code out there as well, we talked about the connection, right? To
take over the connection. But maybe instead of connection, we can step in and change the
code or pull some of the code away.
I think it's like, you know, there's this security thing where if you're securing your
house, the only thing you need to care about is that your house is more secure than your
neighbor because then the robbers will rob your neighbor and not you. And if I'm thinking
about like setting up my, either my own like a hosted agent where my code is built on,
like I have a VM in Azure somewhere that runs and builds all my code, I have the agent installed
on it or I'm just using the Microsoft agent. I prefer to use the cloud version. We at the
company I work in the beginning, we did use the, we just had our own VM. And then, I don't
know, there was one guy that always took care of it, right? He was a great guy. He was really
good at what he did. And at some point he was working with some other stuff. Four years
later we discovered that there were no updates. That's like, and that's the thing. Like if
you're working depends really on the company, on the company culture as well. If there's
actually security specialists in there, if there's ops people on there and they, and
you're building really important code that really needs to stay in your environment and
your people are specialized in keeping that stuff up to date.
Yeah. But if you are self hosting staff, then you need to take care of life cycle management.
Exactly. And update, not only updating, but also, also like from a security perspective
on day one, we have, I have zero days or you need to update on the second day. There's
an open port that is discovered because you need to connect to an agent on a specific
port and that's in that specific scenario needed. But two months later, that specific
scenario has changed. Yeah. You need to change that port, close it, open another one or whatever,
that you need to, you need to take care of all the stuff on prem. And if you're using
a cloud agent, for example, then yeah, Microsoft or someone else is taking care of all that
kind of stuff. And you have it on about your own house, secure it more than your neighbor
because otherwise you're robbed. But what if you are living in an apartment where you
have one house with 10 apartments and the main entrance, the main entrance should be
secured. And that's the people from the apartment building that are securing that entrance. So
in that case, it's the same for the cloud agent where I use the same agent as you do
with your own company and Microsoft or another vendor needs to take care of the security
on top of that agent that is running. But also if it's multi-threat, so I have multiple
organizations that are using the same agent and so on. So Microsoft is taking care of
that. Yeah. But the thing is for me that, you know, I'm not a security specialist. I
do like to think that I know a bit about security, otherwise I might not be sitting here. But
my job is to develop applications. It's to kind of, you know, come up with architecture
for applications. My job is not to secure stuff. So I really, I can't, I can't, I cannot
in good conscience use a self-hosted agent because I know that Microsoft will do a much
better job securing it. And it's, you know, this is, it's, if you're working at a company,
like what is your core thing that you're working on? Like a web shops core thing is building
a web shop. So should they also be building their own identity provider? No, they should
use a built one. Are you, are you in a very, are you working in like, I don't know, like
an airline or like at a bank, you might be using your own agents because it's really
important to you that you need to manage it yourself for normal companies. This is just
not their thing. So I really advocate for using the cloud version and Microsoft has
people working there 24 seven to keep that stuff secure and any zero days should be fixed
immediately and you don't even need to think about it.
From a life cycle perspective, I totally agree. But if you, let's, let's look in that apartment
house where Microsoft needs to close that, that, that the first door, if you are running
multi-threat agents and you are running on the same agent as I do, how do I, how do I
know that is secured, that I can access your code in case of, I think, I think that's more
of an implementation detail, if I'm honest. Like I know that there's probably like, let's
say there's 10 VMs, right? And you're thinking that you're working on the same VM. It's probably
some Docker host. It's probably some hyper V stuff. Maybe it's running on the same machine,
but the only thing you may be sharing is a kernel underwater. I'm not sure even how it
works. I don't need to think about it.
Well, I think it's, it's, they are changing. Definitely with GitHub, um, it hasn't been
a big shift on Azure DevOps. It was mainly on, uh, I hear myself very weird. Is it me
or no, no, no. Okay. Um, on Azure DevOps, it was always a hosted windows and the agent
just run that. Um, but when we went to the migration to GitHub, GitHub's all, uh, hosted,
uh, uh, containers. Um, and I believe Azure DevOps is also now using mainly because they
share that backend. I mean, I've been using hosted Linux agents on Azure DevOps for at
least six years, I think. But are there now really VMs or are there already changed to
a container? I believe.
I'm pretty sure you can run a Docker container with the Azure DevOps agent in it. You can
definitely do that. So that would already make it easier, right? You can just kind of
get the latest version of that agent and it will update all of the vulnerabilities. Again,
something you have to manage yourself.
Yeah. But if I want to deploy that to an Azure subscription or I deploy that, that specific
application on a certain environment, do I need to integrate that agent within my own
environment or is what are you?
That's a good question. I mean, one of the, where do you actually, where do you put it
in your infrastructure somewhere? Where do you put it in your landscape? One of the most,
um, like most often attacked environments, I think, or one of the most popular things
to attack in an, in a company is the build server. Cause sometimes people don't really
care about it. They don't keep it up to date. So if you can just hijack that, you can just
find all the source code there. You can find all the credentials there. So maybe you want
to sandbox it, I guess. I feel like you really do not want it to have access to all of your
own premise or all the other servers. At the other hand, if you're deploying to virtual
machines from the build agent, it has to have access to the VMs, right? Cause it needs to
be able to come to put the artifacts on there.
Yeah. So that's, I think, um, in my opinion, one of the biggest re, uh, wall, one of the
reasons why we see, uh, that people are, or customers ask for having hosted self hosted
agents. It's often, it has to do with networking because they have like networks, databases
or something in the on-prem environment or in a cloud environment, which is not the same
is you will get the same issue when, for example, if you go into cloud, right? If you go to,
let's say you have Azure web app, uh, where you have everything, Fina integrated and you
have Azure SQL as a backend. Um, so your Azure SQL is not publicly accessible anymore. And
then you are going to do a build, uh, you are going to push the changes to your, uh,
SQL database. That's not accessible anymore.
True. I had a little note about that because some time ago I read that GitHub now does
support this. So GitHub supports using a cloud hosted agent. So you don't need to think about
it anymore, but you can then talk to V net integrated stuff on Azure. I'm like 99% sure
I'm sure I'm right that I read about this. And then of course the first million comments
were when will this be released for Azure devils? So I just looked that up because I
believe, I believed it was completely impossible to use like a Microsoft cloud hosted agent
and deployed to V net in Azure. And it kind of is the thing that Microsoft says on their
documentation is that you just need to whitelist some IPs from that. And they change every
week. They say they could change every week. So every week they upload a new list and then
you need to upload those and whitelist those every week. But you can automate that. Yeah.
But that's not new. That's the same as whitelisting the whole Western Europe data center IP address
space. I mean, I had once a customer that said, yeah, give me the, I'm not going to
whitelist that. Give me, I'm not going to open a firewall. Give me the IP address that
I need to whitelist. You can download it. It's thousands and thousands of IP addresses
and the changes that they remove. They remove IP addresses and so on. So changes are. So
one of the, one of the other reasons we use ourselves, we use a self hosted containers
on Kubernetes. And one of the reasons why we did that is because through that model,
we have now the option to use the manage identity. You can do the same as well with the self
hosted. So in Azure DevOps, you have like two versions of VM agent, but the one that's
with the manage identity, I believe, I don't know. You can create like an Azure service
connection with a managed identity and that will be used by the cloud hosted agent. Yeah.
So you, what you do is you use the manage identity of the agent to authenticate to your
Azure subscription, for example, to do some activities. Yes, exactly. So that's also a
really awesome. Well, that's kind of the same thing you're doing, right? You just have a
container. Yeah. You have the kind of the same thing. I want to ask you a question about
this. So let's say that we need to, let's think in a modern way. So GitHub, if I'm correct,
has the option to use a cloud hosted agent, which you don't need to update, which you
don't need to maintain Microsoft or GitHub cares about all the security aspects of it,
which is great. I can sleep at night. And let's say that it's correct that they can
just talk to a V net integrated resources. So things that the internet cannot access,
let's say Azure DevOps gets this as well, which you then still want to use self hosted
agents. So for us it's double. Um, we do it partially for, so we use the build agents
partially to use the manage identity, for example, to configure and maintain our own solutions,
right? So our own platform and that kind of stuff, which are not internet facing. Um,
but we also use that manage identity to do other stuff, uh, other resources, for example,
without going too deep in it. So, um, how do I say it without saying too much about
this? That's the fun part. But what you can do is for example, if you're in a scenario
that you are, you're an MSP, right? You're not doing nothing with security. You're an
MSP and you are using Azure Lighthouse to manage your customers. Normally you need to
have a service principle which is going there. So you have always an epi registration with
a secret, which is vulnerable for that. So in that scenarios, you can also decide to
go to manage identities of your containerized solution. So when you are doing something
as infrastructure scope to your customer or you are doing as your lending zone configuration,
you can use those connections, right? Um, to do the proficienting of that. So in our
opinion, being able to using that manager identity, um, but it solves a lot of the problem
that you initially mentioned.
I don't really understand the difference though between using a, like I configure like a managed
identity in Azure for my Azure DevOps or GitHub action build, right? Build agents. The one
that's made by Microsoft in the cloud somewhere. Um, and I only use that one for pro deployment
permissions. It's not going to be able, it's not going to be allowed to like, I don't know,
change Azure settings, you know, that as little access as possible. You're doing kind of the
same thing. You're hosting your, uh, build agent yourself, but you're also talking about
the managed identity access. But how does that differ from what I'm saying with the
service connection and managed identity?
Um, service connection with the managed identity.
Yeah. You're saying like on your Kubernetes environment, you can like configure, uh, it
can be a VM as well. It can be a Docker, uh, container. Yeah. Your pod in Kubernetes has
a managed identity associated with it. And in my Azure DevOps cloud hosted agent also
has a managed identity because of the service connection. Yeah. But it's on the VM, right?
On the cloud hosted. Yeah. So, okay. So is your, your, your, you don't have on the cloud
hosted one. You don't have managed identity, right?
No, no, no. But the cloud hosted one uses a service connection to talk to Azure. Yeah.
And that service connection uses a managed identity.
That is not possible. All right. Pretty sure it is a federated connection. I mean, I'm
pretty sure it is possible. Okay. Let's just say service principle because the idea behind
managed identity is that the identity is not managed by you. Exactly. So you don't know
the secret of it. Exactly. And I, I, um, maybe we need to take this offline podcast, have
a look around. Maybe we can put it in the show notes or something like that. Something
like that. That would be awesome. In any case, what the thing that, um, let's say that the
security aspect would also be managed by Microsoft. Yeah. So the, the, the update parts would
be managed by Microsoft. I could integrate with V net. I could integrate with my private
resources. Would there still be benefits to using self hosted agents? If the security
aspect was handled by Microsoft in the cloud and if the V net integration was handled by
Microsoft. Um, well, um, depends on your level of security, I think. And the, the application
and how much risk you accept. For example, if you are doing some infrastructure code
and deploying some resources, it's way different than if you are building databases and application
and which are doing financial stuff or true. Yeah. That's a good point. So I think it's
an, there's always a risk with the public hosted by someone else. Um, from a modern
way I do accept it. Like if you're a small organization, you don't have the resources,
don't start with it. Right. I do completely because you need to manage it. You have lifecycle
management, you need to patch it, you need to monitor it. You need to have the fender
on it. You need to alert, react on it. So yeah, on that sense, I do agree with you guys
that that if you don't have the resources and the capacity to do it well, um, then don't
do it. But if you have like, but I think that's in always the case. If you have the resources,
the budgets and to make, if it makes fun, if it makes sense financially, if there's
like, if you did like your risk assessment and you're saying, well, this is a pretty
big risk for organization. It depends on the user. It depends. That's a great consultant
answer here. Definitely. I have one last thing. Um, we talked about to build process, the
built environment, but if we want to access from that development, a developer perspective,
if I want to access that, uh, that the built environment, the, to, to create code and so
on, you're doing that on your own laptop. I'm creating code on my own laptop is what
you're saying, right? Yeah. I'm using my local digital studio or rider, stuff like that.
How can, how can, and maybe we, we, we need to take another episode on that, but how does
dev box within Azure Azure dev box? How does that come into place regarding the things
that we're talking about? Because securing your build, uh, stuff, right. It's also securing
the entrance, how you access your build. Yeah. I think that's, I think, I think we talked
a bit about this in the first episode about workspace and management a bit. Yeah. I talked
to a colleague yesterday cause they were working for a client and I was like, so what kind
of environment are you using? He said, like, I'm using, I'm using the, uh, the Microsoft,
what was it called? Microsoft box in Microsoft or Azure dev box or something. And he said
that works really, really well, which we talked in the first episode about it, that I've heard
of people were using it or like stuff like GitHub code spaces, which is even more like
as a sauce platform. I think it is definitely very interesting. And we could maybe dedicate
another episode to this. Maybe I miss it. And maybe what that also our listeners, can
you maybe, uh, what is that? That's a great point. Yeah. Let's talk about that. The, the
dev box is, um, more or less a workplace within Azure that you can spin up a sort of an ACD
environment. Um, it's not AVD. Don't don't go to virtual desktop. Yeah. It's, it's not
an AVD environment, but it's a virtual workplace that you run in Azure. So you can connect
it remotely, but it's dedicated to a subscription or a, a, a, an environment that is used to
develop stuff. All the, all the software needed for program software development is already
installed on it. So instead of like getting a new laptop from your work and spending two
days configuring visual studio, official leader code, GitHub, get crack. It's already all
done for you. So you can just, you know, click on start or there's a new developer. It's
a VM still it's under the hood. It's a VM AVD probably as well. It's just AVD on steroids,
I guess. Yeah. It's not AVD, but it's, it's a VM where you can, yeah. But otherwise, I
hear people are complaining dedicated decision about definitely, but for interesting about
the dev box and that's, that's one of the nicest experiences that it's dedicated to
a subscription. And with that, from a cost perspective, that subscription belongs to
one project. So you have a project, you're running that project and what are the costs
of that for the prep, the running costs for that project. So you have a subscription,
all the costs are on that subscription. So you have a one project with that, with that
costs. So it helps also when you hire externally, external people, they can get access to the
dev box and exactly. And only if you are running a bill pipeline or whatever, it's only based
on that specific Azure subscription or multiple Azure subscription that you are attached to
the dev box. You can access other resources in your environment. So it's really specific.
But I think it's definitely, I definitely have some more questions or things to say
about this. I think it's definitely good to talk more about this later, but one little
question about this. So how the build process, how is this related to dev box?
Now if we, the thought that was in my head was if we are building a, an application and
so you're running a pipeline or whatever, we can secure that on an optimal way, but
accessing that bill pipeline, if you don't secure the access to your bill pipeline. And
we talked a little bit about in the first episode regarding a workplace, right? But
on the other hand, you can access a GitHub environment, you do authentication and multi-factor
and so on. But if you have your source code, source code within a official study, your
local machine. Yeah. So yeah, exactly.
That was the reason why it pops up in my head.
Exactly. You could, you could definitely secure and like put all the security measures in
place to make sure that your code is deployed securely onto your, well, the thing you're
your, your landing zone, wherever you're putting it. But yeah, then we kind of go back to the
question of how do you secure the access to the thing that is being built. And that could
be exactly something like dev box, like where at the other hand, I feel like you could,
you know, pry this laptop from my cold dead hands. Like don't, I love having a laptop,
but that you're completely right. If I left the company somewhere, I could still have
that code on my laptop. I could take that with me with dev box. Yeah. You could just,
you know, delete, disable my Azure account and you don't have, you don't need to format
my computer anymore. It doesn't need to collect dust for five years in a closet somewhere.
I think I know someone from, from the Microsoft team who is on, on that box. So we probably
can invite a special guest to have a specific scenario or a specific episode on that box.
Let's do that. Sure. It's really good. And regarding the build process environment and
stuff that we do, do we need to take another thing or did we miss something? Maybe, maybe
a simple thing that I sometimes see people do. I think we, we definitely talked a lot
about making sure that your build process is secure, making sure that you have like
a well fitting build environment, like either self hosted or like cloud hosted, making sure
that it fits. One little thing that I sometimes see people do is that they're kind of new
to this build process and they're like, Oh, I need a specific task. Right. And then they're
looking on the marketplace for like GitHub actions or Azure DevOps tasks. You really
want to make sure that the tasks you're using are okay tasks. Like there's even settings
in Azure DevOps, I believe to not allow marketplace tasks. If you could have everything secured,
but then you download a task that just copies all of the source code and sends it to Russia.
You could, so you definitely want to make sure that the tasks you're using are secure.
They are, they're trusted. And yeah, cause you don't know, you don't know the source
code of the task. You don't know what they're doing. So the things that are happening in
your build, you want to make sure that those are okay. Those are secure. That's a really
good one because I think that's happening more often than people. Yeah, exactly. I need
to replace some tokens. Oh, and I know that this one has a 600 downloads and the last
update was four years ago, but it does what it needs to do. So I'm just going to use that
one. Like, ah, so we spoke about the technology and I think the last one was great, but is
there also a human interaction that we can discuss if it comes to our bark, more role
based access requirements? Other, what do we expect from, um, our team when it comes
to, I think if you're like running a build pipeline, it's also definitely, you want to
make sure that it's only specific people can run specific pipelines where like deploying
your cloud resources in a really big corporate environment. Maybe only the, you know, the,
the ops people need to be able to deploy actual Azure production resources. There needs to
be the four I principle where if I want to change the connectivity, uh, stuff in Azure,
you know, all of the V net stuff, all the connections of Azure firewall, you want to
make sure that your colleague Steve, that is also really good and you know, works with
the, is specialized in this. He also needs to have a look at it. That's kind of, I guess
the human aspect. You want to automate your, automate your build process. That's the entire
thing about DevOps and kind of, but you want to also make sure that when specific and important
things are happening, that's, I think where you need a human to double check.
Yeah. But also a role based, you have a pull request. It need to be approved and so on
just before the belt is actually running for ice principle, maybe six eyes that you want
to cover. I think that's, that's important. And also what you mentioned a few minutes
earlier regarding the, the, the, the list that you need to approve the check mark that
you need to do before moving into PR. I think that are things that awareness we talked a
little bit about that, the awareness also within the DevSecOps, the DevOps people. Yeah.
That needs to be, that, that needs to be raised in my opinion.
Yeah. We want to make sure we want to know what we're deploying. Like the, I think that's,
I think that's a good way to say we want to know what we're deploying and we, the deployment
should have an automatically, that is what we're doing all of this for. We just want
to have code and we want it to run on production and we don't want to use Lisa's computer to
do that. We want to use a cloud hosted agent, a self hosted agent, just let it happen. But
what is being deployed? So you don't know that thing. It's like a whole, it's a whole
old joke about, you know, I don't know. I don't know. Introduce her to me. No, but I
don't, it's on the flight and the eight. Sure thing. So like one specific, one team
members laptop is always used to like, deploy to production and that's an old man, whatever
we all know that scenario with customers that there's all always one server that has enough
permission to do the deployment. Exactly. I have faced that personally with a, with a
really big enterprise customer when it came, every need, everything needed to be dev ops
and everything needed to be infrastructure scope. But then you needed to email the code
to somebody and then that person went to an SSH to an Ubuntu server because that server
had the right login already to the Azure site. So they run it manually then and they, oh,
nice. I can add one more thing to add one more thing on top of that. I used to do an
internship and I also like finished my studies at the Dutch tax services to the blessing
needs. And it was also, I worked at like a very modern department there where they built
a mobile app. So it was really fun. And they also started doing like deploy and build agency
ICD in there. So, you know, the code was built and quality was tested, like all the stuff
we talked about today. And then there was a step to go to a specific Bali, you know,
to go to a specific place in the company where you had to fill in an A4 paper. And then afterwards
the process would continue, which I just found really, really funny. It's all super automated
until you get to the deeper and deeper parts where you actually need to write a piece of
paper and go somewhere. And yeah, it's not, it's not, it's not particularly wrong. It's,
it's good that you have to sign off, I guess in a government environment. Yeah. Yeah. On
the other hand, we want to try to digitalize everything. And then if you need to sign a
paper, it could be good. It could be weird for some people, but on the other hand, if
you have, you need to take that step. So you edit for awareness perspective, I think is
really good. Yeah. Okay. I guess. Yeah. I mean, everything could of course be optimized
better, but you do definitely have a point, especially that's more to cover the change
management than I think. Right. I guess a little bit like who is, who is authorizing
the deployment, you know, that kind of stuff. Yeah. Because that's fun to me. Big important
thing, I think in our next step, because I believe the next one is going to be about
release then. I'm not really sure. We'll have to, we'll have to take a look. Yeah. Okay.
But because the change management is going to be a big one, I think, because when it
comes to build and deployment, we deploy more often than we did previously. Right. So maybe
in the past we did collect all the changes and then we did a deployment once a month
or something. But now we are doing multiple ones. So, um, and this is a really, uh, challenge
for this with a couple of customers of mine that say, okay, we need to be in control.
We need to see how much deployments we need to the production environment. Um, but I think
we'll take that for the next episode. Let's close this one. Yeah. We are almost an hour
or so. We make that. Um, let, let's look a little bit back. Um, we covered the build
pro the build process, the built environments. Um, we talked about a little bit dev box,
make a little bit, a little step outside. Um, we talked about securing code, uh, also
within this episode. So, uh, we, I think there are lots of aspects within the build that
comes into place. Um, and that's make the mix. This one of the interesting topics, the
steps within creating, uh, an application or building, um, an infrastructure, uh, you
need to take care of your build because a lot can go wrong within this step. Yeah, definitely.
So we talked about it, about the defender for DevOps, um, in a few episodes earlier
on the, uh, what was it? The defender for service, uh, of the, uh, and for series. So
if you want to have, you are interested in that one, uh, take a, have a look at that
episode. Um, on the other end, that was it for today built and that let's look into the
next one and probably release or something in that, uh, maybe at summit we'll see. Yeah,
let's see if we can connect with, uh, with, with some people over there and see if we
can create, uh, some other content, uh, on the other side of the ocean. Definitely. Let's
do that. So thanks for today. If you liked this, uh, this podcast series, um, like and
subscribe, um, and also, um, let us know if you want to know more about a specific scenario
or what else, uh, we will take care of that and see if we can create, um, an, an next
episode on that. So reach out to us on the socials and, uh, I see you next time. Thank
you guys. Thank you. Thank you.