One of the phrases I've heard on this podcast
very often is, data is the new oil. But let's
be honest, it's more like the new oxygen. Especially
with AI, because without data there is no AI.
And your business can't function without it.
NordLayer, they ensure that your data moves securely
between people, apps and offices through encrypted
tunnels. Encrypted tunnels and strict access
controls. and it keeps sensitive information
out of reach from prying eyes, but gives you
the full visibility over how your network is
used. Now, I've heard many stories on here where
a single exposed endpoint quickly turned into
a company -wide breach, but Nord layer, they
will make sure that that doesn't happen to you,
so if... If this interests you, please go to
nordlair .com slash tech talks daily. And if
you enter the special code tech daily dash 28,
you'll also get 28 % off. So please go check
that out. Let me know your thoughts, but right
now let's get today's guest on. What happens
when observability stops being a tool conversation
and starts becoming a business conversation?
Well today I'm joined by someone who has lived
that shift in the real world. His name's Bill
the Heinlein and he's the Field CTO at Chronosphere.
And before that he spent years inside highly
complex environments at United Airlines where
telemetry volume, cardinality and scale were
much more than just buzzwords. and throughout
his career he has seen first hand how a collect
everything mindset turns data lakes into data
landfills and digital clutter and digital hoarding
and how alert fatigue can drain great teams and
even a hero culture can quietly burn out your
best people. But he does believe observability
should be intentional, service -focused and tied
to SLOs from the very first sprint. So if your
CFO knows observerability for the cost line rather
than the value that you're delivering, he's got
some thoughts that you're going to want to hear
today. So you can expect practical examples,
a few scars and war stories from late night pager
duty, and most importantly, clear steps that
you can use tomorrow. So if you're ready to swap
noise for insights and turn your monitoring into
a momentum, you're going to love this one. But
enough from me. Let's get our guest onto the
podcast now. So thank you for joining me on the
podcast today. Can you tell everyone listening
a little about who you are and what you do? Sure.
I'm Bill Heinlein. I'm the field CTO at Chronosphere.
We're an observability platform. I have been
in the tech industry more time than I care to...
to say, but a little more than 25 years, a lot
of that time spent in the airline industry. And
most recently, before joining Chronosphere, I
was Director of Observability at United Airlines.
So a lot of large company experience, a lot of
battle wounds over the years. So I can go on
for a long time about observability and other
topics. So happy to be here. And that is exactly
why I invited you on the podcast to join me today
because every day we take a different subject,
demystify it, talk about it in a language everyone
can understand, make everyone realize we're all
battling with the same problems and how we can
overcome them as well. So from all the conversations
you've had in your current role, United Airlines,
everything throughout your career, what is that
number one observability challenge that organizations
are grappling with right now? What is it? Well,
I mean, cost is the easy answer, but, you know,
cost isn't the root cause, right? Cost happens
because, you know, over the years, we have all
subscribed in one way, shape or form to this
collect everything model, right? And it's filled
with a little bit of false hope and really creates
what I've started calling a data landfill. You
know, we wind up instead of a lake, you get a
landfill. A lot of data in there, not a lot of
value, hard to measure the value. And there's
a challenge with measuring the usage and the
value of the data that you're collecting. So
it's cost, but it's because there is this collect
everything strategy rather than a more curated
telemetry strategy. And things like logs are
some of the biggest offenders. It's really getting
down that telemetry to things that are collected
with intent and really bringing down the noise
to signal ratio as a result. And we've all been
on somewhat of a journey over the last few years
I find myself The older I get that everything
happens in cycles and we've everyone left on
-prem data centers. That was old school We're
all going to the cloud now these rumors of will
coming back again. So I'm curious How do the
traditional rules of observability? How do they
change in cloud native environments? I would
argue that observability has never been traditional.
It's always had to evolve. I think I don't I
think technology in general is hard to categorize
as traditional. If it is, then it's probably
something we talk about, you know. years ago
that nobody's doing anymore. But in short, we've
had to move from predictable telemetry volumes.
In the world of monolithic applications and they
lived on a few servers in a data center, they
admitted a predictable amount of telemetry. You
were monitoring a less complex infrastructure
and you understood when it needed to scale. You
built that in early on by predicting what your
scale was going to be, and then you knew what
to predict as far as the volume of telemetry
you were dealing with. And telemetry wasn't as
sophisticated back then, right? We were still
in the era of early APM. Some teams were still
just managing component monitoring, right? Is
the database up? Is the server running? Am I
out of disk space? Very, very basic. stuff that
nerds like me used to write a lot of scripts
in Perl for years about. But then it moved to
cloud and everything exploded, right? Because
suddenly we have servers that last for seconds
while they run and then die after they're done.
And we have scale that went from a few servers
to tens of thousands, depending on the size of
the environment and the scale at which you're
operating at the time. And the telemetry that's
all there is deeper and the cardinality just
explodes. And so it's really what's changed is
the volume of telemetry and those more traditional
APM tools, the more traditional and early observability
tools really struggle with keeping up with the
scale of that and doing it in a way that's cost
effective. I also think the rise in cloud costs
has served as somewhat of a virtual intervention
of sorts to face up to our data hoarding ways.
From what you've seen here, what issues do the
collect everything data collection practices
bring when the first step is gaining control
or attempting to gain control? Well, I think
there's certainly an opportunity for us to do
a show on your favorite binge watching network
about data hoarding, you know, similar to people
hoarding in a storage unit somewhere. But the
collect everything, we already mentioned cost,
obviously. But, you know, if you collect everything,
you're not curating. And the problem is, is that
I think we all got very excited about AI and
thought that AI could answer everything. But
it's still a classic garbage in and garbage out.
If you've got telemetry types coming from multiple
sources, different observability suppliers, maybe,
Lord knows a lot of different infrastructure,
whether it be cloud or on -prem, it's all very
different. It has varying levels of enrichment.
You throw in application logs that anybody can
write and that have varying levels of enrichment
and may be missing key things that don't matter
to the developer who wrote it. but may matter
to other people who look at it. So you've got
this blood of data here that's really hard to
correlate. And I always say that if you can't
look at a set of data on some paper and correlate
it together, AI isn't going to be able to do
that either. It needs something to chew on. And
that is where the collect everything starts to
bite you. Because you've got all this data that
you can't correlate easily. And when you're in
the midst of that 3 AM call that we all love
so much, that you wind up digging in and following
a metric that spikes suddenly. But oh, guess
what? It doesn't really. correlate to the problem
you're working on. And that, oh, by the way,
that looks like that happens every third Thursday
for some reason, because somebody's running a
job over here and it's unrelated, but you wasted
20 minutes. So it's that noise to signal ratio
that's killing your MTTR and you're spending
a lot of money. And your CFO probably knows what
observability is for the wrong reason. The cost
instead of the value. Yeah. 100%. And of course,
we also throw in operational burden and the dreaded
alert fatigue, also sources of burnout and frustration
for IT teams. But what real impact does this
have on businesses and what can IT leaders do
to address it? Something we've been talking about
for years, but any way forward here? Yeah, I
think and I mean, look, I had big teams. a team
of more than 500 at United. And so burnout was
something that I worried about every day. You
know, because you get great people and through
no fault of anybody's, those great people often
wind up being the people that that you build
into heroes. Yeah. And a hero culture kills your
best people. They burn out. And then the next
thing you know, they have they've secretly turned
on the open to work flag on LinkedIn. Right.
And then you've got opportunity cost problems
and all sorts of things. But the real impact
there is you've hired these engineers, whether
they be traditional engineers in the data center
or cloud engineers or software engineers, you've
hired these brilliant people to make money, to
build products that are going to displace your
competitors or your market. And this operational
burden that happens because on -call exists,
and you're not doing it efficiently, takes away
that innovation. And so all the capital investment
that you're making that your investors are happy
that you're making in all this technology to
disrupt the marketplace suddenly starts to look
more like some cost because you're not getting
anywhere with it. But at the same time, your
operating costs are on the rise because your
teams are spending all this time keeping the
engine running, so to speak. Sorry. back to airline
metaphors, but to drive or to keep things running.
Right. And so they're not focused on innovating
and that context switching that they have to
do constantly is a real thing, too. I wrote an
article and quoted an APA study not too long
ago that talks about context switching can have
as much as a 40 percent reduction in productivity
for an individual because of that. I mean, imagine,
right? You're in the zone, you're coding, hopefully
you're not vibe coding too much, and you're really
into it, you've got a great thing going, and
all of a sudden, I'll date myself, the pager
goes off, and then you've got to completely stop
what you're doing, and then focus over here on
code that likely somebody else wrote, but you've
got to figure out why it's broken, while you
listen to a group of people ask you, is it fixed
yet? Then you come down from that and then you
go back and you try to pick up where you left
off. The business impact, I mean, certainly is
burnout for that individual, but from a business
perspective, you're not innovating. You're getting
left in the dark. You're losing great people.
There's a lot of business impact that leaders
really have to consider and it's a real thing.
That's where observability can either hurt or
help you there. you collect everything, you're
probably on the pathway to hurt. You're in good
company there. I still get flashbacks from those
days of that pager going off, man. Don't get
me started there. There's a whole world of pain
opened up that I've still not dealt with, I think.
But seriously, moving beyond tools, what does
a strategic shift in today's IT infrastructure
management look like at a high level? And what
kind of conversations need to happen to kickstart
that change? Because it's very difficult to...
get all the stakeholders and get everyone in
a room and kickstart the changes required and
get everybody on board. But what needs to happen
here? I think it's just fundamental change at
the top, right? And by the top, I don't mean
necessarily the CIO or the CDO or the CTO. I
mean at the top level of the executives of the
organization, including those people. IT has
to deliver strategic value. In this world of
cloud costs and the cost of IT and the cost of
IT labor and all of that, you have to be able
to justify the value of your organization because
IT organizations that don't provide value, guess
what? They get outsourced. Those outsource teams
don't provide the great value that is usually
hoped for either. That's some cases they do.
I think what you have to do is really start with,
it's not so much how you manage just infrastructure.
I mean, I know that's the primary focus of your
conversations on this podcast, but I think it's
at a higher level of making sure that your IT
organization and specific, your IT leader has
a seat at the table and that they are participating
in those strategic discussions and advising rather
than order taking. Then it's easier to, number
one, justify, okay, we're spending a lot on Cloud.
Well, we made together this strategic discussion
or decision that we're going to invest in Cloud
because we need to scale, we need to be able
to support things in this region, whatever your
justifications are. But you've done that as a
collective group rather than going off and trying
to deliver what you think the business wants.
and then coming back and then being questioned
whether you're raising the value or not. And
in a world where all these systems are constantly
scaling and shifting and I'm curious, how can
teams ensure that they're collecting the right
data, not collecting everything, collecting the
right data, making that shift to make informed
decisions without being overwhelmed by the volume?
I mean, I know there's an argument that AI will
make that volume a little bit easier to digest,
but anything you're seeing here? I think I joke
sometimes that when people get so excited about
AI, I say it's the new gluten -free, right? We
put AI on a lot of things. At one point, I think
I saw that gluten -free was on a laundry detergent
and I scratched my head. I'm like, why does that
matter? But I think that we've got to go back
to the reality of collecting data with intent.
It goes back to business value. If I work with
a team in my past, they get wrapped around this
and it goes back to they'd want to measure this
component and that component instead of thinking
about the service they're offering. I'll step
back and say, tell me five reasons this feature
exists in your product. Why did the product owner
come to you and say, I want this to happen, code
this? Then what does good look like? And I promise
you, if you focus on traffic, error rate, latency,
and saturation, just as a starting point, your
leaps and bounds ahead of where you'd be if you
started to see, is that server up? Is it up now?
Was it up 10 minutes ago? And instead, looking
at, is my service at the level I hoped? And build
an SLO, spoiler alert, at the beginning, instead
of at the end. I've been just as guilty as anybody
else in this space. Five or six years ago, I
started asking myself, why do we wait until after
the release to think about an SLO? Why aren't
we thinking about that in the first sprint? And
then that informs metrics that you might need
to measure that SLO. It informs how you can build
an error budget. And then you get to a point
where you can look at things with value, and
you know what you specifically need to collect
rather than just throwing everything in a pot
and crossing your fingers. And as things grow
and as you're scaling, that becomes more and
more. It's got to be repeatable. It's got to
be frictionless for a developer. Observability
has to be a utility for people, not a burden.
It has to be a first -class citizen. And I think
observability is dealing with the sentiment in
most organizations that that cyber had to deal
with, you know, 10 years ago when we were really
doubling down in organizations on cyber because
it was becoming a real significant threat for
everybody. Right. Developers had to do it. But
it was another thing I got to do. And I got this
giant delivery cycle observability sort of in
that spot right now. We got to make it a first
class citizen, too. And that includes making
it frictionless and making it repeatable and
doing that by simplifying the approach of how
you start monitoring. And we've both jokingly
shared stories of our past and that page are
going off when we didn't want it to. But I think
for many teams, they've almost been conditioned
to operate with a reactive mindset after years
of firefighting, fixing P1s, et cetera. So what's
the best way to foster that cultural shift that's
required to move towards a much better and stress
-free, proactive, data -driven decision -making?
It almost sounds like a utopia, me saying this
out loud after years of doing the same. What
needs to happen here? I joke regularly that I
generally talk in euphoric sense, you know, like,
or utopic sense rather than, you know, reality.
But it is my opinions are based on reality and
repeated mistakes on my own part. But I think
for us to get people to shift, they have to understand
the power of data and the data that's available
to them. I look at a histogram. The power of
a histogram that shows you things like performance
or error rate from one minute to the next can
help you really think about ways to be more predictive.
If you're using SLOs, if you've decided you're
going to shift your mind and do an SLO first
and you do it in the sprint and lo and behold,
you go to production, you've got a good SLO and
you've got good error budget, even if it's really
simple, You're a step ahead and now you're saying
okay I don't have to react and wait till somebody
calls me because something breaks or maybe I'm
a step ahead and and my tools tell me something
broke before the customer but So many of the
things that I saw that were these big Newsworthy
events not just at my former employer, but it
in general right Started out as small paper cuts
hours, days before. And if there were little
data points to tell you, like, hey, you're starting
to burn your error budget a lot more than you
were. That's the easiest story to tell to get
people to think about how to go from it and get
away from, well, my app's up. Nobody's calling
about it, right? I can log in. It's a shift in
mentality and shift to get them to realize that
Look, you got woken up at three, but this started
at like 10 in the morning while you were having
coffee. Like, you know, what if, what if it interrupted
your coffee and started to sleep? Yeah. Oh, now
that's progress, but a hundred percent. I think.
Thanks to pesky, shiny emerging technologies
like, yes, AI, agentic AI or whatever, the next
big thing is we seem to have a lot of stalled
projects, things caught in pilot. So as a result,
leadership teams now are putting an extra special
caution on all tech projects. Everything's got
to be about ROI, measurable difference, quite
rightly too. So how can organizations effectively
measure the ROI of investing in a new observer?
strategy, especially when benefits aren't always
tangible at first or may appear that way. And
I suspect this is an area you're passionate about
too. I am. And it goes back to my line. If your
CFO knows what observability is because of the
cost rather than the value, you've got a problem.
To get you there, to get your organization there,
to realize that value, observability should underpin
your business objectives. Right. You don't just
decide. I mean, maybe if you're a nerd like me
and you see what the cool things are, then you
might think, yeah, absolutely. We should do observability.
And you may be not thinking about how it's tying
to objectives at a higher level. Right. It's
cool. I had a leader years ago tell me cool is
not a business case. It's not. But. If I can
tie observability back to, if I'm an e -commerce
company, I care about how quickly you can buy
widgets, I care about conversion rate, I care
about how quickly you go through the funnel,
right? And I care about things that interrupt
that funnel and make you abandon your shopping
cart. Observability helps you get there. If you
design observability for that application to
say, I care about response time and my services,
I care about how long it takes somebody to be
presented with the 40 widgets that I sell and
pay for it and all of that, then I'm underpinning
a business objective that is likely sell more
widgets or increase my conversion rate by X percent.
If observability underpins your business strategy,
it's easier for people to swallow. And then instead
of measuring with MTTR, which typically only
technology leaders care about, then you're selling
it with the idea of, hey, look, we drove better
performance, which you, Mr. and Mrs. Product
Owner care about because it's driving better
conversion rate and therefore better revenue
based on what your product is doing. To bring
everything that we've talked about here today
back to your work at Chronosphere, are you able
to share maybe a real world example of how a
strategic observability plan has actually helped
an organization move from that reactive firefighting
model to be more proactive and innovative one
and get that over the line? Anything you can
share on that? You don't have to mention any
names. I think it'd be useful. Well, I'll tell
you, in my former life, we were early embracers
of APM, right? And we were in a state where there
were still a few folks that were saying, you
know, nobody's calling, you know, nobody's calling
the service desk, nobody is, you know, there's
no problem, I can log in kind of a thing. They
were still in that mode with some of the teams.
Getting people to understand the value of, again,
that histogram, getting people to be able to
understand that A lot of the problems that keep
us on phone calls for hours in the middle of
the night started with a paper cut at seven in
the morning, and maybe seven in the morning on
Wednesday, and it's Sunday morning. Changing
that perspective and showing that this data is
possible together, it's not difficult together,
and then when it's there, It gives you a lot
of great things. And one of the best ways is
to help people. I call it mean time to innocence.
Right. Hey, look, I know it's not my problem
and I know it's not because here's data. Right.
Here's my service. It's calling this. And oh,
guess what? It's your service because I'm calling
you. And every other third time I call you, it
fails. You're killing me. Right. So we did that.
We did that. And I was proud to say that some
of the earliest adopters of APM in our organization
were the NetOps team members, because they got
tired of being blamed for it being the network,
because it's always the network, right? That
was always a running joke in every organization,
I'm sure. But these guys and gals would get on,
look at the tools and say, it's not the network.
And as a matter of fact, it's this service right
here that keeps failing. And it started right
when people started calling. So it's, it's, there
are a lot of, I can probably think of hours of
examples of things like that, both in my current
or with my former employer. And as I talk to
customers in my current role. And just listening
to that story reminds me people were so fiercely
defensive back in the day on things like that
with, it almost felt like the Spider -Man meme
with the network guy pointing at the apps guy
and the apps guy pointing at the, you know, everyone's
pointing at each other. But if an IT is listening
to a conversation today, maybe they want to take
one actionable step tomorrow too. begin to improve
their observability posture, what would you recommend
they do? I appreciate it's not as simple as that,
but if they want to take that first baby step,
what should they do? One of my favorite questions
when I walk into a customer as we're talking
to them is, why did you buy observability and
how is it supporting your business goal? Yeah.
I mean, and if the answer is, I don't know, or
they stumble for it, then they probably just
need to take a step back and think about it.
So many people jumped into observability because
they needed something, and it was an evolving
technology and evolving tool. APM promised some
really amazing things easily, and as the cloud
changed things, that got more difficult in it.
diminished a little bit and people had to work
harder to get it. And so it's even more important
to have that strategy. And that strategy starts
with, why am I doing this? I'm doing this so
I can sell more widgets. I know I'm selling more
widgets because I'm driving better conversion.
I'm driving better conversion because my performance
is better. And I know that if I change my performance
by X number of milliseconds, I drive my conversion
up by X percentage. I think it's really sitting
back and asking yourself, why am I doing this?
Because somebody is going to ask you. Don't wait
till your CFO asks you. And I think that is a
powerful moment to end on today. But anyone we've
inspired today, maybe they want to take that
first baby step into improving that observability
posture. Where can they find you or your team
and more information on anything we talked about
today? Where should they go? Certainly, you can
find me on LinkedIn, and I'm always, as you can
tell, happy and passionate to talk about observability.
And part of my role as the CTO at Chronosphere
is not to sell, but to guide and help folks with
how to do things like an observability strategy.
So you can also find me there. We're at chronosphere
.io. And again, we're a cloud -native observability
company focused on giving you control of your
data. rather than drowning in it and not understanding
what your telemetry is doing for you. Love that.
So I will add links to absolutely everything.
Make it easy for people to find. Thank you. We
covered so much today from observability challenges
that organizations are grappling with, collecting
the right data to make informed decisions. As
an XIT guy measuring the ROI of investing in
a new observability strategy, the old belts and
braces approach of you can only improve what
you measure. So great to hear. Right. Just thank
you for shining a light on this today. Thank
you. Thanks for the time. Now, before we wrap,
what will you change this week to collect with
intent rather than just collecting everything?
I think Bill left us with a simple challenge
that almost doubles as a compass. Start with
why your product exists. Define what good looks
like. Set those SLOs early and let your choices
guide the data that you're gathering. And this
is how observability can earn a seat at the strategy
table and how teams reclaim time for building,
not just wasting their days firefighting. So
if this sparked an idea, please connect with
Bill on LinkedIn, check out their website. And
if you've got anything you'd like to share with
me, techtalksnetwork .com, please let me know.
But that's it for today. So thank you for listening
as always and I'll speak with you all again tomorrow