WEBVTT 00:00:00.000 --> 00:00:02.359 One of the phrases I've heard on this podcast 00:00:02.359 --> 00:00:06.900 very often is, data is the new oil. But let's 00:00:06.900 --> 00:00:09.880 be honest, it's more like the new oxygen. Especially 00:00:09.880 --> 00:00:12.820 with AI, because without data there is no AI. 00:00:13.339 --> 00:00:15.960 And your business can't function without it. 00:00:16.839 --> 00:00:20.059 NordLayer, they ensure that your data moves securely 00:00:20.059 --> 00:00:24.500 between people, apps and offices through encrypted 00:00:24.500 --> 00:00:28.399 tunnels. Encrypted tunnels and strict access 00:00:28.399 --> 00:00:31.679 controls. and it keeps sensitive information 00:00:31.679 --> 00:00:35.520 out of reach from prying eyes, but gives you 00:00:35.520 --> 00:00:37.979 the full visibility over how your network is 00:00:37.979 --> 00:00:41.460 used. Now, I've heard many stories on here where 00:00:41.460 --> 00:00:44.780 a single exposed endpoint quickly turned into 00:00:44.780 --> 00:00:48.439 a company -wide breach, but Nord layer, they 00:00:48.439 --> 00:00:50.539 will make sure that that doesn't happen to you, 00:00:50.719 --> 00:00:53.640 so if... If this interests you, please go to 00:00:53.640 --> 00:00:57.420 nordlair .com slash tech talks daily. And if 00:00:57.420 --> 00:01:00.939 you enter the special code tech daily dash 28, 00:01:01.359 --> 00:01:05.159 you'll also get 28 % off. So please go check 00:01:05.159 --> 00:01:07.700 that out. Let me know your thoughts, but right 00:01:07.700 --> 00:01:12.579 now let's get today's guest on. What happens 00:01:12.579 --> 00:01:15.939 when observability stops being a tool conversation 00:01:15.939 --> 00:01:18.859 and starts becoming a business conversation? 00:01:19.260 --> 00:01:21.879 Well today I'm joined by someone who has lived 00:01:21.879 --> 00:01:24.519 that shift in the real world. His name's Bill 00:01:24.519 --> 00:01:28.480 the Heinlein and he's the Field CTO at Chronosphere. 00:01:29.620 --> 00:01:32.500 And before that he spent years inside highly 00:01:32.500 --> 00:01:36.340 complex environments at United Airlines where 00:01:36.340 --> 00:01:39.900 telemetry volume, cardinality and scale were 00:01:39.900 --> 00:01:42.260 much more than just buzzwords. and throughout 00:01:42.260 --> 00:01:45.480 his career he has seen first hand how a collect 00:01:45.480 --> 00:01:48.799 everything mindset turns data lakes into data 00:01:48.799 --> 00:01:52.140 landfills and digital clutter and digital hoarding 00:01:52.140 --> 00:01:55.980 and how alert fatigue can drain great teams and 00:01:55.980 --> 00:01:59.359 even a hero culture can quietly burn out your 00:01:59.359 --> 00:02:03.500 best people. But he does believe observability 00:02:03.500 --> 00:02:06.680 should be intentional, service -focused and tied 00:02:06.680 --> 00:02:11.300 to SLOs from the very first sprint. So if your 00:02:11.300 --> 00:02:14.419 CFO knows observerability for the cost line rather 00:02:14.419 --> 00:02:17.340 than the value that you're delivering, he's got 00:02:17.340 --> 00:02:19.039 some thoughts that you're going to want to hear 00:02:19.039 --> 00:02:21.960 today. So you can expect practical examples, 00:02:22.159 --> 00:02:25.360 a few scars and war stories from late night pager 00:02:25.360 --> 00:02:28.599 duty, and most importantly, clear steps that 00:02:28.599 --> 00:02:31.439 you can use tomorrow. So if you're ready to swap 00:02:31.439 --> 00:02:35.080 noise for insights and turn your monitoring into 00:02:35.080 --> 00:02:38.430 a momentum, you're going to love this one. But 00:02:38.430 --> 00:02:40.710 enough from me. Let's get our guest onto the 00:02:40.710 --> 00:02:44.930 podcast now. So thank you for joining me on the 00:02:44.930 --> 00:02:47.250 podcast today. Can you tell everyone listening 00:02:47.250 --> 00:02:51.069 a little about who you are and what you do? Sure. 00:02:51.509 --> 00:02:55.129 I'm Bill Heinlein. I'm the field CTO at Chronosphere. 00:02:55.289 --> 00:02:59.289 We're an observability platform. I have been 00:02:59.289 --> 00:03:03.349 in the tech industry more time than I care to... 00:03:03.439 --> 00:03:06.400 to say, but a little more than 25 years, a lot 00:03:06.400 --> 00:03:10.120 of that time spent in the airline industry. And 00:03:10.120 --> 00:03:13.580 most recently, before joining Chronosphere, I 00:03:13.580 --> 00:03:17.780 was Director of Observability at United Airlines. 00:03:18.039 --> 00:03:23.300 So a lot of large company experience, a lot of 00:03:23.300 --> 00:03:27.840 battle wounds over the years. So I can go on 00:03:27.840 --> 00:03:30.680 for a long time about observability and other 00:03:30.680 --> 00:03:34.110 topics. So happy to be here. And that is exactly 00:03:34.110 --> 00:03:36.870 why I invited you on the podcast to join me today 00:03:36.870 --> 00:03:39.189 because every day we take a different subject, 00:03:39.650 --> 00:03:41.849 demystify it, talk about it in a language everyone 00:03:41.849 --> 00:03:44.370 can understand, make everyone realize we're all 00:03:44.370 --> 00:03:46.370 battling with the same problems and how we can 00:03:46.370 --> 00:03:49.569 overcome them as well. So from all the conversations 00:03:49.569 --> 00:03:51.789 you've had in your current role, United Airlines, 00:03:51.909 --> 00:03:53.909 everything throughout your career, what is that 00:03:53.909 --> 00:03:57.509 number one observability challenge that organizations 00:03:57.509 --> 00:04:00.770 are grappling with right now? What is it? Well, 00:04:00.930 --> 00:04:04.969 I mean, cost is the easy answer, but, you know, 00:04:05.169 --> 00:04:08.849 cost isn't the root cause, right? Cost happens 00:04:08.849 --> 00:04:13.110 because, you know, over the years, we have all 00:04:13.110 --> 00:04:15.550 subscribed in one way, shape or form to this 00:04:15.550 --> 00:04:19.329 collect everything model, right? And it's filled 00:04:19.329 --> 00:04:22.829 with a little bit of false hope and really creates 00:04:22.829 --> 00:04:25.189 what I've started calling a data landfill. You 00:04:25.189 --> 00:04:27.949 know, we wind up instead of a lake, you get a 00:04:27.949 --> 00:04:31.060 landfill. A lot of data in there, not a lot of 00:04:31.060 --> 00:04:35.660 value, hard to measure the value. And there's 00:04:35.660 --> 00:04:38.180 a challenge with measuring the usage and the 00:04:38.180 --> 00:04:41.579 value of the data that you're collecting. So 00:04:41.579 --> 00:04:46.180 it's cost, but it's because there is this collect 00:04:46.180 --> 00:04:48.860 everything strategy rather than a more curated 00:04:48.860 --> 00:04:52.759 telemetry strategy. And things like logs are 00:04:52.759 --> 00:04:57.199 some of the biggest offenders. It's really getting 00:04:57.199 --> 00:05:01.839 down that telemetry to things that are collected 00:05:01.839 --> 00:05:05.579 with intent and really bringing down the noise 00:05:05.579 --> 00:05:09.620 to signal ratio as a result. And we've all been 00:05:09.620 --> 00:05:12.139 on somewhat of a journey over the last few years 00:05:12.139 --> 00:05:15.519 I find myself The older I get that everything 00:05:15.519 --> 00:05:18.959 happens in cycles and we've everyone left on 00:05:18.959 --> 00:05:20.819 -prem data centers. That was old school We're 00:05:20.819 --> 00:05:22.759 all going to the cloud now these rumors of will 00:05:22.759 --> 00:05:26.060 coming back again. So I'm curious How do the 00:05:26.060 --> 00:05:28.560 traditional rules of observability? How do they 00:05:28.560 --> 00:05:31.899 change in cloud native environments? I would 00:05:31.899 --> 00:05:34.399 argue that observability has never been traditional. 00:05:34.399 --> 00:05:38.529 It's always had to evolve. I think I don't I 00:05:38.529 --> 00:05:41.170 think technology in general is hard to categorize 00:05:41.170 --> 00:05:43.689 as traditional. If it is, then it's probably 00:05:43.689 --> 00:05:47.449 something we talk about, you know. years ago 00:05:47.449 --> 00:05:51.850 that nobody's doing anymore. But in short, we've 00:05:51.850 --> 00:05:55.350 had to move from predictable telemetry volumes. 00:05:55.930 --> 00:06:00.550 In the world of monolithic applications and they 00:06:00.550 --> 00:06:02.910 lived on a few servers in a data center, they 00:06:02.910 --> 00:06:05.589 admitted a predictable amount of telemetry. You 00:06:05.589 --> 00:06:09.689 were monitoring a less complex infrastructure 00:06:09.689 --> 00:06:14.829 and you understood when it needed to scale. You 00:06:14.829 --> 00:06:17.430 built that in early on by predicting what your 00:06:17.430 --> 00:06:20.569 scale was going to be, and then you knew what 00:06:20.569 --> 00:06:23.730 to predict as far as the volume of telemetry 00:06:23.730 --> 00:06:25.810 you were dealing with. And telemetry wasn't as 00:06:25.810 --> 00:06:27.670 sophisticated back then, right? We were still 00:06:27.670 --> 00:06:32.589 in the era of early APM. Some teams were still 00:06:32.589 --> 00:06:37.350 just managing component monitoring, right? Is 00:06:37.350 --> 00:06:39.569 the database up? Is the server running? Am I 00:06:39.569 --> 00:06:42.930 out of disk space? Very, very basic. stuff that 00:06:42.930 --> 00:06:44.870 nerds like me used to write a lot of scripts 00:06:44.870 --> 00:06:48.370 in Perl for years about. But then it moved to 00:06:48.370 --> 00:06:51.170 cloud and everything exploded, right? Because 00:06:51.170 --> 00:06:54.389 suddenly we have servers that last for seconds 00:06:54.389 --> 00:06:56.569 while they run and then die after they're done. 00:06:56.730 --> 00:07:00.889 And we have scale that went from a few servers 00:07:00.889 --> 00:07:04.310 to tens of thousands, depending on the size of 00:07:04.310 --> 00:07:06.750 the environment and the scale at which you're 00:07:06.750 --> 00:07:10.259 operating at the time. And the telemetry that's 00:07:10.259 --> 00:07:14.879 all there is deeper and the cardinality just 00:07:14.879 --> 00:07:18.160 explodes. And so it's really what's changed is 00:07:18.160 --> 00:07:23.939 the volume of telemetry and those more traditional 00:07:23.939 --> 00:07:27.519 APM tools, the more traditional and early observability 00:07:27.519 --> 00:07:30.819 tools really struggle with keeping up with the 00:07:30.819 --> 00:07:33.399 scale of that and doing it in a way that's cost 00:07:33.399 --> 00:07:38.100 effective. I also think the rise in cloud costs 00:07:38.100 --> 00:07:40.560 has served as somewhat of a virtual intervention 00:07:40.560 --> 00:07:44.259 of sorts to face up to our data hoarding ways. 00:07:44.759 --> 00:07:47.680 From what you've seen here, what issues do the 00:07:47.680 --> 00:07:50.120 collect everything data collection practices 00:07:50.120 --> 00:07:53.759 bring when the first step is gaining control 00:07:53.759 --> 00:07:57.060 or attempting to gain control? Well, I think 00:07:57.060 --> 00:07:59.920 there's certainly an opportunity for us to do 00:07:59.920 --> 00:08:04.560 a show on your favorite binge watching network 00:08:04.560 --> 00:08:08.199 about data hoarding, you know, similar to people 00:08:08.199 --> 00:08:13.579 hoarding in a storage unit somewhere. But the 00:08:13.579 --> 00:08:15.319 collect everything, we already mentioned cost, 00:08:15.360 --> 00:08:18.459 obviously. But, you know, if you collect everything, 00:08:18.660 --> 00:08:21.579 you're not curating. And the problem is, is that 00:08:21.579 --> 00:08:25.439 I think we all got very excited about AI and 00:08:25.439 --> 00:08:28.139 thought that AI could answer everything. But 00:08:28.139 --> 00:08:30.639 it's still a classic garbage in and garbage out. 00:08:30.730 --> 00:08:34.870 If you've got telemetry types coming from multiple 00:08:34.870 --> 00:08:40.389 sources, different observability suppliers, maybe, 00:08:41.490 --> 00:08:43.669 Lord knows a lot of different infrastructure, 00:08:43.750 --> 00:08:46.870 whether it be cloud or on -prem, it's all very 00:08:46.870 --> 00:08:49.429 different. It has varying levels of enrichment. 00:08:49.730 --> 00:08:52.549 You throw in application logs that anybody can 00:08:52.549 --> 00:08:54.929 write and that have varying levels of enrichment 00:08:54.929 --> 00:08:57.830 and may be missing key things that don't matter 00:08:57.830 --> 00:09:00.470 to the developer who wrote it. but may matter 00:09:00.470 --> 00:09:03.169 to other people who look at it. So you've got 00:09:03.169 --> 00:09:08.230 this blood of data here that's really hard to 00:09:08.230 --> 00:09:13.169 correlate. And I always say that if you can't 00:09:13.169 --> 00:09:17.009 look at a set of data on some paper and correlate 00:09:17.009 --> 00:09:20.230 it together, AI isn't going to be able to do 00:09:20.230 --> 00:09:23.720 that either. It needs something to chew on. And 00:09:23.720 --> 00:09:26.960 that is where the collect everything starts to 00:09:26.960 --> 00:09:29.379 bite you. Because you've got all this data that 00:09:29.379 --> 00:09:32.919 you can't correlate easily. And when you're in 00:09:32.919 --> 00:09:36.600 the midst of that 3 AM call that we all love 00:09:36.600 --> 00:09:42.080 so much, that you wind up digging in and following 00:09:42.080 --> 00:09:44.840 a metric that spikes suddenly. But oh, guess 00:09:44.840 --> 00:09:47.820 what? It doesn't really. correlate to the problem 00:09:47.820 --> 00:09:49.580 you're working on. And that, oh, by the way, 00:09:49.659 --> 00:09:51.960 that looks like that happens every third Thursday 00:09:51.960 --> 00:09:54.419 for some reason, because somebody's running a 00:09:54.419 --> 00:09:56.299 job over here and it's unrelated, but you wasted 00:09:56.299 --> 00:09:59.620 20 minutes. So it's that noise to signal ratio 00:09:59.620 --> 00:10:02.519 that's killing your MTTR and you're spending 00:10:02.519 --> 00:10:05.320 a lot of money. And your CFO probably knows what 00:10:05.320 --> 00:10:08.620 observability is for the wrong reason. The cost 00:10:08.620 --> 00:10:12.289 instead of the value. Yeah. 100%. And of course, 00:10:12.409 --> 00:10:16.090 we also throw in operational burden and the dreaded 00:10:16.090 --> 00:10:20.529 alert fatigue, also sources of burnout and frustration 00:10:20.529 --> 00:10:23.789 for IT teams. But what real impact does this 00:10:23.789 --> 00:10:26.570 have on businesses and what can IT leaders do 00:10:26.570 --> 00:10:29.110 to address it? Something we've been talking about 00:10:29.110 --> 00:10:33.470 for years, but any way forward here? Yeah, I 00:10:33.470 --> 00:10:38.720 think and I mean, look, I had big teams. a team 00:10:38.720 --> 00:10:41.279 of more than 500 at United. And so burnout was 00:10:41.279 --> 00:10:44.519 something that I worried about every day. You 00:10:44.519 --> 00:10:48.320 know, because you get great people and through 00:10:48.320 --> 00:10:50.899 no fault of anybody's, those great people often 00:10:50.899 --> 00:10:53.000 wind up being the people that that you build 00:10:53.000 --> 00:10:57.039 into heroes. Yeah. And a hero culture kills your 00:10:57.039 --> 00:11:00.480 best people. They burn out. And then the next 00:11:00.480 --> 00:11:02.840 thing you know, they have they've secretly turned 00:11:02.840 --> 00:11:05.679 on the open to work flag on LinkedIn. Right. 00:11:05.679 --> 00:11:09.480 And then you've got opportunity cost problems 00:11:09.480 --> 00:11:11.840 and all sorts of things. But the real impact 00:11:11.840 --> 00:11:15.299 there is you've hired these engineers, whether 00:11:15.299 --> 00:11:18.299 they be traditional engineers in the data center 00:11:18.299 --> 00:11:21.120 or cloud engineers or software engineers, you've 00:11:21.120 --> 00:11:25.360 hired these brilliant people to make money, to 00:11:25.360 --> 00:11:27.700 build products that are going to displace your 00:11:27.700 --> 00:11:32.159 competitors or your market. And this operational 00:11:32.159 --> 00:11:35.659 burden that happens because on -call exists, 00:11:36.250 --> 00:11:39.169 and you're not doing it efficiently, takes away 00:11:39.169 --> 00:11:42.370 that innovation. And so all the capital investment 00:11:42.370 --> 00:11:44.409 that you're making that your investors are happy 00:11:44.409 --> 00:11:47.230 that you're making in all this technology to 00:11:47.230 --> 00:11:49.789 disrupt the marketplace suddenly starts to look 00:11:49.789 --> 00:11:51.570 more like some cost because you're not getting 00:11:51.570 --> 00:11:54.669 anywhere with it. But at the same time, your 00:11:54.669 --> 00:11:57.330 operating costs are on the rise because your 00:11:57.330 --> 00:11:59.610 teams are spending all this time keeping the 00:11:59.610 --> 00:12:03.000 engine running, so to speak. Sorry. back to airline 00:12:03.000 --> 00:12:08.019 metaphors, but to drive or to keep things running. 00:12:08.200 --> 00:12:10.320 Right. And so they're not focused on innovating 00:12:10.320 --> 00:12:12.539 and that context switching that they have to 00:12:12.539 --> 00:12:15.980 do constantly is a real thing, too. I wrote an 00:12:15.980 --> 00:12:18.700 article and quoted an APA study not too long 00:12:18.700 --> 00:12:22.919 ago that talks about context switching can have 00:12:22.919 --> 00:12:27.179 as much as a 40 percent reduction in productivity 00:12:27.179 --> 00:12:30.139 for an individual because of that. I mean, imagine, 00:12:30.200 --> 00:12:33.159 right? You're in the zone, you're coding, hopefully 00:12:33.159 --> 00:12:36.580 you're not vibe coding too much, and you're really 00:12:36.580 --> 00:12:38.200 into it, you've got a great thing going, and 00:12:38.200 --> 00:12:41.019 all of a sudden, I'll date myself, the pager 00:12:41.019 --> 00:12:43.759 goes off, and then you've got to completely stop 00:12:43.759 --> 00:12:46.659 what you're doing, and then focus over here on 00:12:46.659 --> 00:12:48.820 code that likely somebody else wrote, but you've 00:12:48.820 --> 00:12:50.919 got to figure out why it's broken, while you 00:12:50.919 --> 00:12:53.759 listen to a group of people ask you, is it fixed 00:12:53.759 --> 00:12:57.070 yet? Then you come down from that and then you 00:12:57.070 --> 00:12:58.889 go back and you try to pick up where you left 00:12:58.889 --> 00:13:02.710 off. The business impact, I mean, certainly is 00:13:02.710 --> 00:13:06.370 burnout for that individual, but from a business 00:13:06.370 --> 00:13:09.750 perspective, you're not innovating. You're getting 00:13:09.750 --> 00:13:12.789 left in the dark. You're losing great people. 00:13:12.990 --> 00:13:15.409 There's a lot of business impact that leaders 00:13:15.409 --> 00:13:18.190 really have to consider and it's a real thing. 00:13:18.370 --> 00:13:21.870 That's where observability can either hurt or 00:13:21.870 --> 00:13:24.929 help you there. you collect everything, you're 00:13:24.929 --> 00:13:29.690 probably on the pathway to hurt. You're in good 00:13:29.690 --> 00:13:32.070 company there. I still get flashbacks from those 00:13:32.070 --> 00:13:34.909 days of that pager going off, man. Don't get 00:13:34.909 --> 00:13:36.549 me started there. There's a whole world of pain 00:13:36.549 --> 00:13:38.470 opened up that I've still not dealt with, I think. 00:13:39.269 --> 00:13:42.750 But seriously, moving beyond tools, what does 00:13:42.750 --> 00:13:45.950 a strategic shift in today's IT infrastructure 00:13:45.950 --> 00:13:48.769 management look like at a high level? And what 00:13:48.769 --> 00:13:51.710 kind of conversations need to happen to kickstart 00:13:51.710 --> 00:13:54.450 that change? Because it's very difficult to... 00:13:54.519 --> 00:13:56.559 get all the stakeholders and get everyone in 00:13:56.559 --> 00:13:58.639 a room and kickstart the changes required and 00:13:58.639 --> 00:14:00.539 get everybody on board. But what needs to happen 00:14:00.539 --> 00:14:03.740 here? I think it's just fundamental change at 00:14:03.740 --> 00:14:06.200 the top, right? And by the top, I don't mean 00:14:06.200 --> 00:14:09.659 necessarily the CIO or the CDO or the CTO. I 00:14:09.659 --> 00:14:12.960 mean at the top level of the executives of the 00:14:12.960 --> 00:14:17.360 organization, including those people. IT has 00:14:17.360 --> 00:14:21.570 to deliver strategic value. In this world of 00:14:21.570 --> 00:14:25.210 cloud costs and the cost of IT and the cost of 00:14:25.210 --> 00:14:27.830 IT labor and all of that, you have to be able 00:14:27.830 --> 00:14:30.509 to justify the value of your organization because 00:14:30.509 --> 00:14:33.110 IT organizations that don't provide value, guess 00:14:33.110 --> 00:14:36.509 what? They get outsourced. Those outsource teams 00:14:36.509 --> 00:14:39.850 don't provide the great value that is usually 00:14:39.850 --> 00:14:42.570 hoped for either. That's some cases they do. 00:14:44.879 --> 00:14:47.919 I think what you have to do is really start with, 00:14:48.220 --> 00:14:51.559 it's not so much how you manage just infrastructure. 00:14:51.779 --> 00:14:55.980 I mean, I know that's the primary focus of your 00:14:55.980 --> 00:14:58.159 conversations on this podcast, but I think it's 00:14:58.159 --> 00:15:02.200 at a higher level of making sure that your IT 00:15:02.200 --> 00:15:05.879 organization and specific, your IT leader has 00:15:05.879 --> 00:15:08.620 a seat at the table and that they are participating 00:15:08.620 --> 00:15:12.519 in those strategic discussions and advising rather 00:15:12.519 --> 00:15:20.580 than order taking. Then it's easier to, number 00:15:20.580 --> 00:15:22.799 one, justify, okay, we're spending a lot on Cloud. 00:15:22.919 --> 00:15:25.240 Well, we made together this strategic discussion 00:15:25.240 --> 00:15:27.899 or decision that we're going to invest in Cloud 00:15:27.899 --> 00:15:31.480 because we need to scale, we need to be able 00:15:31.480 --> 00:15:34.179 to support things in this region, whatever your 00:15:34.179 --> 00:15:36.539 justifications are. But you've done that as a 00:15:36.539 --> 00:15:40.720 collective group rather than going off and trying 00:15:40.720 --> 00:15:43.059 to deliver what you think the business wants. 00:15:44.080 --> 00:15:46.240 and then coming back and then being questioned 00:15:46.240 --> 00:15:49.639 whether you're raising the value or not. And 00:15:49.639 --> 00:15:52.299 in a world where all these systems are constantly 00:15:52.299 --> 00:15:56.059 scaling and shifting and I'm curious, how can 00:15:56.059 --> 00:15:58.480 teams ensure that they're collecting the right 00:15:58.480 --> 00:16:00.580 data, not collecting everything, collecting the 00:16:00.580 --> 00:16:03.279 right data, making that shift to make informed 00:16:03.279 --> 00:16:06.620 decisions without being overwhelmed by the volume? 00:16:06.820 --> 00:16:08.679 I mean, I know there's an argument that AI will 00:16:08.679 --> 00:16:11.200 make that volume a little bit easier to digest, 00:16:11.480 --> 00:16:18.120 but anything you're seeing here? I think I joke 00:16:18.120 --> 00:16:20.940 sometimes that when people get so excited about 00:16:20.940 --> 00:16:23.159 AI, I say it's the new gluten -free, right? We 00:16:23.159 --> 00:16:27.659 put AI on a lot of things. At one point, I think 00:16:27.659 --> 00:16:31.480 I saw that gluten -free was on a laundry detergent 00:16:31.480 --> 00:16:33.659 and I scratched my head. I'm like, why does that 00:16:33.659 --> 00:16:38.159 matter? But I think that we've got to go back 00:16:38.159 --> 00:16:42.240 to the reality of collecting data with intent. 00:16:43.510 --> 00:16:47.450 It goes back to business value. If I work with 00:16:47.450 --> 00:16:50.710 a team in my past, they get wrapped around this 00:16:50.710 --> 00:16:53.090 and it goes back to they'd want to measure this 00:16:53.090 --> 00:16:55.029 component and that component instead of thinking 00:16:55.029 --> 00:16:58.070 about the service they're offering. I'll step 00:16:58.070 --> 00:17:02.090 back and say, tell me five reasons this feature 00:17:02.090 --> 00:17:05.750 exists in your product. Why did the product owner 00:17:05.750 --> 00:17:08.589 come to you and say, I want this to happen, code 00:17:08.589 --> 00:17:12.519 this? Then what does good look like? And I promise 00:17:12.519 --> 00:17:16.960 you, if you focus on traffic, error rate, latency, 00:17:17.119 --> 00:17:20.480 and saturation, just as a starting point, your 00:17:20.480 --> 00:17:22.619 leaps and bounds ahead of where you'd be if you 00:17:22.619 --> 00:17:25.119 started to see, is that server up? Is it up now? 00:17:25.319 --> 00:17:27.880 Was it up 10 minutes ago? And instead, looking 00:17:27.880 --> 00:17:31.099 at, is my service at the level I hoped? And build 00:17:31.099 --> 00:17:35.339 an SLO, spoiler alert, at the beginning, instead 00:17:35.339 --> 00:17:38.640 of at the end. I've been just as guilty as anybody 00:17:38.640 --> 00:17:42.349 else in this space. Five or six years ago, I 00:17:42.349 --> 00:17:46.390 started asking myself, why do we wait until after 00:17:46.390 --> 00:17:50.430 the release to think about an SLO? Why aren't 00:17:50.430 --> 00:17:54.390 we thinking about that in the first sprint? And 00:17:54.390 --> 00:17:57.009 then that informs metrics that you might need 00:17:57.009 --> 00:17:59.710 to measure that SLO. It informs how you can build 00:17:59.710 --> 00:18:02.789 an error budget. And then you get to a point 00:18:02.789 --> 00:18:07.680 where you can look at things with value, and 00:18:07.680 --> 00:18:09.940 you know what you specifically need to collect 00:18:09.940 --> 00:18:11.960 rather than just throwing everything in a pot 00:18:11.960 --> 00:18:18.200 and crossing your fingers. And as things grow 00:18:18.200 --> 00:18:20.079 and as you're scaling, that becomes more and 00:18:20.079 --> 00:18:22.220 more. It's got to be repeatable. It's got to 00:18:22.220 --> 00:18:24.920 be frictionless for a developer. Observability 00:18:24.920 --> 00:18:28.099 has to be a utility for people, not a burden. 00:18:28.279 --> 00:18:30.880 It has to be a first -class citizen. And I think 00:18:30.880 --> 00:18:35.900 observability is dealing with the sentiment in 00:18:35.900 --> 00:18:39.500 most organizations that that cyber had to deal 00:18:39.500 --> 00:18:43.059 with, you know, 10 years ago when we were really 00:18:43.059 --> 00:18:45.819 doubling down in organizations on cyber because 00:18:45.819 --> 00:18:48.799 it was becoming a real significant threat for 00:18:48.799 --> 00:18:51.019 everybody. Right. Developers had to do it. But 00:18:51.019 --> 00:18:53.259 it was another thing I got to do. And I got this 00:18:53.259 --> 00:18:55.960 giant delivery cycle observability sort of in 00:18:55.960 --> 00:18:57.819 that spot right now. We got to make it a first 00:18:57.819 --> 00:19:00.359 class citizen, too. And that includes making 00:19:00.359 --> 00:19:02.940 it frictionless and making it repeatable and 00:19:02.940 --> 00:19:05.740 doing that by simplifying the approach of how 00:19:05.740 --> 00:19:09.210 you start monitoring. And we've both jokingly 00:19:09.210 --> 00:19:11.430 shared stories of our past and that page are 00:19:11.430 --> 00:19:13.450 going off when we didn't want it to. But I think 00:19:13.450 --> 00:19:16.349 for many teams, they've almost been conditioned 00:19:16.349 --> 00:19:19.630 to operate with a reactive mindset after years 00:19:19.630 --> 00:19:23.309 of firefighting, fixing P1s, et cetera. So what's 00:19:23.309 --> 00:19:25.910 the best way to foster that cultural shift that's 00:19:25.910 --> 00:19:29.670 required to move towards a much better and stress 00:19:29.670 --> 00:19:32.509 -free, proactive, data -driven decision -making? 00:19:32.670 --> 00:19:35.269 It almost sounds like a utopia, me saying this 00:19:35.269 --> 00:19:38.319 out loud after years of doing the same. What 00:19:38.319 --> 00:19:42.019 needs to happen here? I joke regularly that I 00:19:42.019 --> 00:19:44.680 generally talk in euphoric sense, you know, like, 00:19:44.759 --> 00:19:48.000 or utopic sense rather than, you know, reality. 00:19:48.000 --> 00:19:51.259 But it is my opinions are based on reality and 00:19:51.259 --> 00:19:55.619 repeated mistakes on my own part. But I think 00:19:55.619 --> 00:19:58.779 for us to get people to shift, they have to understand 00:19:58.779 --> 00:20:00.900 the power of data and the data that's available 00:20:00.900 --> 00:20:08.240 to them. I look at a histogram. The power of 00:20:08.240 --> 00:20:11.440 a histogram that shows you things like performance 00:20:11.440 --> 00:20:14.700 or error rate from one minute to the next can 00:20:14.700 --> 00:20:18.539 help you really think about ways to be more predictive. 00:20:18.839 --> 00:20:21.579 If you're using SLOs, if you've decided you're 00:20:21.579 --> 00:20:25.740 going to shift your mind and do an SLO first 00:20:25.740 --> 00:20:28.380 and you do it in the sprint and lo and behold, 00:20:28.539 --> 00:20:30.880 you go to production, you've got a good SLO and 00:20:30.880 --> 00:20:32.779 you've got good error budget, even if it's really 00:20:32.779 --> 00:20:37.119 simple, You're a step ahead and now you're saying 00:20:37.119 --> 00:20:40.539 okay I don't have to react and wait till somebody 00:20:40.539 --> 00:20:43.319 calls me because something breaks or maybe I'm 00:20:43.319 --> 00:20:45.940 a step ahead and and my tools tell me something 00:20:45.940 --> 00:20:50.579 broke before the customer but So many of the 00:20:50.579 --> 00:20:54.980 things that I saw that were these big Newsworthy 00:20:54.980 --> 00:20:58.299 events not just at my former employer, but it 00:20:58.299 --> 00:21:02.400 in general right Started out as small paper cuts 00:21:03.490 --> 00:21:06.809 hours, days before. And if there were little 00:21:06.809 --> 00:21:09.829 data points to tell you, like, hey, you're starting 00:21:09.829 --> 00:21:11.569 to burn your error budget a lot more than you 00:21:11.569 --> 00:21:15.630 were. That's the easiest story to tell to get 00:21:15.630 --> 00:21:18.170 people to think about how to go from it and get 00:21:18.170 --> 00:21:20.089 away from, well, my app's up. Nobody's calling 00:21:20.089 --> 00:21:24.930 about it, right? I can log in. It's a shift in 00:21:24.930 --> 00:21:27.970 mentality and shift to get them to realize that 00:21:28.079 --> 00:21:30.279 Look, you got woken up at three, but this started 00:21:30.279 --> 00:21:32.019 at like 10 in the morning while you were having 00:21:32.019 --> 00:21:36.759 coffee. Like, you know, what if, what if it interrupted 00:21:36.759 --> 00:21:39.579 your coffee and started to sleep? Yeah. Oh, now 00:21:39.579 --> 00:21:43.059 that's progress, but a hundred percent. I think. 00:21:43.400 --> 00:21:46.779 Thanks to pesky, shiny emerging technologies 00:21:46.779 --> 00:21:50.559 like, yes, AI, agentic AI or whatever, the next 00:21:50.559 --> 00:21:54.140 big thing is we seem to have a lot of stalled 00:21:54.140 --> 00:21:57.220 projects, things caught in pilot. So as a result, 00:21:57.539 --> 00:22:00.640 leadership teams now are putting an extra special 00:22:00.640 --> 00:22:02.759 caution on all tech projects. Everything's got 00:22:02.759 --> 00:22:06.079 to be about ROI, measurable difference, quite 00:22:06.079 --> 00:22:09.500 rightly too. So how can organizations effectively 00:22:09.500 --> 00:22:12.859 measure the ROI of investing in a new observer? 00:22:12.750 --> 00:22:16.210 strategy, especially when benefits aren't always 00:22:16.210 --> 00:22:19.250 tangible at first or may appear that way. And 00:22:19.250 --> 00:22:21.990 I suspect this is an area you're passionate about 00:22:21.990 --> 00:22:27.109 too. I am. And it goes back to my line. If your 00:22:27.109 --> 00:22:30.250 CFO knows what observability is because of the 00:22:30.250 --> 00:22:32.509 cost rather than the value, you've got a problem. 00:22:33.329 --> 00:22:36.430 To get you there, to get your organization there, 00:22:36.490 --> 00:22:39.890 to realize that value, observability should underpin 00:22:39.890 --> 00:22:43.299 your business objectives. Right. You don't just 00:22:43.299 --> 00:22:46.519 decide. I mean, maybe if you're a nerd like me 00:22:46.519 --> 00:22:48.519 and you see what the cool things are, then you 00:22:48.519 --> 00:22:51.519 might think, yeah, absolutely. We should do observability. 00:22:51.779 --> 00:22:53.940 And you may be not thinking about how it's tying 00:22:53.940 --> 00:22:56.579 to objectives at a higher level. Right. It's 00:22:56.579 --> 00:23:00.140 cool. I had a leader years ago tell me cool is 00:23:00.140 --> 00:23:06.359 not a business case. It's not. But. If I can 00:23:06.359 --> 00:23:09.019 tie observability back to, if I'm an e -commerce 00:23:09.019 --> 00:23:11.220 company, I care about how quickly you can buy 00:23:11.220 --> 00:23:13.640 widgets, I care about conversion rate, I care 00:23:13.640 --> 00:23:16.599 about how quickly you go through the funnel, 00:23:17.240 --> 00:23:19.900 right? And I care about things that interrupt 00:23:19.900 --> 00:23:21.839 that funnel and make you abandon your shopping 00:23:21.839 --> 00:23:25.210 cart. Observability helps you get there. If you 00:23:25.210 --> 00:23:28.289 design observability for that application to 00:23:28.289 --> 00:23:31.250 say, I care about response time and my services, 00:23:31.390 --> 00:23:33.829 I care about how long it takes somebody to be 00:23:33.829 --> 00:23:37.130 presented with the 40 widgets that I sell and 00:23:37.130 --> 00:23:40.250 pay for it and all of that, then I'm underpinning 00:23:40.250 --> 00:23:42.450 a business objective that is likely sell more 00:23:42.450 --> 00:23:46.109 widgets or increase my conversion rate by X percent. 00:23:46.529 --> 00:23:49.589 If observability underpins your business strategy, 00:23:49.789 --> 00:23:52.869 it's easier for people to swallow. And then instead 00:23:52.869 --> 00:23:57.410 of measuring with MTTR, which typically only 00:23:57.410 --> 00:24:01.190 technology leaders care about, then you're selling 00:24:01.190 --> 00:24:05.170 it with the idea of, hey, look, we drove better 00:24:05.170 --> 00:24:07.509 performance, which you, Mr. and Mrs. Product 00:24:07.509 --> 00:24:11.029 Owner care about because it's driving better 00:24:11.029 --> 00:24:12.930 conversion rate and therefore better revenue 00:24:12.930 --> 00:24:18.059 based on what your product is doing. To bring 00:24:18.059 --> 00:24:20.559 everything that we've talked about here today 00:24:20.559 --> 00:24:24.000 back to your work at Chronosphere, are you able 00:24:24.000 --> 00:24:26.599 to share maybe a real world example of how a 00:24:26.599 --> 00:24:29.220 strategic observability plan has actually helped 00:24:29.220 --> 00:24:32.299 an organization move from that reactive firefighting 00:24:32.299 --> 00:24:35.680 model to be more proactive and innovative one 00:24:35.680 --> 00:24:38.440 and get that over the line? Anything you can 00:24:38.440 --> 00:24:39.720 share on that? You don't have to mention any 00:24:39.720 --> 00:24:43.400 names. I think it'd be useful. Well, I'll tell 00:24:43.400 --> 00:24:48.019 you, in my former life, we were early embracers 00:24:48.019 --> 00:24:51.440 of APM, right? And we were in a state where there 00:24:51.440 --> 00:24:53.059 were still a few folks that were saying, you 00:24:53.059 --> 00:24:56.819 know, nobody's calling, you know, nobody's calling 00:24:56.819 --> 00:24:59.019 the service desk, nobody is, you know, there's 00:24:59.019 --> 00:25:00.900 no problem, I can log in kind of a thing. They 00:25:00.900 --> 00:25:03.420 were still in that mode with some of the teams. 00:25:04.799 --> 00:25:09.039 Getting people to understand the value of, again, 00:25:09.119 --> 00:25:11.380 that histogram, getting people to be able to 00:25:11.380 --> 00:25:16.289 understand that A lot of the problems that keep 00:25:16.289 --> 00:25:19.009 us on phone calls for hours in the middle of 00:25:19.009 --> 00:25:23.089 the night started with a paper cut at seven in 00:25:23.089 --> 00:25:26.130 the morning, and maybe seven in the morning on 00:25:26.130 --> 00:25:32.869 Wednesday, and it's Sunday morning. Changing 00:25:32.869 --> 00:25:36.289 that perspective and showing that this data is 00:25:36.289 --> 00:25:39.130 possible together, it's not difficult together, 00:25:39.130 --> 00:25:42.480 and then when it's there, It gives you a lot 00:25:42.480 --> 00:25:46.539 of great things. And one of the best ways is 00:25:46.539 --> 00:25:51.660 to help people. I call it mean time to innocence. 00:25:52.180 --> 00:25:55.240 Right. Hey, look, I know it's not my problem 00:25:55.240 --> 00:25:58.420 and I know it's not because here's data. Right. 00:25:58.420 --> 00:26:00.599 Here's my service. It's calling this. And oh, 00:26:00.640 --> 00:26:03.579 guess what? It's your service because I'm calling 00:26:03.579 --> 00:26:06.420 you. And every other third time I call you, it 00:26:06.420 --> 00:26:12.480 fails. You're killing me. Right. So we did that. 00:26:12.480 --> 00:26:15.299 We did that. And I was proud to say that some 00:26:15.299 --> 00:26:18.099 of the earliest adopters of APM in our organization 00:26:18.099 --> 00:26:22.039 were the NetOps team members, because they got 00:26:22.039 --> 00:26:24.039 tired of being blamed for it being the network, 00:26:24.539 --> 00:26:26.000 because it's always the network, right? That 00:26:26.000 --> 00:26:28.259 was always a running joke in every organization, 00:26:28.359 --> 00:26:32.119 I'm sure. But these guys and gals would get on, 00:26:32.279 --> 00:26:34.930 look at the tools and say, it's not the network. 00:26:35.190 --> 00:26:37.529 And as a matter of fact, it's this service right 00:26:37.529 --> 00:26:39.869 here that keeps failing. And it started right 00:26:39.869 --> 00:26:45.589 when people started calling. So it's, it's, there 00:26:45.589 --> 00:26:48.269 are a lot of, I can probably think of hours of 00:26:48.269 --> 00:26:51.269 examples of things like that, both in my current 00:26:51.269 --> 00:26:53.930 or with my former employer. And as I talk to 00:26:53.930 --> 00:26:56.539 customers in my current role. And just listening 00:26:56.539 --> 00:26:59.259 to that story reminds me people were so fiercely 00:26:59.259 --> 00:27:01.619 defensive back in the day on things like that 00:27:01.619 --> 00:27:04.079 with, it almost felt like the Spider -Man meme 00:27:04.079 --> 00:27:06.740 with the network guy pointing at the apps guy 00:27:06.740 --> 00:27:08.740 and the apps guy pointing at the, you know, everyone's 00:27:08.740 --> 00:27:12.019 pointing at each other. But if an IT is listening 00:27:12.019 --> 00:27:14.880 to a conversation today, maybe they want to take 00:27:14.880 --> 00:27:18.599 one actionable step tomorrow too. begin to improve 00:27:18.599 --> 00:27:21.400 their observability posture, what would you recommend 00:27:21.400 --> 00:27:24.059 they do? I appreciate it's not as simple as that, 00:27:24.279 --> 00:27:26.099 but if they want to take that first baby step, 00:27:26.240 --> 00:27:28.859 what should they do? One of my favorite questions 00:27:28.859 --> 00:27:30.680 when I walk into a customer as we're talking 00:27:30.680 --> 00:27:33.599 to them is, why did you buy observability and 00:27:33.599 --> 00:27:36.220 how is it supporting your business goal? Yeah. 00:27:37.400 --> 00:27:41.599 I mean, and if the answer is, I don't know, or 00:27:41.599 --> 00:27:44.500 they stumble for it, then they probably just 00:27:44.500 --> 00:27:48.920 need to take a step back and think about it. 00:27:49.019 --> 00:27:51.880 So many people jumped into observability because 00:27:51.880 --> 00:27:56.039 they needed something, and it was an evolving 00:27:56.039 --> 00:28:01.799 technology and evolving tool. APM promised some 00:28:01.799 --> 00:28:04.359 really amazing things easily, and as the cloud 00:28:04.359 --> 00:28:07.039 changed things, that got more difficult in it. 00:28:08.430 --> 00:28:10.390 diminished a little bit and people had to work 00:28:10.390 --> 00:28:13.710 harder to get it. And so it's even more important 00:28:13.710 --> 00:28:15.950 to have that strategy. And that strategy starts 00:28:15.950 --> 00:28:18.589 with, why am I doing this? I'm doing this so 00:28:18.589 --> 00:28:21.190 I can sell more widgets. I know I'm selling more 00:28:21.190 --> 00:28:23.930 widgets because I'm driving better conversion. 00:28:24.009 --> 00:28:26.269 I'm driving better conversion because my performance 00:28:26.269 --> 00:28:29.609 is better. And I know that if I change my performance 00:28:29.609 --> 00:28:33.150 by X number of milliseconds, I drive my conversion 00:28:33.150 --> 00:28:39.339 up by X percentage. I think it's really sitting 00:28:39.339 --> 00:28:41.339 back and asking yourself, why am I doing this? 00:28:41.680 --> 00:28:44.160 Because somebody is going to ask you. Don't wait 00:28:44.160 --> 00:28:47.839 till your CFO asks you. And I think that is a 00:28:47.839 --> 00:28:50.559 powerful moment to end on today. But anyone we've 00:28:50.559 --> 00:28:52.400 inspired today, maybe they want to take that 00:28:52.400 --> 00:28:55.380 first baby step into improving that observability 00:28:55.380 --> 00:28:58.559 posture. Where can they find you or your team 00:28:58.559 --> 00:29:01.240 and more information on anything we talked about 00:29:01.240 --> 00:29:04.319 today? Where should they go? Certainly, you can 00:29:04.319 --> 00:29:07.680 find me on LinkedIn, and I'm always, as you can 00:29:07.680 --> 00:29:10.220 tell, happy and passionate to talk about observability. 00:29:10.900 --> 00:29:14.839 And part of my role as the CTO at Chronosphere 00:29:14.839 --> 00:29:19.259 is not to sell, but to guide and help folks with 00:29:19.259 --> 00:29:21.599 how to do things like an observability strategy. 00:29:21.859 --> 00:29:24.819 So you can also find me there. We're at chronosphere 00:29:24.819 --> 00:29:28.400 .io. And again, we're a cloud -native observability 00:29:28.400 --> 00:29:30.900 company focused on giving you control of your 00:29:30.900 --> 00:29:35.079 data. rather than drowning in it and not understanding 00:29:35.079 --> 00:29:38.680 what your telemetry is doing for you. Love that. 00:29:38.799 --> 00:29:41.039 So I will add links to absolutely everything. 00:29:41.059 --> 00:29:43.980 Make it easy for people to find. Thank you. We 00:29:43.980 --> 00:29:46.920 covered so much today from observability challenges 00:29:46.920 --> 00:29:49.420 that organizations are grappling with, collecting 00:29:49.420 --> 00:29:53.150 the right data to make informed decisions. As 00:29:53.150 --> 00:29:55.890 an XIT guy measuring the ROI of investing in 00:29:55.890 --> 00:29:58.670 a new observability strategy, the old belts and 00:29:58.670 --> 00:30:01.190 braces approach of you can only improve what 00:30:01.190 --> 00:30:03.630 you measure. So great to hear. Right. Just thank 00:30:03.630 --> 00:30:06.190 you for shining a light on this today. Thank 00:30:06.190 --> 00:30:09.230 you. Thanks for the time. Now, before we wrap, 00:30:09.390 --> 00:30:11.990 what will you change this week to collect with 00:30:11.990 --> 00:30:14.349 intent rather than just collecting everything? 00:30:15.089 --> 00:30:17.089 I think Bill left us with a simple challenge 00:30:17.089 --> 00:30:20.569 that almost doubles as a compass. Start with 00:30:20.569 --> 00:30:24.539 why your product exists. Define what good looks 00:30:24.539 --> 00:30:28.119 like. Set those SLOs early and let your choices 00:30:28.119 --> 00:30:31.420 guide the data that you're gathering. And this 00:30:31.420 --> 00:30:34.259 is how observability can earn a seat at the strategy 00:30:34.259 --> 00:30:37.240 table and how teams reclaim time for building, 00:30:37.900 --> 00:30:41.420 not just wasting their days firefighting. So 00:30:41.420 --> 00:30:43.819 if this sparked an idea, please connect with 00:30:43.819 --> 00:30:45.960 Bill on LinkedIn, check out their website. And 00:30:45.960 --> 00:30:47.299 if you've got anything you'd like to share with 00:30:47.299 --> 00:30:50.539 me, techtalksnetwork .com, please let me know. 00:30:50.700 --> 00:30:53.380 But that's it for today. So thank you for listening 00:30:53.380 --> 00:30:55.819 as always and I'll speak with you all again tomorrow