WEBVTT 00:00.000 --> 00:03.052 So we actually came up with a new metric which we call merge efficiency. 00:04.088 --> 00:07.770 How many builds on a branch does it take before you can merge something to main? 00:07.770 --> 00:13.974 And the really successful companies are seeing the same sort 00:13.974 --> 00:16.995 of merge rate or higher as they get better, 00:16.995 --> 00:20.947 right? And everything is increasing because they're applying AI, 00:20.947 --> 00:23.309 they're creating more PRs, they're creating them faster, 00:23.309 --> 00:25.300 they're creating even bigger ones, 00:25.300 --> 00:28.542 yet still faster and still merging faster. 00:28.542 --> 00:32.704 The sort of median company is creating a lot. 00:32.856 --> 00:37.201 more builds on a branch because their work is broken and it fails and 00:37.201 --> 00:41.245 it fails and it fails and it fails and then they finally get it merged into main 00:41.525 --> 00:44.118 or whatever your default branch is that we assume goes to production. 00:43.556 --> 00:44.556 Sure. 00:44.483 --> 00:45.483 Yep. 00:46.186 --> 00:49.757 And your final productivity, you're doing a lot more work. 00:49.757 --> 00:52.678 You're using more tokens and more energy and whatever. 00:52.678 --> 00:55.879 But then the final output is kind of the same. 00:55.879 --> 00:56.879 And so that's 00:56.356 --> 00:57.356 Mm. 00:56.839 --> 01:00.010 a really interesting gap. We've been trying to help people solve the for that gap, 01:00.010 --> 01:03.051 right? Like, how do you get more confidence earlier? 01:03.051 --> 01:06.392 How do you steer your agent? Assuming you're using an agent to build 01:06.412 --> 01:11.033 How do you steer your tools so that you're getting good outcomes earlier? 01:11.033 --> 01:14.955 Rob Zuber is a veteran software leader with over two decades of startup experience, 01:14.955 --> 01:18.658 currently serving as the chief technology officer at CircleCI. 01:18.658 --> 01:23.221 He joined the company back in 2014 following Circle CI's acquisition of Distiller, 01:23.221 --> 01:26.283 a continuous integration and deployment platform that he co-founded. 01:26.283 --> 01:30.366 Prior to that, Rob established a strong track record as a serial entrepreneur, 01:30.366 --> 01:33.178 co-founding technology companies such as Copius and Yoohoot 01:34.012 --> 01:38.135 Currently, he oversees product innovation and leads CircleCI's globally distributed 01:38.135 --> 01:42.388 engineering team. And today, Rob Zuber is our guest on the Engineering with 01:42.388 --> 01:43.388 AI podcast. 02:00.939 --> 02:03.120 cool. Rob, thank you so much for coming on the show today. 02:03.120 --> 02:04.311 We really appreciate having you on. 02:05.315 --> 02:06.315 Yeah, thanks for having me, Kyle. 02:06.239 --> 02:07.239 I'm excited to be here. 02:07.523 --> 02:12.664 Yeah, awesome. Well it's I'm I'm really keen to to find out more about your specific 02:12.664 --> 02:14.895 position at Circle CI. ~ you know, 02:14.895 --> 02:18.536 I'm I'm on teams that use it. So I'm I'm curious to know like 02:17.456 --> 02:18.456 Mm-hmm. Awesome. 02:18.536 --> 02:20.346 what you know about like what all's going on. 02:20.346 --> 02:25.218 ~ but let's start with you. you you caught the little bit of a bio that I read back. 02:25.218 --> 02:27.449 what would you add to that? what do you think about when you think about your 02:27.449 --> 02:28.680 career? What story would you tell? 02:28.680 --> 02:29.680 Yeah. 02:29.830 --> 02:31.631 Yeah, it's it's a great question. 02:31.631 --> 02:34.223 I mean I despite being forced to talk about it all the time, 02:34.223 --> 02:35.223 I don't think about it that much. 02:35.173 --> 02:38.486 But I think, you know, w when what one of the things that happens to me 02:38.486 --> 02:40.087 a lot is people ask me, you know, 02:40.087 --> 02:42.849 what advice do you have for me as sort of someone entering software? 02:42.373 --> 02:43.373 Mm. Mm-hmm. 02:42.849 --> 02:43.849 And there's 02:43.519 --> 02:45.390 two reasons I have no advice for them, 02:45.390 --> 02:47.192 or I could say this is what I did, 02:47.192 --> 02:48.993 but you know, maybe I don't know if that's gonna help you. 02:48.993 --> 02:52.335 One, obviously AI is changing everything and I'm just having this conversation 02:51.291 --> 02:52.313 Yeah. Mm-hmm. 02:52.335 --> 02:55.117 with juniors all the time of like I don't even know what you should 02:55.117 --> 02:56.568 be thinking about learning. Like, 02:56.568 --> 02:57.568 let's learn together. 02:57.842 --> 02:58.842 Which is really 02:57.927 --> 02:58.927 Mm-hmm. 02:58.362 --> 02:59.913 interesting and different. I can't just, 02:59.913 --> 03:04.366 you know, my my sage advice from 25 years ago is totally useless at this point, 03:04.366 --> 03:09.270 right? And the other is I I came into software after graduating college. 03:09.270 --> 03:11.341 So I didn't study, I didn't study computer science. 03:11.341 --> 03:14.093 ~ I studied engineering physics for whatever that's for. 03:14.093 --> 03:15.464 So I mean I technical background, 03:15.464 --> 03:16.684 but but ~ 03:16.085 --> 03:17.085 Yeah. 03:16.965 --> 03:21.057 sort of it was the late 90s, it was the original dot-com boom. 03:21.057 --> 03:25.550 I mean, anybody kind of that was interested in finding a job was was moving into 03:25.572 --> 03:26.572 ~ tech. 03:27.323 --> 03:28.984 But then continued that pattern. 03:28.984 --> 03:30.985 I spent some time doing product management, 03:30.985 --> 03:32.495 some time doing business development, 03:32.495 --> 03:36.667 some time teaching. And so I think a lot of that exploration, 03:36.667 --> 03:39.528 ~ you know, I always came back to software, 03:39.528 --> 03:40.528 ~ 03:40.177 --> 03:41.177 Mm. 03:40.529 --> 03:45.511 but helped me build a really good understanding as an executive of what 03:45.511 --> 03:50.584 my my partners and my colleagues need from me and a lot more empathy. 03:50.584 --> 03:51.974 Like I think it's pretty classic 03:51.548 --> 03:52.548 Mm. 03:51.974 --> 03:53.775 as a software engineer to be like, 03:54.331 --> 03:55.872 What do these salespeople even do? 03:55.872 --> 03:56.872 We build 03:56.336 --> 03:57.336 Yeah. 03:56.422 --> 03:57.422 everything, 03:57.023 --> 03:58.582 they just get paid for it. You know, 03:58.582 --> 04:04.617 like this kind of nonsense. Spend five minutes watching a salesperson do their job, 04:04.617 --> 04:08.179 and you'll be like, I wouldn't have the slightest clue how to make this work. 04:08.179 --> 04:09.179 Right. Like I 04:08.453 --> 04:09.453 Yeah, yeah, yeah. 04:09.140 --> 04:11.301 would be terrified of every single part of this. 04:11.301 --> 04:12.882 And I I I remember a few moments, 04:12.882 --> 04:16.685 both as an engineer supporting salespeople in the field, 04:16.685 --> 04:19.028 and then as someone again doing business development and sort 04:19.028 --> 04:20.267 of sales-related activities. 04:20.987 --> 04:23.168 ~ just being like, no, I know what I'm good at, 04:23.168 --> 04:26.179 and it's not this. And I'm so thankful there are people who are good at this, 04:24.653 --> 04:27.117 Yeah. Yeah, yeah. They they do this all the time. 04:26.179 --> 04:28.469 right? Like, cause then because then when you go and build companies, 04:27.117 --> 04:28.696 Yeah. Yeah. 04:29.450 --> 04:31.720 you like early, you have to do all those jobs, 04:31.720 --> 04:34.651 which is another great way to get that rounded experience to allow you 04:34.651 --> 04:36.462 to become an executive that isn't just like, 04:36.462 --> 04:37.632 I know how to write software now, 04:37.632 --> 04:38.632 how do I run a company 04:37.851 --> 04:38.851 Yeah, yeah. 04:38.422 --> 04:44.094 sort of thing? ~ but you're so thankful when people show up with like tons 04:44.094 --> 04:46.795 of experience, you know, doing that job. 04:45.157 --> 04:47.480 Right. Know how it's actually done. 04:46.795 --> 04:47.825 ~ 04:48.314 --> 04:49.914 Yeah, exactly. Like, you know, 04:49.914 --> 04:52.266 ev even at Circle CI where I I joined, 04:52.266 --> 04:56.648 we were about ~ I think 14 people or something when when three of 04:56.648 --> 04:59.630 us joined through that acquisition ~ that you mentioned of Distiller, 04:59.630 --> 05:03.952 you know, we were doing sales for a long time. 05:03.952 --> 05:06.874 And when the first, you know, proper salespeople came in, 05:06.874 --> 05:09.185 I was like, thank you for being here. 05:09.185 --> 05:11.066 I'm so glad that I don't you know, 05:11.066 --> 05:14.158 like I I I'm okay. I'm not afraid to talk to customers, 05:14.158 --> 05:15.939 but there's just, you know, 05:16.453 --> 05:18.204 The process, the steps you go through, 05:18.204 --> 05:20.945 the repeatability, all of it is just not what what you do. 05:20.945 --> 05:23.726 Right. But I I do think dabbling, 05:23.726 --> 05:25.527 getting some experience, 05:24.335 --> 05:25.484 Mm-hmm. Breath. 05:25.527 --> 05:27.318 you know, playing a salesperson on TV, 05:27.318 --> 05:28.869 whatever, like it gives you enough, 05:28.869 --> 05:32.180 again, empathy and then understanding of what those folks 05:29.713 --> 05:32.002 Yeah. Yep. 05:32.180 --> 05:35.932 need from you. Cause I think technical people really get into like, 05:35.932 --> 05:39.984 I explain my technical details and it's very logical. 05:39.984 --> 05:41.875 So I don't understand why you don't understand. 05:40.913 --> 05:42.453 Mm. ~ 05:42.247 --> 05:43.641 Do you know what mean? And learning 05:42.453 --> 05:43.470 right, right. Mm-hmm. 05:43.641 --> 05:46.751 how other people think and what they're listening for and what they need 05:46.751 --> 05:49.048 out of a conversation or about out of your information, 05:48.103 --> 05:49.784 It was different. Yeah. 05:49.048 --> 05:50.048 I think is is really 05:50.465 --> 05:51.916 Yeah. Yeah. For sure. For sure. 05:51.916 --> 05:53.307 I I I love that you mentioned that. 05:53.307 --> 05:57.489 I will never forget the day working as this is when I was at Bindley, 05:57.489 --> 05:59.267 a little software as a service company. 05:59.267 --> 06:00.732 We ended up getting acquired by Workday. 06:00.732 --> 06:05.323 And ~ I I was sort of on the front line of a team that had I 06:05.495 --> 06:09.878 you know involved somewhat in the sale and was part of the conversations about what 06:09.878 --> 06:12.179 we did and didn't have and what they did and didn't need. 06:12.179 --> 06:14.641 And and then later you get in there and you realize, 06:14.641 --> 06:17.343 there there was things you didn't know you needed to tell us. 06:17.595 --> 06:18.806 You know, like they're they're 06:18.236 --> 06:19.236 Mm-hmm. 06:18.806 --> 06:22.478 thinking about the current solution and it pisses me off in the following two ways. 06:22.478 --> 06:23.898 So you better have it better than that. 06:23.898 --> 06:26.030 But they don't tell you the hundred ways in which it's already fine, 06:26.030 --> 06:27.280 you know? And so they don't 06:26.768 --> 06:27.768 Right. 06:27.280 --> 06:31.643 necessarily think to check and and you don't think to ask and yeah, 06:31.643 --> 06:34.524 it's all just like we're all just sort of crashing into each other in terms 06:34.524 --> 06:37.526 of how all this works. ~ and we're all doing our best. 06:36.337 --> 06:37.337 Yeah. Yeah. 06:37.526 --> 06:41.008 And I can remember being on the other side when you're not involved with that 06:41.008 --> 06:42.269 and you're just like, What on earth? 06:42.269 --> 06:43.950 Who said yes to this deal? You know? 06:43.950 --> 06:44.960 And you learn some 06:44.165 --> 06:45.165 Right. Right. 06:44.960 --> 06:46.311 empathy of like, okay, actually. 06:46.531 --> 06:48.112 It's not as easy as it sounds, 06:48.112 --> 06:51.686 right? It it feels like you ought to just be able to check all these details, 06:51.686 --> 06:53.457 but you kinda can't in the end. 06:53.457 --> 06:58.639 It's yeah, it's I mean, the kind of asymmetry of hindsight, 06:58.639 --> 06:59.639 meaning 06:59.242 --> 07:00.242 Yeah. 06:59.979 --> 07:00.979 one 07:00.229 --> 07:03.771 person looked at it without the hindsight and then another person comes along 07:03.771 --> 07:05.401 and looks at it, you know, after the fact. 07:05.401 --> 07:06.401 And 07:05.611 --> 07:06.611 Right. 07:06.081 --> 07:11.083 it's like it's every engineer walking into someone else's code base is, 07:11.083 --> 07:13.980 you know, you don't even have to take it to like selling a deal or whatever. 07:11.113 --> 07:13.097 Yeah, yeah. Everything's terrible. 07:13.097 --> 07:16.503 I hate all of this. Yeah. Yeah, 07:13.980 --> 07:16.685 You're like, why would anyone have ever done this this way? 07:16.503 --> 07:17.503 yeah. 07:16.685 --> 07:17.685 But 07:17.846 --> 07:20.787 but someone very rational, right? 07:20.799 --> 07:21.799 Did. Yeah. 07:21.823 --> 07:24.066 Who would happily look at you some of your stuff and be like, 07:24.066 --> 07:26.309 what on earth? Right. They they 07:24.681 --> 07:26.086 Yeah, for sure. 07:26.309 --> 07:29.583 had a different context. They were they had half the information that you now have, 07:29.583 --> 07:32.036 all those things, right? And ~ I yeah, 07:32.036 --> 07:33.548 I think there's a huge amount in there, 07:33.548 --> 07:36.272 even within our own domain, but then especially across domains, 07:36.272 --> 07:40.167 to just say, wow, like there's so many things about this that I don't understand. 07:36.755 --> 07:38.976 Hundred percent. Well 07:40.577 --> 07:42.477 and I like how you mentioned AI is changing everything. 07:42.477 --> 07:44.628 That I mean that is partially what we're here to talk about. 07:44.628 --> 07:45.919 And it is true, right? 07:44.970 --> 07:45.970 Yeah. Yeah. 07:45.919 --> 07:49.360 Like this this feels like one of the biggest changes in my career. 07:49.360 --> 07:51.061 And it sounds like our careers are about as long. 07:51.061 --> 07:54.143 ~ so so let's get into that. Like when you're using 07:53.707 --> 07:54.707 Mm-hmm. 07:54.143 --> 07:57.044 these tools, when you're sitting down to do technical work, 07:57.044 --> 07:58.464 if you're keeping your skills sharp, 07:58.464 --> 08:01.276 ~ or or you know, organizing, you know, 08:01.276 --> 08:04.087 strategy or anything else, like what what tools are you reaching for 08:04.760 --> 08:07.562 ~ as part of your day to day like practitioner work these days. 08:07.562 --> 08:08.562 Yeah. 08:08.265 --> 08:11.297 Yeah. And ~ just before I talk about specific tools, 08:11.297 --> 08:13.579 it's worth saying, because you you kind of said if this or that, 08:13.579 --> 08:18.862 like I I am. I I'm thinking strategically we're a software developer tool company. 08:18.862 --> 08:24.576 In order to do that, I can't not I for the way my brain works and the 08:24.576 --> 08:26.207 way that I think about what our customer needs, 08:26.207 --> 08:30.811 I can't not use the tools. if I'm just reading blog posts and saying this 08:30.811 --> 08:32.452 is probably the direction we should go as a company, 08:32.452 --> 08:33.972 I feel disconnected from my people. 08:33.972 --> 08:35.443 I just feel disconnected from my customers. 08:35.443 --> 08:37.075 So I am a very hands-on learner. 08:37.772 --> 08:41.714 ~ and so I'd say i the basics, 08:41.714 --> 08:48.264 right? I mean, I spend a decent amount of time with both Claude and ChatGPT. 08:48.264 --> 08:49.266 It kind of depends on the day. 08:49.266 --> 08:50.327 I have funny stories about 08:49.941 --> 08:50.941 Mm-hmm. 08:50.327 --> 08:52.868 how they interact with me and their personas that some days I'm just 08:52.868 --> 08:54.528 not comfortable with one of them versus the other, 08:54.528 --> 08:57.039 but it doesn't matter. So a bunch of just like, 08:57.039 --> 08:58.626 how would I go about this? You know, 08:58.626 --> 09:02.411 what what would be what tools are available to me to try to achieve this thing? 09:02.411 --> 09:03.562 What does good look like in this case? 09:03.562 --> 09:06.283 And I always end up in this weird situation where then. 09:06.655 --> 09:09.797 You know, if I'm using one of the more chat tools as opposed to like codex 09:09.797 --> 09:13.420 or claw code, they'll spit out some code and I go, 09:13.420 --> 09:15.111 What am I doing with this tool? 09:15.111 --> 09:18.263 I should be, you know, in a terminal right now. 09:18.263 --> 09:19.364 Like if you're gonna give me code, 09:18.901 --> 09:20.582 Right. Right. Right. 09:19.364 --> 09:20.595 let me go to and then I 09:20.595 --> 09:24.007 I let's say I'm in clo like Claude Chat. 09:24.007 --> 09:27.029 Then I'll go to Claude Code and say, 09:27.029 --> 09:30.792 Your friend Claude Chat told me and then like paste some stuff. 09:30.419 --> 09:32.904 Yeah, yeah. huh. Yeah. 09:30.792 --> 09:32.413 I don't know why. I feel it's really important. 09:32.413 --> 09:33.413 Like, I 09:32.993 --> 09:34.865 haven't been cheating on you, Claude Code. 09:34.865 --> 09:35.865 Like I'm 09:35.667 --> 09:38.218 I'm seeing your brother Claude on the side or whatever. 09:38.218 --> 09:42.391 Like it's a very weird thing. And ~ I don't know why. 09:42.391 --> 09:47.814 Like I it and I I try so hard in a way not to personify 09:47.814 --> 09:49.895 or anthropomorphize like these tools, 09:48.159 --> 09:50.886 Yeah, I know. But it's hard. Yeah. 09:49.895 --> 09:53.197 but they try really hard to have me do that. 09:50.886 --> 09:54.927 That's right. To anthropomorphize themselves. 09:53.197 --> 09:55.118 And so there's like this weird interaction. 09:54.927 --> 09:55.927 Yeah. 09:55.118 --> 09:57.329 Anyway, so so I, you 09:57.329 --> 09:59.190 know, codex and claude code I use, 09:59.190 --> 10:00.641 I wouldn't say interchangeably, 10:00.641 --> 10:03.593 but sometimes I feel like I'm going down a weird 10:04.467 --> 10:07.820 rabbit hole or something or feel like I'm off track and I I'll find opening 10:07.820 --> 10:11.783 up another one. ~ mostly because it's going to give me a different model. 10:11.783 --> 10:14.536 Right. And then of course I've dabbled in pi (agent SDK) and open weight models 10:14.536 --> 10:17.218 and other stuff just to kind of see where the state of those 10:15.467 --> 10:17.820 Mm-hmm. Okay. Cool. 10:18.048 --> 10:21.702 because that matters to me in terms of like everybody's talking about token cost 10:21.702 --> 10:24.284 management. Everybody's talking about like the evolution of these tools 10:22.623 --> 10:25.393 Yeah, yeah. Yep. 10:24.284 --> 10:25.845 and whatever and optionality. 10:25.845 --> 10:32.821 And but I will switch between sometimes just for like a fresh perspective. 10:33.344 --> 10:36.486 I guess, right? Like, hey, I got the you know, 10:36.486 --> 10:37.746 this was built by something else. 10:37.746 --> 10:40.317 ~ you know, what's your take on it? 10:40.317 --> 10:41.317 Is there another, you know, 10:40.757 --> 10:41.757 Mm-hmm. 10:41.108 --> 10:44.489 w look at or or you know, and sometimes I'll do this with the same tool. 10:44.489 --> 10:46.630 Like, okay, cool, we built that together. 10:46.630 --> 10:49.467 One, we have some internal review tools that we've constructed. 10:49.467 --> 10:52.253 I think everybody's built some review tools of some point, 10:52.253 --> 10:54.824 right? But like now put your adversarial hat on, 10:54.824 --> 10:56.095 right? Like if you were checking for this 10:55.637 --> 10:56.637 Mm-hmm. 10:56.095 --> 10:57.756 specifically, do you see 10:58.844 --> 11:02.656 You know, do you see patterns that don't align with our coding standards? 11:02.656 --> 11:04.427 Do you see issues with security? 11:04.427 --> 11:07.749 Like we we have a bunch of tooling to kind of force that cycle. 11:07.749 --> 11:09.987 ~ a lot of this is is, you know, 11:09.987 --> 11:14.533 we're we're homegrowing and solving our problems in an effort to build things 11:14.533 --> 11:16.274 we think will solve our customers' problems. 11:16.274 --> 11:17.274 ~ 11:16.351 --> 11:17.351 Yeah, yeah. 11:17.294 --> 11:19.325 and so we're, you know, we're trying to get stuff out there, 11:19.325 --> 11:22.067 but then we're also trying to push the limits internally and see what what works 11:22.067 --> 11:25.859 and what doesn't. ~ but I think that like both 11:27.051 --> 11:30.173 Fresh perspective, some and also sometimes, 11:30.173 --> 11:34.495 you know, like the model providers are having so many issues with capacity that 11:34.495 --> 11:37.797 sometimes it's just like who who's gonna answer my query right 11:37.797 --> 11:44.551 now is another driver. So like we we as an organization earlier this 11:44.551 --> 11:47.942 year, so funny, it feels like it was maybe 10 years ago at the rate that things 11:47.942 --> 11:49.543 are changing, right? But earlier this year, 11:49.543 --> 11:56.117 we we had been in a place where we were encouraging folks to go explore any tools. 11:57.045 --> 11:59.316 Pick any of the tool any of the AI tools off the shelf, 11:59.316 --> 12:01.536 you know, we and we had subscriptions to a bunch of different ones. 12:01.536 --> 12:03.117 Like we were doing the evaluation, 12:03.117 --> 12:04.627 are we comfortable with their terms of service, 12:04.627 --> 12:06.177 that kind of stuff. But then saying within this, 12:06.177 --> 12:09.558 within this pool, because we're a developer tool provider, 12:09.558 --> 12:12.959 we want people to have experience with all these tools. 12:12.959 --> 12:18.241 And we found two things. One, a number of our folks might be trying these things, 12:18.241 --> 12:19.761 but not really giving feedback. 12:19.761 --> 12:21.782 So it wasn't actually valuable to us. 12:20.325 --> 12:22.366 Mm. Mm. 12:21.782 --> 12:22.782 So it 12:22.432 --> 12:25.603 was better to constrain down to a particular tool set. 12:25.603 --> 12:26.603 And two, 12:27.263 --> 12:30.565 While we like to believe that every engineer wants to futz about with 12:30.565 --> 12:33.366 all these different tools and is gonna be really excited about them and whatever, 12:33.366 --> 12:35.288 a lot of people were just like, 12:34.271 --> 12:36.401 Yeah, right. I wanna get my work done. 12:35.288 --> 12:36.869 I have work to get done. 12:36.401 --> 12:37.429 Yeah. Mm-hmm. 12:37.809 --> 12:40.250 Either tell me how to use it and I'll use it, 12:40.250 --> 12:44.543 or don't tell me and I won't try any of this stuff because you told me I need 12:44.543 --> 12:46.274 to get this thing done by next week or whatever, 12:46.274 --> 12:47.945 which is like whatever. It's a such. 12:46.678 --> 12:47.969 I love this. You're you're you're playing 12:47.969 --> 12:49.640 into my into my next set of questions, 12:49.640 --> 12:53.103 right? We we normally sort of open with like what tools you like versus like what 12:53.103 --> 12:54.944 are you seeing in the time teams you observe 12:54.739 --> 12:55.739 Yeah. 12:54.944 --> 12:55.944 and and 12:55.925 --> 12:57.006 and it's and it's interesting, 12:57.006 --> 12:59.758 Rob. I mean, you obviously as a tech leader, 12:59.758 --> 13:01.129 you've got your own team, right, 13:01.129 --> 13:03.841 that you serve and you've got your own observations there. 13:03.841 --> 13:06.804 But I wonder, like, given your role at Circle CI, 13:06.804 --> 13:08.705 like you can kind of see what we're all doing. 13:08.705 --> 13:10.026 what insight does that bring you? 13:10.026 --> 13:15.921 Yeah. And we do. ~ we we have this report that we put out. 13:15.921 --> 13:17.251 We used to put out every year. 13:17.251 --> 13:18.251 I think 13:17.603 --> 13:18.603 ~ 13:17.932 --> 13:19.613 we still do like the annual version of it, 13:19.613 --> 13:21.334 right? It's called the state of software delivery report, 13:21.334 --> 13:23.966 where we started from, you know, 13:23.966 --> 13:27.865 pre-AI era. I think we're in year seven or eight or something. 13:27.865 --> 13:31.362 ~ pre-AI era, you know, Dora metrics, 13:31.362 --> 13:34.134 what can we see about how you know PR throughput and what 13:32.195 --> 13:33.996 Yeah. Yeah. 13:34.134 --> 13:36.005 like what's actually flowing through the system? 13:36.389 --> 13:37.389 Mm-hmm. 13:36.389 --> 13:37.389 So a lot, 13:36.749 --> 13:39.842 a lot of what's out there is survey-driven, 13:39.842 --> 13:42.074 self-reported, which is valuable for sure. 13:42.074 --> 13:46.948 And developers' perception of what's happening is often more valuable than what's 13:46.948 --> 13:48.219 actually happening, right? In some cases, 13:48.219 --> 13:52.392 as you're trying to diagnose, but people don't have the best memories, 13:52.392 --> 13:53.903 right? I can't remember what I had for lunch. 13:53.123 --> 13:55.029 Right. Right. 13:53.903 --> 13:55.164 So, like, can I remember how many 13:55.164 --> 13:56.906 PRs I got through last week? Like, 13:56.906 --> 13:57.906 not necessarily, 13:57.039 --> 13:58.039 Mm-hmm. 13:57.847 --> 13:59.348 but the systems will tell us, right? 13:59.348 --> 14:02.891 So we do this large analysis across our entire customer base. 14:02.891 --> 14:05.623 And ~ and we've actually gone to 14:05.829 --> 14:09.911 putting out quarterly updates now because the state of software delivery 14:09.911 --> 14:12.242 is changing so quickly that 14:11.909 --> 14:12.909 Yeah. 14:12.242 --> 14:14.394 comparing year over year numbers is just like, 14:14.394 --> 14:15.954 wait, I what was happening last year? 14:15.954 --> 14:17.575 I don't even remember, right? So, 14:17.575 --> 14:20.626 one of the really interesting things that we saw earlier this year 14:20.626 --> 14:22.198 and we've been seeing continue. 14:22.198 --> 14:25.119 I mean, unsurprisingly we see significant spreads, 14:25.119 --> 14:30.502 right? Like the difference between median and P95 is shocking. 14:30.502 --> 14:33.914 And ~ in particular something that shows up in 14:34.233 --> 14:37.075 In CI numbers, that's been quite telling is 14:37.636 --> 14:44.400 the difference between increased rate of branch builds, 14:44.400 --> 14:46.762 which I'll explain why this matters in a second, 14:46.762 --> 14:49.764 and the rate of main builds, right? 14:49.752 --> 14:50.752 Okay. Yep. 14:49.764 --> 14:50.764 So 14:51.386 --> 14:55.368 if I'm I'm we're seeing significant increase in CI consumption, 14:55.368 --> 14:58.471 leading companies. So we actually came up with a new metric which 14:58.471 --> 14:59.992 we call merge efficiency. 15:01.028 --> 15:04.710 How many builds on a branch does it take before you can merge something to main? 15:04.710 --> 15:10.914 And the really successful companies are seeing the same sort 15:10.914 --> 15:13.935 of merge rate or higher as they get better, 15:13.935 --> 15:17.887 right? And everything is increasing because they're applying AI, 15:17.887 --> 15:20.249 they're creating more PRs, they're creating them faster, 15:20.249 --> 15:22.240 they're creating even bigger ones, 15:22.240 --> 15:25.482 yet still faster and still merging faster. 15:25.482 --> 15:29.644 The sort of median company is creating a lot. 15:29.796 --> 15:34.141 more builds on a branch because their work is broken and it fails and 15:34.141 --> 15:38.185 it fails and it fails and it fails and then they finally get it merged into main 15:38.465 --> 15:41.058 or whatever your default branch is that we assume goes to production. 15:40.496 --> 15:41.496 Sure. 15:41.423 --> 15:42.423 Yep. 15:43.126 --> 15:46.697 And your final productivity, you're doing a lot more work. 15:46.697 --> 15:49.618 You're using more tokens and more energy and whatever. 15:49.618 --> 15:52.819 But then the final output is kind of the same. 15:52.819 --> 15:53.819 And so that's 15:53.296 --> 15:54.296 Mm. 15:53.779 --> 15:56.950 a really interesting gap. We've been trying to help people solve the for that gap, 15:56.950 --> 15:59.991 right? Like, how do you get more confidence earlier? 15:59.991 --> 16:03.332 How do you steer your agent? Assuming you're using an agent to build 16:03.332 --> 16:05.883 or s some version of that, right? 16:05.883 --> 16:10.744 How do you steer your tools so that you're getting good outcomes earlier? 16:11.248 --> 16:12.248 Interesting. 16:11.290 --> 16:12.290 With less 16:11.800 --> 16:13.461 effort, less token spend, et cetera, 16:13.461 --> 16:17.022 so that your your merge ratio goes back up, 16:17.022 --> 16:21.663 right? It takes me one or two builds to know the thing is good to then merge it, 16:21.663 --> 16:24.444 right? Versus like I'm creating all of this, 16:24.444 --> 16:27.094 you know, it's like all heat, but it's all friction. 16:27.094 --> 16:29.975 So I, you know, all I've or like I've dissipated all of that energy. 16:29.975 --> 16:31.516 This is where my physics background, 16:30.404 --> 16:32.470 Yeah. Yeah. 16:31.516 --> 16:32.866 like I always like to use physics analogies. 16:32.886 --> 16:35.567 But like we had all this energy, 16:35.567 --> 16:36.917 and yes, energy was conserved, 16:36.917 --> 16:39.488 but it's been dissipated into the environment as heat instead 16:38.830 --> 16:39.830 Yeah. 16:39.488 --> 16:40.488 of. 16:40.091 --> 16:41.824 Light. Yeah. 16:40.347 --> 16:41.878 Turning into actual, like, yeah, 16:41.878 --> 16:43.599 light and output output to our customers. 16:43.599 --> 16:48.222 Right. So ~ it's super interesting to see. 16:48.222 --> 16:52.865 And I would say the spreads are getting bigger, 16:52.865 --> 16:57.658 not smaller. Right. And so if you if you imagine, 16:57.658 --> 17:02.041 and lots of people have used exp expressions like AI as an amplifier, 17:02.041 --> 17:03.902 right? Like if you're if you 17:02.810 --> 17:03.832 Absolutely. Yeah. 17:03.902 --> 17:05.443 have brokenness in your organization, 17:05.443 --> 17:09.224 you just see them brokenness in more stark contrast than you 17:08.900 --> 17:09.900 Yep. 17:09.224 --> 17:10.224 did before. 17:09.887 --> 17:12.421 If you made messes before, you're gonna make them faster now. 17:10.604 --> 17:13.345 And right. 17:12.421 --> 17:13.421 Yep. 17:13.405 --> 17:15.675 Really big ones. And you're still not going to get stuff out. 17:15.675 --> 17:17.646 So, like, if you were really good, 17:17.646 --> 17:21.067 right? If we're all moving along the track and like some of us are slightly ahead 17:21.067 --> 17:25.468 or whatever, the really good folks are going way faster. 17:25.468 --> 17:29.809 And the people who were struggling are still struggling. 17:29.809 --> 17:35.301 So their their velocity hasn't really changed if you're talking about like actual 17:35.301 --> 17:38.442 impact to customers, right? Like we could debate story points all day long. 17:38.442 --> 17:39.442 But like 17:39.386 --> 17:42.307 the actual impact to customers that you're delivering isn't really shifting 17:42.307 --> 17:46.410 and the folks that really have it dialed or realized how they needed 17:46.410 --> 17:48.751 to get things dialed really quickly in order to be able to take advantage 17:48.751 --> 17:52.273 of the tools, those folks are breaking away from the pack. 17:52.273 --> 17:55.034 So the spread is is even bigger. 17:55.480 --> 17:58.791 That's really interesting. I I wondered I if you would say something like, 17:58.791 --> 18:01.172 people are like starting stuff and starting stuff and starting stuff 18:01.172 --> 18:03.093 and it's never finishing. That that was my own, 18:03.093 --> 18:04.284 especially early on, 18:03.864 --> 18:04.864 Mm. 18:04.284 --> 18:05.824 you know, experience was like, 18:05.824 --> 18:09.076 you're so good at greenfield, I'm just gonna greenfield all the greenfield. 18:09.076 --> 18:10.076 And and it's like, wait, wait, 18:10.053 --> 18:11.126 wait, I'm not shipping anything. 18:11.126 --> 18:12.847 But but I 18:12.249 --> 18:13.249 Yeah. 18:12.887 --> 18:13.887 I love what you're talking about. 18:13.784 --> 18:19.990 So so to your point, the the better organizations ~ sort of have less time 18:19.990 --> 18:23.051 on branch then and and fewer builds before it actually gets merged 18:23.051 --> 18:24.652 and and actually gets shipped and deployed. 18:25.486 --> 18:26.486 Yeah. 18:25.498 --> 18:26.498 and 18:25.769 --> 18:29.263 one assumes released versus ~ there's there's this other sort 18:29.263 --> 18:32.137 of set of organizations which you know that they're spending more time 18:32.137 --> 18:36.312 on that feature branch. ~ and so were they doing that before 18:36.312 --> 18:39.656 and now they're spending even longer or did the other ones get faster 18:39.656 --> 18:41.238 or both or help me with that part. 18:42.085 --> 18:46.387 Yeah, I think we've seen an increase in the kind of 18:47.067 --> 18:50.128 not great branch build. 18:50.128 --> 18:53.647 Although there's a second like thinking of my own experience, 18:53.647 --> 18:56.691 there's a secondary factor there that I don't know if we've been able to isolate. 18:56.691 --> 19:00.132 So, but we're seeing more total builds, 19:00.132 --> 19:05.574 right? Take that more total builds in a in a leading organization, 19:05.574 --> 19:08.526 and more of those will be on main trunk, 19:08.526 --> 19:09.646 whatever you happen to call that. 19:11.150 --> 19:14.641 take a struggling organization and it'll be more just piling up on branches. 19:14.641 --> 19:16.612 Right. One of the interesting, 19:15.322 --> 19:16.322 Yeah, yeah. 19:16.612 --> 19:20.934 like, because I personally ~ am working had been working on something, 19:20.934 --> 19:23.825 but I was like, you know, I'm I I'm in a bit of a tough spot. 19:23.825 --> 19:26.687 I spend a lot of time in meetings and then a bit like a big week might come 19:26.687 --> 19:27.787 up where I don't have a lot of time. 19:27.787 --> 19:29.838 And so maybe I'm not a great example, 19:29.838 --> 19:33.990 but I know this is real for folks where I I get something to complete. 19:33.990 --> 19:36.081 And then the next day I'm like, 19:36.081 --> 19:39.312 yeah, maybe I'll maybe I'll actually get it put up in a PR today or whatever. 19:39.684 --> 19:43.125 But let me just rebase because so much activity has happened. 19:42.874 --> 19:43.874 Mm-hmm. Mm-hmm. 19:43.125 --> 19:44.125 And it's not just 19:43.936 --> 19:47.007 a git rebase anymore because you can do so much work. 19:47.007 --> 19:48.007 It's like, 19:47.396 --> 19:48.396 Yeah. 19:47.888 --> 19:53.700 cool, take this intent and reapply it to main so that it's not a massive diversion. 19:53.700 --> 19:56.841 I'm looking for, you know, I'm looking for merge conflicts that aren't just this 19:53.924 --> 19:57.452 Right. Right. Yep. 19:56.841 --> 19:58.232 line was changed in two places, 19:57.452 --> 19:59.116 Right. 19:58.232 --> 19:59.252 but this 19:59.252 --> 20:00.453 no longer makes sense together, 20:00.453 --> 20:03.734 or another capability was implemented that we were leveraging. 20:03.734 --> 20:05.355 So we have to adjust our implementation, 20:05.355 --> 20:07.406 right? And so sometimes my merge 20:07.576 --> 20:12.858 Or my my branch build count is high because every day I just rebased 20:13.018 --> 20:15.950 in hopes that maybe that day I would get the thing done and actually be able 20:15.950 --> 20:17.160 to put it up. Or maybe I put it up 20:16.836 --> 20:17.836 Right. 20:17.160 --> 20:19.821 in a PR and it sat there and a different thing got merged. 20:19.821 --> 20:21.942 And so like that, that kind 20:21.594 --> 20:22.594 Yeah. 20:21.942 --> 20:25.664 of ~ like merge conflict. 20:24.080 --> 20:26.101 But when you rebase, you found more changes 20:26.101 --> 20:28.432 than you would have a year ago. 20:28.432 --> 20:32.194 ~ a year ago there would have been like a couple of new files, 20:32.194 --> 20:34.315 but you know, there wouldn't have been a lot of wholesale change, 20:34.315 --> 20:38.117 but now it's like nope, it moves I'm this is the assumption I'm making based 20:37.903 --> 20:39.754 Yeah, yeah, exactly. Like 20:38.117 --> 20:39.608 on what you're saying. Yeah. 20:39.948 --> 20:40.948 Yeah. 20:40.995 --> 20:42.855 and this is on I won't call it greenfield, 20:42.855 --> 20:44.636 but something we're building that's pretty new. 20:44.636 --> 20:45.777 And so 20:45.136 --> 20:46.136 Yeah. 20:46.077 --> 20:50.219 we might one morning, like every every stand up every morning is like 20:50.219 --> 20:53.391 a pretty deep discussion about where's the market and what do we want 20:53.391 --> 20:55.131 to do today and what did we learn from yesterday. 20:55.131 --> 20:57.323 Not like still working on this ticket. 20:57.323 --> 20:59.354 Like no one's still working on that ticket. 20:57.838 --> 21:00.438 Yeah. Right. Yeah. 20:59.354 --> 21:00.354 You know what I mean? It's like 21:01.208 --> 21:02.208 I mean, other than me, 21:01.898 --> 21:03.199 because I got distracted by meetings or whatever. 21:03.199 --> 21:04.380 But like it's like you you know, 21:04.380 --> 21:05.619 those were stand-ups back in the day. 21:05.619 --> 21:07.141 It's like, well, it's day four. 21:06.842 --> 21:07.842 Yeah. Yeah. 21:07.141 --> 21:08.141 I'm still working on 21:07.782 --> 21:09.181 this ticket. Maybe I don't know what I'm doing. 21:09.181 --> 21:10.473 Could someone help me? Right. Like, 21:10.262 --> 21:11.262 yeah yeah yeah. 21:10.873 --> 21:12.764 and now it's like, well, I did those seven tickets, 21:12.764 --> 21:15.736 right? And like, you know, Tuesday we have a chat, 21:15.736 --> 21:17.276 we agree on something, and by Wednesday, 21:17.276 --> 21:20.078 we fundamentally re-architected some part of the system. 21:19.876 --> 21:20.876 Yeah, right. 21:20.078 --> 21:21.078 And 21:20.739 --> 21:22.360 then that thing that we were stringing along, 21:22.360 --> 21:23.360 we're like, you know, you're like, 21:23.210 --> 21:26.362 which of these two major re-architectures can we land first? 21:26.362 --> 21:28.983 And then we'll rebuild the other one on top of that, 21:28.983 --> 21:29.983 right? And 21:30.436 --> 21:33.818 That would be so frustrating five years ago. 21:34.053 --> 21:35.053 Yeah. 21:34.439 --> 21:36.788 And now I'm like, hey, buddy, because I, 21:36.788 --> 21:40.083 you know, I totally have turned my agents into friends now. 21:40.083 --> 21:42.665 Hey, buddy, hate to tell you, but you know, 21:42.665 --> 21:44.946 Maine has totally changed. Can you just 21:44.324 --> 21:45.324 Ha ha 21:44.946 --> 21:48.789 reapply this? And Buddy, who might be codex or Claude or whatever that day, 21:48.789 --> 21:52.211 is like, no problem. That's you you pay me to do that. 21:50.446 --> 21:53.708 Yeah, yeah. Ego less, yeah. 21:52.211 --> 21:53.552 So I'm happy to do it, right? I'm 21:53.793 --> 21:57.265 I'm serving the goal of whichever foundation shop is getting all 21:57.265 --> 21:58.746 the token money right now. So 21:58.384 --> 21:59.384 Yeah, yeah, yeah. 21:59.322 --> 22:00.663 Yeah, I'll have at it. And so, 22:00.663 --> 22:05.115 like that you yes, exactly. You're making like 22:05.355 --> 22:09.818 big thematic changes and then reapplying a different goal on top 22:09.818 --> 22:14.160 of those thematic changes as opposed to like the little angle brackets 22:14.160 --> 22:15.541 to the right and left that say, 22:15.541 --> 22:17.562 someone changed this function call, 22:17.562 --> 22:19.537 they added this parameter, and you added this parameter. 22:18.970 --> 22:21.750 Right. Right. Yeah. 22:19.537 --> 22:21.344 You're like, cool, I'll put both parameters 22:22.025 --> 22:25.056 and find the two call sites, and now I can merge it, 22:23.886 --> 22:25.568 Right. Right. 22:25.056 --> 22:26.056 right? Because 22:25.597 --> 22:27.828 that's the amount that someone else could have done. 22:28.143 --> 22:29.143 In that amount of time. 22:28.812 --> 22:29.812 in 22:28.972 --> 22:30.302 the period where I was, you know, 22:30.302 --> 22:31.604 where we were sort of conflicting. 22:31.604 --> 22:32.604 And so I I think 22:31.844 --> 22:32.844 That's interesting. 22:32.495 --> 22:38.580 that that that whole area of of merging intent 22:37.969 --> 22:38.969 There's like a hygiene 22:38.829 --> 22:43.752 now around ~ you need to get it into Maine faster almost, 22:43.752 --> 22:46.813 ~ somehow. You so working even smaller, 22:46.813 --> 22:49.955 even more atomically i is the solution that pops into my head. 22:49.955 --> 22:50.955 I wonder what you think though. 22:50.935 --> 22:51.935 Yeah. 22:51.854 --> 22:52.854 Yeah, 22:52.224 --> 22:55.886 I think I think the reason we're driven is 22:54.756 --> 22:55.798 Well, yeah. 22:55.886 --> 22:59.048 that we can drive bigger changes in a small amount of time, 22:57.902 --> 22:59.715 Yeah, yeah, yeah, yeah, yeah, right, 22:59.048 --> 23:04.651 right? Like if I can write a thousand useful lines of code, 22:59.715 --> 23:02.080 right. So that's not gonna happen. 23:02.080 --> 23:05.367 ~ Right. 23:04.651 --> 23:07.773 like good, productive, useful lines of code in an hour or two, 23:05.367 --> 23:08.002 Right. Right. 23:08.293 --> 23:10.594 do you want me to ship you 10 of those at a time? 23:10.594 --> 23:13.006 Right? Like, hey Kyle, here's another PR. 23:13.006 --> 23:15.147 Like, what you know, here's and here's one, 23:15.147 --> 23:17.488 and here's one, and here's what like that wouldn't actually make sense, 23:17.488 --> 23:20.402 right? Like, yeah. 23:18.160 --> 23:20.461 Well, okay, I've had two guests say that 23:20.461 --> 23:25.783 they do that on purpose. One said that he's basically always telling Claude for him, 23:25.783 --> 23:31.265 ~ that like we what we do in this is going to be ten lines of code or fewer. 23:31.265 --> 23:33.780 That'll be like a constraint that he'll give it. 23:33.780 --> 23:34.780 and 23:34.117 --> 23:35.117 Mm-hmm. 23:34.687 --> 23:38.198 and another guest ~ said for s for similar reasons, 23:38.198 --> 23:39.509 but it's a different approach, 23:39.509 --> 23:44.091 was like, Okay, tell me what pull requests we're going to do. 23:44.091 --> 23:46.752 So like really thinking it ahead of time in terms of 23:46.414 --> 23:47.414 Yeah. 23:47.172 --> 23:50.503 We're not just gonna this isn't just gonna be this giant ball that we're we're gonna 23:50.503 --> 23:53.246 we're gonna plan this out. ~ but I'm with you. 23:53.246 --> 23:57.869 I think y one is sort of like trying to avoid and pretend it's not happening 23:57.869 --> 23:59.880 and and the other is sort of like trying to well, 23:59.880 --> 24:01.381 it is happening, but let's deal with it, 24:01.381 --> 24:02.762 you know, let's let's plan for this. 24:02.244 --> 24:03.244 Yeah. 24:03.242 --> 24:07.425 ~ but you're right though. Like the the idea that we're all gonna be as atomic 24:07.425 --> 24:10.247 as we used to be, yeah, that was a naive point on my on my point, 24:10.247 --> 24:11.478 I think. Yeah. Sorry, go ahead. 24:12.612 --> 24:14.774 I mean, we're all figuring it out. 24:14.774 --> 24:15.774 I think a little bit 24:15.172 --> 24:16.172 Yeah. 24:15.534 --> 24:18.597 depends on your on your code base, 24:18.597 --> 24:20.008 the maturity of that code base, 24:20.008 --> 24:21.008 the maturity 24:20.302 --> 24:21.302 Yeah. 24:20.619 --> 24:23.151 of your business. Although I would argue that like it's 24:23.151 --> 24:26.037 not just software development that's been threatened by AI or changed by AI. 24:25.998 --> 24:28.021 Yeah. Yep. 24:26.037 --> 24:27.504 Like everyone's business model 24:28.565 --> 24:30.407 is immature all of a sudden, right? 24:30.407 --> 24:32.509 Because it's like, what is my business model in this world? 24:32.509 --> 24:35.611 And so like rate of change, 24:35.611 --> 24:39.014 ad adaptability, all of those things are more important. 24:39.014 --> 24:41.096 and 24:42.725 --> 24:46.017 That means, I mean, 24:46.017 --> 24:50.230 one of the things that that means to me is we have to not just be throwing stuff 24:50.230 --> 24:52.952 out like here's an enormous PR and here's an enormous PR and it seems to work. 24:52.952 --> 24:58.175 So let's just keep going. Because we do have to be capable of changing and adapting, 24:58.175 --> 25:02.778 right? Which is one of the reasons we put a lot of energy into our design, 25:02.778 --> 25:04.089 into small incremental change, 25:04.089 --> 25:06.240 into making sure that other people understood it. 25:06.240 --> 25:08.501 So then if you go kind of first principles, 25:08.501 --> 25:10.853 like those were our goals, right? 25:10.853 --> 25:12.314 But a lot of those were aimed at. 25:12.644 --> 25:17.176 The human developer, right? Like a book that I read three years ago, 25:17.176 --> 25:18.546 four years ago, I can't remember, 25:18.546 --> 25:20.277 called The Art of Readable Code. 25:20.277 --> 25:21.277 I think that's the name of it. 25:21.124 --> 25:22.124 Mm, mm. 25:22.048 --> 25:27.050 Right. Super interesting. I I was super passionate about readable code 25:27.034 --> 25:28.897 Sure. Yeah. 25:27.270 --> 25:28.931 for all the reasons that I 25:28.931 --> 25:32.232 I need humans to be able to make safe future changes. 25:32.232 --> 25:35.714 Right. But I think, I'm not sure yet, 25:35.714 --> 25:36.754 we're trying to figure it out. 25:36.754 --> 25:40.866 Readable to a human and readable to an agent are different. 25:40.856 --> 25:41.856 gonna 25:40.997 --> 25:42.234 be different thing. Yep. 25:42.508 --> 25:45.730 And so the goal was safe, 25:45.730 --> 25:48.182 continued, safe future changes, 25:48.182 --> 25:50.013 therefore adaptability and velocity, 25:50.013 --> 25:55.137 right? And if I assume that my tool is me prompting an agent, 25:55.137 --> 25:59.280 then the things that I looked at originally when I saw agents write code 25:59.280 --> 26:02.262 or LLMs or whatever, we weren't really like agentic or whatever yet, 26:02.262 --> 26:05.785 I was like, that's terrible. And I would delete all their comments, 26:05.785 --> 26:08.086 you know, this doesn't look like the way that I want it, 26:08.086 --> 26:09.287 whatever. And I'm like, but if that 26:08.216 --> 26:09.278 Yeah, yeah. 26:09.287 --> 26:10.287 makes the agent 26:10.482 --> 26:11.482 Right. Right. 26:10.948 --> 26:11.948 Work better 26:11.539 --> 26:15.893 on the next pass and is gonna in like lead to higher chances of success. 26:15.893 --> 26:20.807 Why do I care? I'm not I'm not the primary sort of like reader and writer 26:20.807 --> 26:22.279 of the of the code at this point. 26:22.279 --> 26:25.241 And I think we're gonna continue to to shift and sort of like 26:25.762 --> 26:29.265 an emergent property that we will discover of software 26:29.906 --> 26:36.412 is what is software that is highly maintainable. 26:37.473 --> 26:39.054 If maintenance is the right tool, 26:39.054 --> 26:41.605 like with you know, can we componentize and then just throw things 26:41.605 --> 26:42.626 out and replace them at you know, 26:42.626 --> 26:45.997 whatever that is? What does it look like for long-term, 26:44.059 --> 26:47.328 Phoenix architecture right. Right. 26:45.997 --> 26:47.488 ongoing, sustainable, 26:47.488 --> 26:49.379 high-quality development driven by agents? 26:49.379 --> 26:50.379 And it'll 26:49.679 --> 26:50.679 Yeah. 26:50.050 --> 26:52.061 be everything from, yeah, like design, 26:52.061 --> 26:54.982 language selection, tooling choices, 26:54.982 --> 26:57.744 documentation, repo structure, 26:57.744 --> 26:59.985 like I'm sure we're gonna have mono repo, 26:58.350 --> 27:01.159 Yeah. Yeah, yeah, yeah, yeah, yeah, 26:59.985 --> 27:01.776 multi-repo debates again. You know, 27:01.159 --> 27:02.242 yeah. Yeah. 27:01.776 --> 27:02.776 like 27:02.886 --> 27:05.938 how do we enable these tools? And I think we're we're still discovering that 27:05.938 --> 27:06.938 and we'll 27:06.500 --> 27:09.971 We'll see based on who's most successful, 27:09.971 --> 27:11.256 you know. 27:11.100 --> 27:12.100 Okay, 27:11.490 --> 27:14.542 so Roll, I I don't know that you'll have an answer to this, 27:14.542 --> 27:15.923 but it occurs to me that you might. 27:15.923 --> 27:20.925 ~ can you do you have any sort of like economic indicators around like growth 27:20.925 --> 27:22.606 of hiring or shrinking of hiring? 27:22.606 --> 27:25.478 My my own anecdot, which, you know, 27:25.478 --> 27:26.478 is is 27:25.488 --> 27:26.488 Mm-hmm. 27:26.128 --> 27:28.980 never much to go on, is that most of my guests have reported, 27:28.980 --> 27:31.991 like, I we actually haven't hired anybody since AI, 27:31.991 --> 27:35.183 but nobody has talked too much about having laid anybody off. 27:35.183 --> 27:37.034 Now, of course, there are high profile, 27:37.034 --> 27:38.615 you know, CEOs of Bangor Corps, 27:38.615 --> 27:40.316 and you can kind of assume 27:40.526 --> 27:41.526 l 27:40.806 --> 27:42.418 that they would have lots of reasons to do that. 27:42.418 --> 27:46.962 ~ but you have non anek data. You you have actual data, 27:46.962 --> 27:49.114 maybe? Like can you see much there? 27:48.130 --> 27:51.322 Yeah. ~ 27:51.743 --> 27:57.446 I would say what we look at is I well, 27:57.446 --> 28:02.330 I mean, we we have we have both and we don't always know like when we look 28:02.330 --> 28:03.741 at our data, we have to do some like, 28:03.741 --> 28:06.313 okay, but who's that company and what do we know about them sort of thing, 28:06.313 --> 28:11.566 right? Like so you have organizations that are raising enormous amounts 28:11.566 --> 28:16.400 of capital and you know, growing on like very 28:16.963 --> 28:19.314 investment driven growth and they are hiring. 28:19.314 --> 28:24.198 And then we see a lot of our base staying the course, 28:24.198 --> 28:26.879 but growing their delivery velocity, 28:26.879 --> 28:30.317 right? Like they're again without judging whether they're getting customer outcomes, 28:30.317 --> 28:31.753 we're seeing more builds, more throughput, 28:31.753 --> 28:32.753 more creation. 28:32.208 --> 28:33.208 Yep. Yep. 28:33.183 --> 28:39.608 And and so a lot of what we think about is in the way that you calculate 28:39.608 --> 28:41.699 net dollar retention, I think is mostly an engineering 28:41.977 --> 28:43.897 like audience, but so maybe I'll explain it for a second. 28:43.897 --> 28:47.341 But like you take a cohort and say from that cohort in the future, 28:47.341 --> 28:51.865 what is our revenue based on the s the people that were here at this time 28:51.865 --> 28:53.686 a year ago, right? As like net dollar retention. 28:52.922 --> 28:53.922 Uh-huh. 28:53.686 --> 28:59.551 ~ and so you can think of like net pipeline retention as a number almost like 28:59.551 --> 29:04.905 are we growing from the same number of like projects and the same number 29:04.656 --> 29:07.097 Mm. Mm. 29:04.905 --> 29:06.996 of users sort of thing. Like so we can see 29:07.317 --> 29:09.518 some health in our business independent of the other things we're doing. 29:09.518 --> 29:11.257 Like we're signing new customers and all that kind of stuff, 29:11.257 --> 29:12.257 right? 29:12.098 --> 29:15.112 And we definitely see growth within a cohort, 29:15.112 --> 29:17.975 despite not a ton of growth. I mean, 29:17.975 --> 29:19.247 again, it's different company by company, 29:19.247 --> 29:22.060 but like a lot of organizations are pretty stable. 29:20.644 --> 29:22.677 Right. Right. 29:22.060 --> 29:23.060 Although 29:23.002 --> 29:25.064 we we had a really interesting as you know, 29:25.064 --> 29:26.656 you can only trust your data so much. 29:26.656 --> 29:28.428 What we see is who's committing, 29:28.428 --> 29:31.108 right? So we see more. 29:33.444 --> 29:35.164 The GitHub IDs, let's call it, 29:35.164 --> 29:37.545 associated with a project over a period. 29:35.182 --> 29:38.300 Right, right. Mm-hmm. 29:37.545 --> 29:38.906 Right. And so we're like, cool, 29:38.300 --> 29:40.898 More humans, yeah. 29:38.906 --> 29:40.686 they hired engineers to do this. 29:41.146 --> 29:44.387 And I was in a conversation recently where we said, 29:44.387 --> 29:46.308 ~ you know, we were talking to a customer. 29:46.308 --> 29:47.888 We we see you've done a lot of hiring. 29:47.888 --> 29:49.149 Like, are you are you doing a lot of hiring? 29:49.149 --> 29:50.829 And they were like, No, we haven't. 29:50.829 --> 29:53.830 And we're like, that's odd. Like, 29:52.058 --> 29:55.318 Hmm. Your count of GitHubes went up? 29:53.830 --> 29:55.651 and it what what we learned 29:56.431 --> 30:00.182 was everyone in the organization is now committing. 30:01.284 --> 30:03.655 So they're designers and their PMs and everybody. 30:03.655 --> 30:05.645 So they hadn't hired net new people, 30:05.645 --> 30:10.817 but they had fundamentally changed the way they work such that to us, 30:10.817 --> 30:12.598 we're like, there's net new contributors, 30:12.598 --> 30:16.949 and our simple brains are like there's more engineers in the organization. 30:16.949 --> 30:18.220 But what they explained 30:17.487 --> 30:18.487 No, it's not 30:18.220 --> 30:20.210 to us was actually happening, they had hired some people, 30:20.210 --> 30:23.752 and then they had more people contributing to the project, 30:23.376 --> 30:24.708 That's amazing. And you hear about 30:23.752 --> 30:24.752 right? 30:24.708 --> 30:25.960 that, you know, and you wonder, 30:25.960 --> 30:29.596 well, you know, the anthropic product manager goes on a podcast 30:29.596 --> 30:30.908 and says that's happening and you're like, 30:30.908 --> 30:33.442 cool, but but but you're to your point, 30:33.442 --> 30:35.655 you're seeing actual proof of that beyond just, 30:35.257 --> 30:36.257 Yeah. 30:35.896 --> 30:36.896 yeah. 30:36.398 --> 30:41.269 Right. And it's not like necessarily the designer showed up and launched like 30:41.269 --> 30:44.541 50 PRs that week right next to the engineer who put up 50 PRs 30:44.541 --> 30:45.961 or whatever the numbers are like these days. 30:45.961 --> 30:49.710 But you know, we say, we we see people and of course, 30:49.710 --> 30:52.024 you know, if you model it, you'd this is 30:51.512 --> 30:52.512 Interesting. 30:52.024 --> 30:53.694 a new engineer, therefore they've only, 30:53.694 --> 30:55.865 you know, they made a small amount of contribution or whatever. 30:55.865 --> 30:59.557 But it's like at some point, we probably could have done this a while ago. 30:59.557 --> 31:02.708 The designer is like, I mean, I don't know if you remember the days, 31:02.708 --> 31:03.708 probably. 31:03.830 --> 31:07.221 they would like make a new Photoshop image, 31:07.221 --> 31:09.181 right? With where the color and they want you to 31:08.826 --> 31:09.826 Mm-hmm. 31:09.181 --> 31:11.942 like get the eyedropper and sample the color and then change the color 31:11.492 --> 31:15.594 Yeah, yeah, yeah, yeah. Yeah. Yeah. 31:11.942 --> 31:15.023 on the button. I'm like, could you please just edit the CSS? 31:15.023 --> 31:16.023 Like why why are 31:15.703 --> 31:18.424 we doing this? Right? Like this whole process is ridiculous. 31:18.192 --> 31:22.505 Stop making me slice up PSDs. It is not ninety-eight anymore. 31:18.424 --> 31:21.485 Right. my God. Yeah, exactly. 31:21.485 --> 31:24.726 Exactly. Right. I mean like slice 31:24.726 --> 31:27.206 it up into little images and try to place them on the screen and whatever. 31:26.969 --> 31:27.969 Yeah. 31:27.206 --> 31:28.206 And 31:27.446 --> 31:31.228 and so, you know, we we absolutely have that internally as well. 31:31.228 --> 31:32.888 Just like, just do it. 31:32.452 --> 31:33.452 Yeah. 31:33.230 --> 31:34.600 Right. And and 31:33.321 --> 31:34.551 Cool. Cool. 31:35.791 --> 31:36.841 yeah, I mean, we have the tools, 31:36.841 --> 31:38.552 we have the capacity. So we're we're 31:38.096 --> 31:39.096 Amazing. 31:38.552 --> 31:39.942 I mean, that's a little off of the hiring. 31:39.942 --> 31:42.743 We are seeing what you see on the news, 31:42.743 --> 31:46.105 you know, huge organizations thinning 31:43.236 --> 31:45.749 Yeah, yeah. Yeah. 31:46.105 --> 31:49.606 out, sometimes because they truly believe AI is shifting things, 31:49.606 --> 31:52.517 sometimes because they're trying to come up with a reason to justify what they're 31:52.517 --> 31:54.808 doing. And then we're seeing, you know, 31:53.304 --> 31:55.637 Zero interest rates over sure. 31:54.808 --> 31:55.808 yeah. 31:55.637 --> 31:57.310 I mean, yeah, all sorts of reasons. 31:57.310 --> 31:58.552 Yep. Interesting. 31:57.946 --> 31:58.946 company's taking 31:58.701 --> 32:00.011 off and and still hiring you. 32:00.688 --> 32:04.139 Cool. So okay, we ~ that that was really fun. 32:04.139 --> 32:09.110 ~ normally what we shift to ~ right after like what tools you reach 32:09.110 --> 32:11.241 for is what you're seeing in teams that you observe. 32:11.241 --> 32:13.612 And we've kinda had that nicely. 32:11.908 --> 32:12.908 Mm-hmm. 32:13.612 --> 32:14.612 You you've got all 32:14.282 --> 32:16.032 sorts of stories about sort of like all of our teams. 32:16.032 --> 32:19.893 ~ but let's let's zoom in a little bit and to stu and into your 32:19.893 --> 32:21.284 own observations a little bit more, 32:21.284 --> 32:23.914 direct, ~ and less less just out of the data. 32:23.914 --> 32:28.306 Wha how are you what are you seeing in terms of how this well actually honestly 32:28.306 --> 32:29.626 you can answer from either position. 32:29.957 --> 32:30.957 But 32:30.167 --> 32:32.249 we'll we'll go through sort of S DLC left or right. 32:32.249 --> 32:35.742 So how how what do you think you're seeing change in terms of how 32:35.742 --> 32:37.783 we think about requirements? You know, 32:37.783 --> 32:40.671 in in the old days it might have been epics and stories. 32:40.671 --> 32:43.108 ~ for some organizations it might have been B R D's, 32:43.108 --> 32:47.962 PRDs, whatever. are we just having the agents write them now and 32:47.962 --> 32:51.345 the agents read them? Like what what what are you seeing change in how 32:51.345 --> 32:52.866 all of that works here? Yeah. 32:53.540 --> 32:55.087 Yeah. So 32:57.464 --> 33:02.086 See the whole spectrum, ~ e even internally, 33:02.086 --> 33:04.428 right? Even internally at Circle CI, 33:04.428 --> 33:06.809 I think the jury is very much still out. 33:06.809 --> 33:10.051 And part of that might be, you know, 33:10.051 --> 33:12.412 your background and the metaphor that you cling to, 33:12.412 --> 33:15.374 and part of it might be like your particular form of success that you've 33:15.374 --> 33:16.774 had with your tools that you're trying, 33:16.774 --> 33:22.277 right? And and we see, 33:22.277 --> 33:25.459 you know, like spec driven development, 33:25.459 --> 33:26.780 that whole sort of 33:27.674 --> 33:30.526 ~ umbrella term, right, which means different things to different people. 33:30.526 --> 33:33.568 And to your point, like sitting down and hammering something out, 33:33.568 --> 33:36.991 maybe with with Claude or whatever you happen to use, 33:36.991 --> 33:40.553 right? ~ can be really effective as a thought partner. 33:40.553 --> 33:41.553 Like I think just saying, hey, 33:41.444 --> 33:43.255 I want to build a thing, can you write me a spec? 33:43.255 --> 33:45.136 I might as well just say, hey, 33:45.136 --> 33:46.757 build me the thing. Right. 33:46.084 --> 33:47.084 Yeah, yep. 33:47.018 --> 33:48.759 But if I say, hey, okay, cool, 33:48.759 --> 33:49.880 let's go look at this company, 33:49.880 --> 33:51.031 I think they're very successful. 33:51.031 --> 33:52.031 They're one of our competitors. 33:52.021 --> 33:54.183 Like what's different about what they're doing from what we're doing, 33:54.183 --> 33:55.874 right? And what 33:56.036 --> 33:57.557 What is no one considered, right? 33:57.557 --> 34:02.179 Like I I find these tools are really helpful for giving me like my blind spots 34:02.179 --> 34:03.779 or like things I haven't thought of or whatever. 34:03.779 --> 34:08.281 But I don't tend to find that sitting down and like writing a long spec 34:08.281 --> 34:10.839 and trying to line it up and then we're gonna do this and then we're gonna 34:10.839 --> 34:11.839 do this and then we're gonna do this. 34:11.822 --> 34:14.244 I mean, I'm like trying to waterfall my agent at that point, 34:14.244 --> 34:15.244 right? That 34:14.543 --> 34:15.543 Right, right. 34:15.114 --> 34:17.804 doesn't that doesn't really work well for me. 34:17.804 --> 34:23.547 I find that I can open up Claude Code or Codex or like a coding agent, 34:23.547 --> 34:24.547 which 34:25.339 --> 34:27.909 Pretty much the chat bot plus writing code better. 34:27.909 --> 34:31.132 And by the time I've figured out what I want to build, 34:31.132 --> 34:33.413 I've built it. Right. I was like, 34:33.413 --> 34:36.295 if we should do that, then why don't you go ahead and do that and I'll fire 34:36.295 --> 34:38.505 up another subagent or whatever and we'll have a chat about 34:36.954 --> 34:38.357 Yeah. Mm-hmm. 34:38.505 --> 34:39.617 something else. It's like, cool, 34:39.617 --> 34:40.888 why don't okay, why don't you go do that? 34:40.888 --> 34:43.239 let's fix it like this. And so for me, 34:43.239 --> 34:46.501 like I'm often reasoning about the thing. 34:46.501 --> 34:48.141 If it's new, like if it's pretty novel, 34:48.141 --> 34:50.342 I'm reasoning 34:48.570 --> 34:49.748 Yep. Right. 34:50.342 --> 34:52.364 about it as I'm building it. 34:52.792 --> 34:55.263 Instead of reasoning about the document that I'm going to write about 34:55.263 --> 34:57.623 how to build it, which then I'm going to feed to some other agents, 34:57.623 --> 34:58.775 and then I'm going to come back and say, 34:58.775 --> 35:00.335 ~ you all did it wrong, whatever. 35:00.335 --> 35:04.267 And so I I see people having success on both ends. 35:04.267 --> 35:09.819 I lean towards the like, I think my whole career I've thought about software 35:09.819 --> 35:13.031 by opening Vim. I know everyone's going to yell at me and say that 35:13.031 --> 35:16.352 I should use Emacs or like get into the 2000s and get an IDE or whatever, 35:16.352 --> 35:19.684 but like opening and reading the code and being like, 35:19.684 --> 35:20.684 okay, how does this work? 35:21.614 --> 35:24.165 What if I did it like this? Like make little experiments and stuff. 35:24.165 --> 35:26.246 It's just the way that my my brain works. 35:26.246 --> 35:31.508 And so for me, I find there's less ceremony up front, 35:31.508 --> 35:35.910 more like, and we ~ we're a a linear shop. 35:35.910 --> 35:38.151 So we, you know, we use linear tickets and they're like 35:36.858 --> 35:38.295 Mm-hmm. Mm-hmm. 35:39.001 --> 35:41.552 half a sentence. We got distracted while 35:40.590 --> 35:41.590 Right. Right. 35:41.552 --> 35:42.783 we were writing the ticket, but someone's like, 35:42.783 --> 35:44.013 Yeah, I think I was working on something like that, 35:44.013 --> 35:47.645 you know? And and so because our space is moving also, 35:47.645 --> 35:50.646 like in the and the part that I'm particularly working on with the team that 35:50.646 --> 35:51.646 I'm working with. 35:51.644 --> 35:57.426 ~ it's moving so quickly that like let's lay out this plan for the next month 35:57.426 --> 35:59.367 or whatever. Like it just doesn't make a lot of sense. 35:59.367 --> 36:02.729 It's very, you know, we're thinking it through and we're trying things and whatever. 36:02.729 --> 36:08.471 ~ and so I I find that the spec is the functioning code, 36:08.471 --> 36:10.072 you know, like and and 36:09.829 --> 36:10.829 Yeah, yeah. 36:10.642 --> 36:12.743 I find myself quoting the Agile Manifesto a lot. 36:12.743 --> 36:16.064 Like back in 2000 or 2001, I think it was 2001, 36:16.064 --> 36:18.035 right? When that got written, you know, 36:18.035 --> 36:19.035 there's a statement, 36:20.613 --> 36:23.255 I I think it's functional software over documents or something like that, 36:22.916 --> 36:26.059 Yeah. Mm-hmm. Mm-hmm. Working software over documentation. 36:23.255 --> 36:26.667 right? And right, 36:26.059 --> 36:27.059 Yeah. 36:26.667 --> 36:28.848 right. Read it more recently than I have, 36:28.848 --> 36:33.001 or just have a better memory. And so that like we can do that now. 36:33.001 --> 36:35.122 Right? We could do that in minutes. 36:34.458 --> 36:37.804 Yeah. Yeah, yeah. Yeah. Yep. 36:35.122 --> 36:36.323 Like, you want it to work like this here? 36:36.323 --> 36:37.384 Let me just try that real quick. 36:37.384 --> 36:38.384 And okay, 36:37.984 --> 36:40.376 let's try it. that's weird. It does have this weird behavior. 36:40.376 --> 36:41.546 Well, we hadn't considered that. 36:41.546 --> 36:44.388 And to me, like that evolutionary, 36:44.388 --> 36:46.400 like that process of evolving software is yeah, 36:44.941 --> 36:47.362 Immediacy is really addictive. 36:46.400 --> 36:49.952 is really right. But then go 36:47.362 --> 36:49.823 Yeah. Yeah. I I I work like you, 36:49.952 --> 36:50.952 ahead. 36:50.083 --> 36:52.844 for sure. I can relate to to what you're talking about. 36:52.844 --> 36:56.616 Yeah. All the all the sort of spectrum adherents are trying to convince 36:56.616 --> 36:58.147 me to to give up on my immediacy, 36:58.147 --> 37:00.047 but ~ but but that's a whole other show. 36:59.376 --> 37:01.758 Yeah, I do want 37:00.047 --> 37:01.328 Yeah. Okay. 37:02.018 --> 37:04.059 some things that will take a long time. 37:04.059 --> 37:05.059 Cause 37:04.549 --> 37:05.549 Yeah. 37:04.930 --> 37:07.462 I have this weird, like someone showed me a chart of this that 37:07.462 --> 37:09.163 was that really resonated. But basically, 37:09.163 --> 37:13.686 like the feedback isn't immediate because the agent goes and does something. 37:13.686 --> 37:15.247 But it's 37:14.062 --> 37:15.107 Yeah. Yeah. 37:15.247 --> 37:19.029 not far enough off that I can really get engaged with something else, 37:19.029 --> 37:20.480 right? So I'm trying to find, 37:19.246 --> 37:20.331 Right, right, right, right. 37:20.480 --> 37:23.973 okay, can I give you a little and I have built harnesses that 37:22.456 --> 37:23.819 Yeah. Right. 37:23.973 --> 37:27.775 are like goal seeking, like you'll know you're done when you achieve this. 37:27.266 --> 37:28.397 Okay, yep. Mm-hmm. 37:27.775 --> 37:28.775 That 37:28.526 --> 37:32.978 That to me is I mean, it's different from the like I write a like a markdown spec, 37:32.978 --> 37:33.978 but 37:33.306 --> 37:34.306 Mm-hmm. 37:33.669 --> 37:35.800 I will have an agent build up a harness, 37:35.800 --> 37:39.482 right? Which might just be my test suite in a for loop, 37:39.482 --> 37:41.813 right? But like it's a little test driven development. 37:41.813 --> 37:44.096 I think we started calling it harness engineering because we didn't want 37:44.096 --> 37:45.645 to have any more debates about test driven development. 37:45.645 --> 37:48.787 But like saying, Okay, like this is what complete would look like, 37:48.787 --> 37:53.259 right? When you meet these w when this all passes or this all works properly 37:53.259 --> 37:55.220 or whatever. And certainly I'll say things like, 37:55.220 --> 37:56.501 Okay, what else haven't we thought of? 37:56.501 --> 37:57.892 What would break? And then 37:58.128 --> 37:59.628 Claude will this would break and this would break. 37:59.628 --> 38:01.631 I'm like, Could you write tests that would prove that that doesn't work? 38:01.631 --> 38:04.334 Absolutely. Great. Now can you make them work? 38:03.246 --> 38:07.212 Yeah. Yeah. Yeah. 38:04.334 --> 38:06.586 Like that that cycle gets me longer windows. 38:06.586 --> 38:07.586 And when I have longer 38:07.346 --> 38:10.009 windows, I open up another terminal and start another agent kind of thing, 38:09.496 --> 38:11.207 And start another one. Yep. 100%. 38:10.009 --> 38:11.500 right? And it's like, Yeah. Yeah. 38:11.207 --> 38:12.207 Yep. Yep. 38:13.018 --> 38:18.440 Cool. Cool. Thank you. ~ the actual so you've talked in there about 38:18.440 --> 38:20.732 we specification and thinking about planning. 38:20.732 --> 38:23.433 And we talked about a little bit into the actual development. 38:23.433 --> 38:24.804 You even touched on testing as well. 38:24.804 --> 38:28.245 ~ any other changes in either I mean, 38:28.245 --> 38:31.447 obviously the the development itself is the thing which has changed the most. 38:31.447 --> 38:32.447 So anything you 38:32.015 --> 38:33.015 Yeah. 38:32.017 --> 38:33.017 wanna say 38:32.337 --> 38:33.648 about that that isn't obvious. 38:34.229 --> 38:35.954 ~ or or anything on the testing side. 38:35.954 --> 38:38.593 Like what what are you seeing there that that as far 38:38.255 --> 38:39.255 Yeah. 38:38.593 --> 38:40.098 as change to our industry? Yeah. 38:40.815 --> 38:42.966 Yeah, so so probably two things. 38:42.966 --> 38:44.427 I would say on the development, 38:44.427 --> 38:49.460 like I just described designing and interacting with the customer problem 38:49.460 --> 38:51.681 as the process of building. Like why I might as well be built, 38:51.681 --> 38:53.632 right? By the time I figured it out. 38:53.632 --> 38:55.614 What's interesting to me about that, 38:55.614 --> 38:57.114 and I said this a little bit earlier, 38:57.114 --> 38:59.776 is our stand-ups with my team, 38:59.776 --> 39:03.118 which ranges from some other very senior engineers down to like new grads. 39:03.118 --> 39:06.300 We talk every day about 39:06.862 --> 39:08.934 The business, the state of the business, 39:08.934 --> 39:11.797 the state of the market, what matters to our customer. 39:11.797 --> 39:15.380 So that when those folks leave and spend a day working on something, 39:15.380 --> 39:17.202 they can make those decisions as well. 39:17.202 --> 39:22.346 Right. So we're sort of like, and it's probably a little weird because you're 39:22.346 --> 39:24.749 sitting with the CTO who like knows a lot about the business or whatever, 39:24.749 --> 39:28.352 but but we're distributing the product thinking. 39:28.352 --> 39:29.523 And we have a product 39:28.686 --> 39:29.686 Yeah, yeah. 39:29.523 --> 39:31.114 manager who's helping us as well, 39:31.114 --> 39:32.696 does a lot of the like, 39:33.454 --> 39:35.475 I went and looked at the metrics and it looks like this is happening 39:35.475 --> 39:37.187 and whatever and brings that to our discussions. 39:37.187 --> 39:41.161 but it's not like a product manager writing tickets and handing them out to people. 39:41.161 --> 39:42.492 Like if you move to this model, 39:42.492 --> 39:45.074 which is like we just we build and we discover as we go, 39:45.074 --> 39:49.837 the engineers need to really understand the problem we're trying to solve. 39:49.837 --> 39:51.309 We happen to build developer tools, 39:51.309 --> 39:53.821 which probably makes it a little easier to have engineers that understand that. 39:53.584 --> 39:54.584 Right. Right. 39:53.821 --> 39:54.842 But junior 39:54.842 --> 39:59.066 engineers don't necessarily understand the problems of like large engineering 39:57.914 --> 40:00.299 Yeah. That's right. 39:59.066 --> 40:00.487 systems, right? So we really have 40:00.487 --> 40:01.728 to bring those folks along. 40:02.364 --> 40:04.315 ~ and I think it's I think it's a worthwhile investment. 40:04.315 --> 40:06.786 I think that's how you build leaders in your organization anyway. 40:06.786 --> 40:12.470 But I think we've had a split over the last I always quote 25 years because it's 40:12.470 --> 40:13.987 kind of like I hell, I guess I've doing it longer than that. 40:13.987 --> 40:19.163 my god. Anyway, ~ over that time into like really product thinking engineers 40:19.163 --> 40:21.895 and then very sort of like ticket taking engineers, 40:21.895 --> 40:22.895 right? And I think the 40:22.201 --> 40:23.201 Is it? 40:22.595 --> 40:25.497 ticket taking is is gonna be hard to figure out in this world. 40:25.497 --> 40:27.978 So I think how we think about what we're gonna build, 40:27.978 --> 40:30.810 it's not just that we don't necessarily write out a long spec. 40:31.353 --> 40:35.614 It's that that requires an investment in the people participating, 40:35.614 --> 40:37.955 that they really do have a good 40:37.092 --> 40:38.092 Yeah. Yep. 40:37.955 --> 40:40.566 understanding and are comfortable just saying, 40:40.566 --> 40:42.436 like, hey, yo, I actually don't understand. 40:42.436 --> 40:45.117 Like I did this and it doesn't seem like it's working, 40:45.117 --> 40:47.367 or I went and talked to this other engineer and they don't understand what 40:47.367 --> 40:48.828 I'm trying to tell them. I must be missing something, 40:48.828 --> 40:51.418 like just creating that sort of that that free flow. 40:51.418 --> 40:55.440 So then on the on the testing side, 40:55.440 --> 40:56.820 like, and we talked about this. 40:57.592 --> 41:00.723 you know, merge efficiency, like are my am I getting all these failed builds 41:00.723 --> 41:04.425 on my branch, et cetera. ~ it is actually part of what I've been working 41:04.425 --> 41:07.496 on with this team is like trying to shift that earlier, 41:07.496 --> 41:14.239 meaning putting sort of what I would call CI level validation into 41:14.239 --> 41:17.190 the hands of the agent. So as the agent is working, 41:17.190 --> 41:18.951 we're saying, your test suite would pass, 41:18.951 --> 41:21.292 your test suite would pass. Actually your test suite failed right now. 41:21.292 --> 41:22.822 Like here's some feedback about that. 41:22.822 --> 41:27.094 And then trying to build some tools around that that increase the or val like 41:27.468 --> 41:30.550 check and then can help improve the quality of the testing. 41:30.550 --> 41:32.912 Because I think one of the fears that we have 41:31.693 --> 41:32.971 Mm-hmm. Yep. 41:33.372 --> 41:36.124 is if my agent writes the code and my agent writes the test, 41:36.124 --> 41:37.986 like we've all been in this scenario where the agent's like, 41:37.986 --> 41:40.157 well, this test passed, so I or this test failed, 41:40.157 --> 41:41.458 so I deleted it. And you're like, 41:41.458 --> 41:43.600 that does get us to a green build, 41:43.600 --> 41:45.631 but it's not necessarily the green build we wanted, 41:45.631 --> 41:48.163 right? Like well, we we needed to know that that was failing. 41:48.163 --> 41:50.325 Let's fix the thing. Also, 41:49.828 --> 41:50.828 Yeah. 41:50.668 --> 41:54.458 you know, I for there's this like model from the 41:54.896 --> 41:56.387 I think it was originally created in the 70s, 41:56.387 --> 41:58.919 like all cool things in software called mutation testing, 41:58.919 --> 42:04.624 where you basically manipulate the implementation and then check to 42:01.552 --> 42:06.819 Amen. Yep. Mm-hmm. I'll make sure it actually fails. 42:04.624 --> 42:06.016 see whether your test would catch it, 42:06.016 --> 42:07.777 right? And it's 42:06.819 --> 42:07.819 Yeah. Yep. 42:07.777 --> 42:10.049 right. And it's like super expensive. 42:10.049 --> 42:12.822 And probably like these things were invented in the 70s and nobody 42:12.822 --> 42:14.833 did them because it's like so much compute power. 42:14.833 --> 42:19.087 But like we all have access to ridiculous amounts of compute right now, 42:17.902 --> 42:20.740 Well, in the seventies we were probably doing it by hand. 42:19.087 --> 42:22.276 right? So yeah, exactly. 42:20.740 --> 42:22.004 We're probably going in a l 42:22.285 --> 42:24.631 chain the 42:24.185 --> 42:25.185 What if I 42:24.665 --> 42:27.986 put some paper over this hole in the punch card and I like punch 42:27.322 --> 42:28.322 Yeah, yeah. 42:27.986 --> 42:31.107 out this hole instead? So, so ~ yeah, 42:31.107 --> 42:33.107 we're, you know, we built some tooling around that. 42:33.107 --> 42:38.168 and and trying to push all that to the left and think about what else we could do 42:36.366 --> 42:38.165 Yeah. Yep. 42:38.168 --> 42:42.449 to improve the quality so that we have really high confidence and we 42:42.449 --> 42:44.590 can get that confidence as early as possible. 42:44.590 --> 42:46.850 So, so really like on the testing front, 42:46.850 --> 42:51.631 trying to create, again, improve the quality of the testing and ensure that 42:51.631 --> 42:53.685 by the time we get to 42:54.190 --> 42:56.852 that place we're gonna run into the merge conflicts and wait for the build time 42:56.852 --> 43:00.044 and all that stuff, that it's in really good shape and we have really high 43:00.044 --> 43:03.336 confidence. And now we're just worried about the merge as opposed 43:03.336 --> 43:06.118 to worried about like, does my thing even pass the local test suite? 43:06.118 --> 43:09.911 Right. And I think for people with large monoliths, 43:09.911 --> 43:11.312 which I think is a lot of people, 43:11.312 --> 43:13.493 right? Even people who have microservices architectures, 43:13.493 --> 43:16.155 90% of their code is still in a giant monolith somewhere. 43:16.155 --> 43:18.516 Those things are really hard to test, 43:18.516 --> 43:20.758 right? And they're really hard to test locally usually. 43:20.758 --> 43:23.810 and so how do you kind of like 43:24.256 --> 43:26.617 We've built a bunch of different tools to optimize testing. 43:26.617 --> 43:29.528 So, like the I guess the other thing when we talk about testing that we're seeing 43:29.528 --> 43:31.539 we're feeling it, our customers are feeling it, 43:31.539 --> 43:35.241 is like some fundamental queuing theory, 43:35.241 --> 43:41.124 right? Like your arrival rate of change is escalating to the point where 43:41.504 --> 43:44.996 the the time spent executing or validating that change, 43:44.996 --> 43:46.927 particularly if you have to serialize that, 43:46.927 --> 43:48.647 right, as you get something out to deploy. 43:50.436 --> 43:52.237 You're gonna reach a point where the arrival, 43:52.237 --> 43:58.161 like it's impossible to service the change faster than the arrival rate. 43:58.161 --> 44:02.024 And as a result, you're going to infinitely cue. 44:02.024 --> 44:07.008 Right. So then you have a real like you have a real fundamental problem 44:07.008 --> 44:10.370 of the design of your system. And so then it's like, 44:10.370 --> 44:11.911 okay, how fast can I get that? 44:11.911 --> 44:14.953 Like, how tight can I make that little serialization point? 44:14.953 --> 44:18.906 Right. And how can I minimize the risk? 44:19.459 --> 44:21.400 Of us of something breaking at that point, 44:21.400 --> 44:23.722 right? Like, and there's there's techniques that we have already 44:23.722 --> 44:27.534 and then other techniques that we're introducing to try to to try to make that work. 44:27.534 --> 44:30.036 And then that's where when I'm when we talked about architecture, 44:30.036 --> 44:33.318 like yes, architecture for agents for them to be able to reason about a system 44:33.318 --> 44:37.861 is is important. But then architecture of the system in terms of how do 44:37.861 --> 44:42.204 I get that throughput, right, at some cost optimized approach, 44:42.204 --> 44:44.325 in a way that I'm not just gonna 44:44.801 --> 44:46.475 know, queue all night and be waiting. 44:46.475 --> 44:48.068 Mm you know, I come in Tuesday morning, 44:48.068 --> 44:50.283 I'm still waiting for Monday night's builds to go out so that we 44:50.283 --> 44:51.606 can start working again, sort of thing. 44:51.536 --> 44:52.536 I 44:51.626 --> 44:55.048 see. I see. Got it, gotta gotta 'cause I was I was about to to ask you, 44:55.048 --> 44:57.829 like, okay, we we talked earlier about how changes tend to be bigger now, 44:57.829 --> 45:00.440 but you're you're talking about optimizing that loop and making sure that 45:00.440 --> 45:03.282 it doesn't but but to your point that's sort of in the overnights then where it's 45:03.502 --> 45:06.483 going to the mutation extras and and all these other things. 45:06.483 --> 45:07.483 Yeah. 45:06.628 --> 45:07.628 Well, I think I 45:07.148 --> 45:12.172 think it's all I I've I've met companies where they're they're building 45:12.172 --> 45:14.634 up so much backlog over the course of the day. 45:14.634 --> 45:17.172 I'm gonna say they're primarily not Circle CI customers. 45:17.172 --> 45:21.140 They're building up so much backlog over the course of the day that they're hoping 45:21.140 --> 45:22.781 that when they come back in the morning, 45:22.781 --> 45:24.592 yesterday's stuff will be done. 45:24.592 --> 45:27.525 And I'm just talking about like the builds that were piling up waiting 45:27.456 --> 45:30.465 Mm. Yeah. ~ wow. Wow. 45:27.525 --> 45:30.187 to go through, right? And so, I mean, 45:30.187 --> 45:31.228 the the 45:30.465 --> 45:31.465 ~ 45:31.228 --> 45:33.300 goal of C D is the immediacy, right? 45:33.300 --> 45:34.510 Like, we just broke production. 45:34.510 --> 45:36.312 Awesome. That's 'cause I hit merge four minutes ago. 45:36.802 --> 45:37.802 Right. Not 45:36.805 --> 45:37.805 Right. 45:37.974 --> 45:40.267 three o'clock in the morning, someone's getting page 'cause their three 45:40.267 --> 45:42.600 PM thing finally made it to production. 45:42.600 --> 45:46.394 Right. So I there is some like just how do we again, 45:46.394 --> 45:49.038 how do we optimize this and minimize it to Yeah. 45:46.434 --> 45:49.265 Okay, no, I I get it. You're you're 45:49.265 --> 45:53.899 you're helping me flash back to client sites I got to play with and sort 45:53.899 --> 45:56.101 of twenty seventeen, eighteen where yeah, 45:56.101 --> 45:59.013 t like QA wouldn't get something for, 45:59.013 --> 46:01.214 you know, like a month because they were so queued up. 46:01.214 --> 46:04.266 and to your point, like all the jobs were waiting on the other jobs in front 46:04.266 --> 46:05.737 of them in the c in the CI. Okay, 46:05.242 --> 46:06.242 Yeah. 46:05.737 --> 46:06.918 fine. Yeah, yeah, yeah, yeah, yeah. 46:06.918 --> 46:09.320 Right. Yes, fair points. Okay, 46:09.320 --> 46:10.320 cool. 46:09.946 --> 46:12.814 So yeah, sp speed matters a lot now. 46:12.984 --> 46:15.146 Yeah. Yeah. Yeah. For sure. Yeah. 46:15.146 --> 46:17.658 Otherwise it that that Q just grows even faster. 46:17.658 --> 46:23.593 Hmm. W what do you think change in terms of like infrastructure and and 46:23.593 --> 46:27.716 how we think about like how we ship ~ and ~ those kinds of things. 46:29.155 --> 46:33.976 Yeah. ~ AI begets AI. 46:33.976 --> 46:40.528 That's one thing I would say. Like we're as we move faster with less less, 46:40.528 --> 46:44.059 I'm gonna say less comprehensive understanding of every detail, 46:44.059 --> 46:48.210 right? Like we've got the same number of humans with the same brains processing 46:47.150 --> 46:49.655 Yeah. Yep. Mm-hmm. 46:48.210 --> 46:49.760 way more change, right? So 46:49.760 --> 46:53.001 the likelihood is higher that we put something into a production environment that 46:53.001 --> 46:58.262 we don't fully understand. ~ and we are building all the tools that I mean we are 46:58.436 --> 47:00.226 Building it for ourselves, but like as a software industry, 47:00.226 --> 47:03.197 there's a ton of investment going into the tools to make that as safe as possible. 47:03.197 --> 47:06.468 Right. And in many cases, it's gonna be safer than what we did before. 47:06.468 --> 47:10.149 Like, you know, we could spend a sidebar on humans reviewing PRs, 47:10.149 --> 47:12.350 like, were you even paying attention while you're reviewing that? 47:12.350 --> 47:14.211 Like, so so there's that, 47:12.424 --> 47:13.985 I know. huh. 47:14.211 --> 47:16.231 but then but then you so you get stuff into production. 47:16.231 --> 47:18.752 I mean, ~ like telemetry, observability, 47:18.752 --> 47:21.823 like pick your favorite word is gonna be critical, 47:21.823 --> 47:26.074 right? Because and what I will say. 47:26.368 --> 47:31.379 Think about debugging production systems is I know people, 47:31.379 --> 47:36.711 I know engineers who are not great at debugging the thing they just wrote. 47:37.136 --> 47:38.136 Mm-hmm. 47:37.691 --> 47:41.932 And I know engineers who are great at debugging a system they've never seen before. 47:41.932 --> 47:42.932 So 47:42.192 --> 47:43.192 Mm-hmm. 47:43.913 --> 47:47.414 we're gonna be probably more in that latter category, 47:47.414 --> 47:50.245 right? Like this notion that we can't put something into production that 47:50.245 --> 47:54.096 we don't understand. Like I I've watched people parachute into someone else's 47:54.096 --> 47:55.606 environment and be like, it's right there. 47:56.026 --> 47:58.049 Yeah. Yeah. 47:56.590 --> 47:58.251 That's the problem. And that 47:58.251 --> 48:02.215 takes good tooling, right? Like either they work really hard and it's slow 48:02.215 --> 48:06.909 to get to that understanding, or they have really good tooling. 48:06.909 --> 48:11.062 ~ and specifically, I'm thinking about like traces and understanding like 48:11.062 --> 48:12.473 how is stuff flowing through the system. 48:12.473 --> 48:15.496 And here's the full stack trace when something goes wrong. 48:15.496 --> 48:20.060 and here's the source that I can reference back to so I can go look at it. 48:20.060 --> 48:22.042 And and now when I said AI begets AI, 48:22.042 --> 48:24.824 like I'm going to use, first of all, 48:25.723 --> 48:27.154 To find that in the first place, 48:27.154 --> 48:29.615 right? Debugging production systems, 48:29.615 --> 48:31.006 particularly distributed systems, 48:31.006 --> 48:36.218 is an exercise in pattern matching and correlation across enormous data sets. 48:36.442 --> 48:37.442 Yeah. Mm-hmm. 48:37.339 --> 48:38.760 You know what's really good at that? 48:38.760 --> 48:41.501 Is AI. Like way better than humans, 48:39.448 --> 48:41.997 AI. Mm-hmm. Mm-hmm. 48:41.501 --> 48:42.501 right? 48:42.151 --> 48:43.442 Like you may not have written it, 48:43.442 --> 48:46.714 but just saying this all appears to be this is related to that, 48:46.714 --> 48:49.425 and this is related to that. Like that was my sound of a computer anyway, 48:49.425 --> 48:50.786 is like my modem from when I was kid. 48:50.786 --> 48:54.908 Anyway, like that connection exists. 48:55.320 --> 48:57.581 And this is likely this. And I went through this code, 48:57.581 --> 48:59.922 right? Like I have access to all the repos and I have access to all this data. 48:59.922 --> 49:02.503 Like that will be a remarkable improvement. 49:02.503 --> 49:05.145 I think people have nibbled at the edges of that so far, 49:05.145 --> 49:06.965 but I think it's we're gonna see more of that. 49:06.965 --> 49:11.708 And there's going to be a greater need, 49:11.708 --> 49:17.350 I think, for the humans to really be able to do that large scale thinking. 49:17.350 --> 49:19.011 It'll be it'll be tool assisted, 49:19.011 --> 49:21.092 right? But 49:21.358 --> 49:22.649 You know, the systems will be complex, 49:22.649 --> 49:25.610 they'll be fresh. You won't you won't actually know what the structure of 49:25.610 --> 49:28.810 all of it is. But I mean, I've been doing this a long time. 49:28.810 --> 49:31.634 I've been to a lot of incidents where I had nothing to do with the code that 49:31.634 --> 49:35.596 was acting badly, and I still was able to figure out what what was happening. 49:35.596 --> 49:36.916 And I know peop lots of people like that, 49:35.952 --> 49:37.515 Yeah. Yeah. 49:36.916 --> 49:37.916 right? So 49:37.548 --> 49:42.379 I I think I mean, well, we responded to 49:42.379 --> 49:47.562 a Linux CVE recently that we were trying to fit how does this impact us? 49:47.562 --> 49:50.894 And we got four people on a call reading the Linux kernel. 49:51.652 --> 49:55.175 very claw-assisted, right? But and then building, 49:55.175 --> 49:57.697 you know, tests to validate, you know, 49:57.697 --> 49:59.019 whether or not it impacts us kind of thing. 49:59.019 --> 50:01.081 And like that's that's going to be the world that we're in, 49:59.707 --> 50:01.911 wow. Yeah. Yeah. 50:01.081 --> 50:02.081 right? And that's like that was 50:02.021 --> 50:03.803 not the Linux kernel wasn't written by AI, 50:03.803 --> 50:05.264 but it also wasn't written by us. 50:04.430 --> 50:06.966 Right. Right. Right. 50:05.264 --> 50:06.996 Right. And so 50:06.996 --> 50:09.098 like a very let's it's not a distributed system, 50:09.098 --> 50:12.851 but a very large scale application that's extremely complex, 50:12.851 --> 50:16.725 where I can be guided to a point and then make rational decisions about 50:16.725 --> 50:18.136 how that impacts my system. 50:18.905 --> 50:21.587 Right, I think is is going to be a real part of the job. 50:21.587 --> 50:25.851 And so I think being really comfortable leveraging the tools that we have, 50:25.851 --> 50:31.636 but also like I don't I'm sad to say I don't know how to get people 50:31.636 --> 50:34.168 to be great debuggers. It's not something I've spent a lot of time on, 50:34.168 --> 50:35.780 but I think about it. ~ 50:35.016 --> 50:36.016 yeah, yeah. 50:35.780 --> 50:39.314 as I s, you know, I've spent time in incidents like anybody else that runs 50:39.314 --> 50:45.548 a distributed system. And I I was I was working with one group at some point 50:45.548 --> 50:46.548 and I was like, 50:47.108 --> 50:49.313 I want you all to spend the next week playing Wordle. 50:51.524 --> 50:53.115 And then I want you to tell me at the end, 50:53.115 --> 50:57.907 like how you thought about reducing the problem space. 50:57.907 --> 51:01.800 Right? Like, how do you take because I've got the alphabet and I've 51:01.800 --> 51:02.920 got every word that's ever ex well, 51:02.920 --> 51:04.181 every five-letter word that exists, 51:04.181 --> 51:05.181 right? I think it's five letters. 51:05.111 --> 51:06.111 I haven't played Wordle in a while. 51:05.986 --> 51:06.986 It is. It is fine. 51:08.113 --> 51:10.354 And I've got a fixed number of guesses, 51:10.354 --> 51:14.527 right? I mean, anyone that's listening to this doesn't know how to play Wordle, 51:14.527 --> 51:15.837 just look it up. Like, I know why I'm explaining. 51:15.837 --> 51:20.660 But like the the I have this broad problem space and I use techniques. 51:21.306 --> 51:25.318 To analyze, to probe the space to try to determine likelihoods, 51:25.318 --> 51:28.969 right? Of where I'm going. And of course the game is kind of designed 51:28.969 --> 51:30.080 to drive you through that, right? 51:30.080 --> 51:31.236 But that was like part of the. 51:31.236 --> 51:34.522 And so I think like that kind of reasoning. 51:34.522 --> 51:37.283 There's also this other thing that I'm totally obsessed with. 51:37.283 --> 51:38.823 I'm so sorry. This is totally off the rails. 51:38.823 --> 51:40.964 But it's my God, I'm gonna forget the name of it. 51:40.964 --> 51:43.946 I think it's called the Matson Consensus Protocol. 51:43.946 --> 51:47.447 And it's used in wilderness search and rescue, 51:47.447 --> 51:49.328 where 51:50.018 --> 51:52.809 so someone is missing or an aircraft is I think it came from aircraft originally. 51:52.809 --> 51:56.140 And someone is missing or an airplane is missing or whatever, 51:56.140 --> 51:57.911 and you get together a bunch of experts, 51:57.911 --> 52:04.513 and they all get points to allocate to search areas based 52:04.513 --> 52:07.794 on their expertise and understanding of likelihood. 52:07.794 --> 52:12.546 And that's how you decide with a limited search party where you should go look. 52:13.326 --> 52:15.628 Right. And one of the things I love about software is we always have 52:15.628 --> 52:17.029 to invent everything for ourselves. 52:17.029 --> 52:20.361 Like there are so many industries and disciplines where people have solved these 52:20.361 --> 52:24.594 things, right? And so this notion that like we have a bunch of experts on 52:24.594 --> 52:27.706 a call or whatever, like in like talking about production debugging. 52:27.706 --> 52:31.299 Sorry, this is all built from the assumption that AI is gonna make 52:31.299 --> 52:32.553 a mistake and then we're gonna put something in production. 52:32.553 --> 52:36.312 We're gonna try like having tool and you could automate that too, 52:36.312 --> 52:41.276 right? Like having tools and techniques to say to drive towards 52:41.637 --> 52:44.539 the likely cause of the problem instead of, 52:44.539 --> 52:46.861 I don't know, AI built all of this. 52:46.861 --> 52:50.863 Let's start at line one and read through to line fifty million 52:50.863 --> 52:52.905 and maybe somewhere in there we'll see something that's wrong. 52:51.236 --> 52:53.579 Yeah. Yeah, yeah. 52:52.905 --> 52:53.905 Right. And so I think 52:53.785 --> 52:56.687 I think we'll be forced to build, 52:56.687 --> 52:59.549 you know, better tooling, better approaches, 52:59.549 --> 53:01.951 better practices. Again, I assume this will 53:01.110 --> 53:02.110 Mm, interesting. 53:01.951 --> 53:03.652 all show up in software because that's how we solve things. 53:04.526 --> 53:07.428 I I like I like how you're talking about we we talked earlier about 53:07.428 --> 53:11.340 how you were saying that like we all need to have more of a product lean these days. 53:11.340 --> 53:13.461 and lots of guests have talked about like, 53:13.461 --> 53:15.922 yeah, we we're all closer to the product now. 53:15.922 --> 53:18.233 ~ that that shows up in my life, 53:18.233 --> 53:23.827 you know. ~ I I advise people talk to the AI about the domain. 53:23.827 --> 53:26.098 If you're talking to it about the curly braces and the semicolons, 53:26.098 --> 53:28.199 you're missing the point. Tell it about 53:27.600 --> 53:28.600 Mm-hmm. 53:28.199 --> 53:29.860 the business problem you're trying to solve, 53:29.860 --> 53:32.831 right? And and it will be better at helping you solve that business problem. 53:32.831 --> 53:33.831 So I think 53:34.074 --> 53:35.074 There's 53:34.334 --> 53:37.136 a there was a whole category of software developers who really wanted 53:37.136 --> 53:41.169 to make sure that, you know, each line was perfectly idiomatic and you know that we, 53:41.169 --> 53:45.713 you know, and I'm not sure that skill set is as important as it used to be. 53:45.713 --> 53:51.838 But to your point, this I can debug something ~ skill set 53:51.838 --> 53:55.721 is still quite valuable, especially in these complex distributed systems that we've 53:55.721 --> 53:57.863 created. So I think for a lot of us, 53:57.863 --> 53:59.183 like we're just gonna ask Claude to do it, 53:59.183 --> 54:00.183 you know. 53:59.970 --> 54:03.801 ~ it's funny when when I'm doing that with my own sort of smaller systems, 54:03.801 --> 54:07.172 ~ if this is part of my solo work, 54:07.172 --> 54:09.423 like I will it's not that I'm racing it, 54:09.423 --> 54:11.603 but I will I will get it kicked off and say, 54:11.603 --> 54:12.734 start looking at this production issue, 54:12.734 --> 54:14.964 please, right away. And then I'll go do the same thing. 54:14.964 --> 54:17.845 And it's it's not so much that I think I'm going to win, 54:17.845 --> 54:19.876 but that I might spot something that I can share with it, 54:19.876 --> 54:21.706 you know, to to to help it along. 54:21.706 --> 54:23.226 And every now and again I win, 54:23.226 --> 54:25.387 but it's pretty rare. Usually Claude gets there first. 54:25.387 --> 54:28.778 But I I I think the point you're making is a different one though. 54:28.869 --> 54:33.291 Which is that like the if you've tapped out the agent's ability to figure 54:33.291 --> 54:36.903 out what's going on, then that other skill set of being the human who 54:36.903 --> 54:41.566 can jump on that, you know, severity one call and like spot something that might 54:41.566 --> 54:45.318 help d also continues to be an important skill set, 54:45.318 --> 54:47.189 you know. That maybe the idiomatic 54:46.478 --> 54:47.478 Yeah. 54:47.189 --> 54:51.491 code guy or or or or developer is less useful than they used to be, 54:51.491 --> 54:55.133 but the being able to spot the issue in a large distributed system needle 54:55.133 --> 54:56.724 in a haystack is still quite valuable. 54:56.724 --> 54:57.724 Yeah. 54:57.828 --> 55:00.289 Yeah, I mean, I would describe it as systems thinking, 55:00.289 --> 55:01.410 right? Like 55:00.622 --> 55:01.622 Yeah, yeah. 55:01.710 --> 55:04.792 understanding this thing over here impacts this thing over here, 55:04.762 --> 55:05.762 Mm-hmm. 55:05.112 --> 55:08.674 right? And you could allow your agent to build up that context. 55:08.674 --> 55:13.957 And like we'll probably build more permanent memory and context around these systems 55:13.957 --> 55:17.209 so that it isn't, hey, could you read all this code right now like 55:17.209 --> 55:19.431 you know nothing about our code base and start from scratch and here's 55:19.431 --> 55:22.382 the error kind of thing? Like I think that's a thing we'll we'll advance. 55:22.382 --> 55:24.954 But I also think the the 55:26.148 --> 55:30.960 The fun and terrifying thing about prompting is we pass on our human biases, 55:30.960 --> 55:32.591 right? Like we we we had 55:31.760 --> 55:32.760 Course. Yeah. 55:32.591 --> 55:35.092 an incident recently where a couple of us were trying to 55:35.092 --> 55:37.853 use some production debugging agent tools. 55:37.853 --> 55:41.275 And after the incident, 55:41.275 --> 55:44.216 in our review of like what what could we have done better, 55:44.216 --> 55:46.738 we both kind of independently went and retried, 55:46.738 --> 55:49.529 knowing what had happened, right? 55:49.529 --> 55:51.550 No now we know, so it's a little bit of a cheat, 55:51.550 --> 55:52.550 but like. 55:52.324 --> 55:54.916 What could I have asked that would have got me there sooner? 55:54.916 --> 55:59.889 And you could see the questions we initially asked were so burdened with bias that 55:59.889 --> 56:03.392 we sent our agents off on wild goose chases instead 56:01.220 --> 56:03.189 Down the wrong direction. Yeah. 56:03.392 --> 56:05.093 of giving them the freedom to be like, 56:05.093 --> 56:07.675 I don't know, I think it might be over here. 56:07.675 --> 56:08.749 You know what I mean? Cause it was like, 56:08.749 --> 56:10.427 what you know, what is causing X to happen? 56:10.427 --> 56:12.138 But X was a symptom, not a cause, 56:12.138 --> 56:13.138 sort of thing, right? 56:12.196 --> 56:13.196 Right, right. 56:13.259 --> 56:17.341 And ~ and so I think it takes both, 56:17.341 --> 56:21.024 right? I I one, I need to guide the agent in the right direction. 56:21.601 --> 56:25.562 Or spin up a bunch. I mean, in that that whole sort of like search protocol idea, 56:24.590 --> 56:26.447 Yeah. It does. It does. 56:25.562 --> 56:26.573 the thing is I kind 56:26.573 --> 56:28.964 of have unbounded agents if I want them, 56:28.964 --> 56:31.865 assuming you know, if soon of my foundation model is actually 56:31.865 --> 56:33.125 up and servicing requests right now. 56:33.125 --> 56:34.125 And 56:33.252 --> 56:34.252 Yeah. 56:33.635 --> 56:37.377 so if it's their incident, they're they probably I wonder if they 56:37.377 --> 56:41.398 use other people's models just let's not talk about that. 56:41.398 --> 56:44.590 Okay, so ~ yeah, 56:42.528 --> 56:43.630 It turtles all the way down. 56:44.590 --> 56:49.641 yeah, exactly. So ~ they pr they have like some someone has a like open source model 56:49.641 --> 56:51.112 under their desk that they just point all that at. 56:51.364 --> 56:56.206 So you, you know, you have kind of this unlimited capacity and you 56:56.206 --> 57:01.307 can but but knowing how to seed not random 57:01.307 --> 57:04.869 but multi-path investigation, like that's what a good IC does, 57:03.834 --> 57:05.738 Yeah. Right. 57:04.869 --> 57:05.869 right? You go 57:05.869 --> 57:07.610 look at this, you go look at that. 57:07.610 --> 57:10.311 In in a less well managed incident, 57:10.311 --> 57:12.671 12 people are looking at the same thing when the 57:12.324 --> 57:13.324 Yeah. 57:12.671 --> 57:13.822 real problem is somewhere else, 57:13.822 --> 57:14.822 right? 57:14.125 --> 57:15.366 It's funny when when you say that, 57:15.366 --> 57:16.745 what what occurs to me is like, 57:16.745 --> 57:20.599 yeah, I feel like it goes better if I describe to Claude, 57:20.599 --> 57:23.210 inevitably, ~ in my world anyway, 57:23.210 --> 57:25.971 what I'm observing and not try to, 57:25.390 --> 57:26.390 Mm. 57:26.112 --> 57:28.193 you know, just just for observations. 57:28.193 --> 57:29.483 Like even 57:28.996 --> 57:29.996 Yeah. 57:29.483 --> 57:32.205 something as dumb as is down can be really helpful. 57:32.205 --> 57:35.817 I mean, i because it'll just go go start doing getting its own observations, 57:35.817 --> 57:36.817 you know, and it might be 57:36.421 --> 57:37.421 Yeah. 57:36.777 --> 57:41.220 faster at it than I am. So I will have had an observation that pulled me to it. 57:41.970 --> 57:42.970 so maybe 57:42.029 --> 57:43.029 Uh-huh. 57:42.430 --> 57:44.761 it's like an AWS Cloud Watch alert or something. 57:44.761 --> 57:48.153 I might like copy the cut text of that and just paste it in and say go. 57:48.153 --> 57:52.325 ~ and I I don't bother to do much theorizing at that at that stage. 57:52.229 --> 57:53.229 Yeah. 57:52.765 --> 57:55.486 But but I also wanna there there's something you said about how like, 57:55.486 --> 57:56.727 Kyle, that's just systems thinking. 57:56.727 --> 58:00.259 And and I think you're right, but I have known people that I thought 58:00.259 --> 58:03.770 of as good systems thinkers forward in terms of like designing a system, 58:03.770 --> 58:05.191 but weren't necessarily 58:04.496 --> 58:05.496 Mm. 58:05.191 --> 58:07.432 the person you would call when it all breaks. 58:08.142 --> 58:12.877 Right. I think there there were other people that were really good at like gisting 58:12.877 --> 58:18.302 a situation very quickly. ~ that were and and they might also have been good forward 58:18.302 --> 58:20.975 systems thinkers, who knows? But sometimes people who I think of 58:20.975 --> 58:24.288 as very good forward systems thinkers were not good at the quick gist. 58:24.288 --> 58:27.111 ~ so yeah, I mean, you're right. 58:27.111 --> 58:28.392 It is a kind of systems thinking, 58:28.392 --> 58:30.715 but I still think that there's like specializations on it, 58:30.715 --> 58:31.876 I guess, if that makes any sense. 58:32.996 --> 58:35.703 Yeah yeah. I think 58:38.191 --> 58:44.213 Di yeah, diagnosing is different than ~ than building a system. 58:44.213 --> 58:47.039 and I th I think 58:49.200 --> 58:51.391 I don't know. Yeah, the other the other big thing that I think about there 58:51.391 --> 58:53.031 in there is is this pattern matching. 58:53.031 --> 58:55.182 And maybe that's not quite systems thing. 58:55.182 --> 58:56.182 And it's like 58:56.250 --> 58:57.250 Yeah. 58:56.723 --> 58:57.833 it's like thinking fast and slow. 58:57.833 --> 58:59.684 We run into the like, well, we've seen this before, 58:58.638 --> 59:01.356 Right, right. Right. 58:59.684 --> 59:01.404 therefore it must be this. That's the worst kind 59:01.404 --> 59:07.046 of pattern matching, right? As opposed to that thing there feels like 59:07.046 --> 59:08.697 it might be related to this thing over here, 59:08.697 --> 59:09.837 even though no one's seeing that. 59:09.837 --> 59:11.568 And it's it's ~ 59:10.372 --> 59:11.843 Right. That 59:11.843 --> 59:14.514 that story about the firefighter who runs into the building and just says 59:14.514 --> 59:17.505 get out and does can't really explain why or how he knows, 59:17.505 --> 59:19.135 but like just get out. Yeah, 59:17.912 --> 59:18.999 Yeah. Yeah. 59:19.176 --> 59:22.587 and that is useful, right? And and sometimes thinking fast is not what you need. 59:22.587 --> 59:23.947 It's thinking slow that you need. 59:23.947 --> 59:26.819 But yeah. Wow. We are really 59:25.805 --> 59:26.805 Right. 59:26.819 --> 59:27.919 solving the world's problems now, 59:27.919 --> 59:28.919 Rob. Sorry, 59:28.802 --> 59:30.583 Well, I mean, I I do 59:28.879 --> 59:29.879 keep going. 59:30.583 --> 59:32.714 think someone is gonna build a really good version of this, 59:32.714 --> 59:34.755 right? Which is like, you know, 59:33.801 --> 59:37.091 Mm. Mm. Yeah. 59:34.755 --> 59:36.156 I said, Well, that's a good IC. 59:36.156 --> 59:37.257 But do I 59:37.537 --> 59:42.259 like being a good IC, like there there are these things I I always joke about this, 59:42.259 --> 59:43.540 or at least joke about it recently, 59:43.540 --> 59:46.492 like in the late nineties, I think it was the late nineties, 59:46.492 --> 59:48.873 we were really into this concept of expert systems, 59:48.873 --> 59:49.873 which is like we're 59:49.072 --> 59:50.072 Mm-hmm. 59:49.472 --> 59:51.894 gonna take all the world's knowledge and build it into computers and then 59:51.894 --> 59:53.525 we can just ask it and it'll solve anything. 59:53.525 --> 59:56.587 And we obviously gave up like so many other things, 59:56.587 --> 59:58.388 that in the unified email inbox. 59:58.722 --> 01:00:02.216 And like in 2026, 01:00:02.216 --> 01:00:04.508 we're accidentally building those things, 01:00:04.508 --> 01:00:06.150 right? We're taking all of our knowledge. 01:00:05.550 --> 01:00:08.569 Mm, yeah. Hmm. 01:00:06.150 --> 01:00:07.251 And every time we have a conversation, 01:00:07.251 --> 01:00:08.652 we're like, cool, write a markdown file, 01:00:08.652 --> 01:00:11.425 write a markdown file. And we just have these enormous markdown files. 01:00:11.425 --> 01:00:16.170 But we can give like useful direct guidance and direction to an LLM 01:00:16.430 --> 01:00:21.395 and have it execute on something based on our learned experience, 01:00:21.395 --> 01:00:22.395 right? Like 01:00:21.668 --> 01:00:22.668 Yeah, 01:00:22.041 --> 01:00:23.041 yeah. 01:00:22.287 --> 01:00:23.287 Here's 01:00:22.537 --> 01:00:24.569 how you review code. We talked about everyone building their 01:00:24.569 --> 01:00:27.931 own code reviewers because we've done the same review over and over. 01:00:27.931 --> 01:00:32.034 Like the way that we built ours was we just pulled from all of our existing repos. 01:00:32.034 --> 01:00:36.227 We said, go read all the reviews and find the authors that comment the most 01:00:36.227 --> 01:00:38.297 and that people trust, and then figure 01:00:37.818 --> 01:00:38.818 Hmm. 01:00:38.297 --> 01:00:42.511 out what they say over and over and over and just say it yourself. 01:00:42.511 --> 01:00:45.053 Right? Because every every senior engineer is like, 01:00:45.053 --> 01:00:47.534 why am I telling you the same thing again today? 01:00:47.534 --> 01:00:49.916 We talked about it yesterday. So let's not bother. 01:00:50.170 --> 01:00:51.170 Nice. 01:00:50.810 --> 01:00:53.261 Giving that broken thing to the senior engineer to say, 01:00:53.261 --> 01:00:54.541 again, like 01:00:53.520 --> 01:00:54.520 Yeah, yeah. 01:00:54.721 --> 01:00:56.302 we talked about this, do it this way, 01:00:56.302 --> 01:00:57.942 right? Like we we can encode all that stuff. 01:00:57.942 --> 01:01:03.882 And like when it became really easy and and the the 01:01:03.882 --> 01:01:08.275 f the ris return on investment for that engineer became really clear, 01:01:08.275 --> 01:01:10.786 like, wow, I can stop having this conversation because like, 01:01:10.786 --> 01:01:13.526 you know, cute little Claude told you not to do that. 01:01:13.526 --> 01:01:14.667 And like you listen to Claude, 01:01:14.667 --> 01:01:16.927 great, like I can move on and do higher value things. 01:01:16.927 --> 01:01:19.068 And I think, you know. 01:01:19.492 --> 01:01:22.333 These things like being a great incident commander, 01:01:22.333 --> 01:01:24.603 I keep saying I see, but incident commander is what I mean, 01:01:24.603 --> 01:01:26.355 where you 01:01:26.071 --> 01:01:27.071 Yep. 01:01:26.615 --> 01:01:31.337 where it's encoded knowledge that you've built through reps, 01:01:31.337 --> 01:01:34.679 and then you try to train up a new person, 01:01:34.679 --> 01:01:38.881 right? And like I think, I think we could build the tooling with 01:01:38.881 --> 01:01:42.742 all of the knowledge that we have to guide you through that process. 01:01:42.742 --> 01:01:44.603 And it would be some combination of like, 01:01:44.603 --> 01:01:46.234 I fed it, you know. 01:01:47.246 --> 01:01:50.999 Pager duty or incident IOs, like how to be an incident commander training guide. 01:01:50.999 --> 01:01:54.641 And then I fed it all of our historical incident reports. 01:01:54.641 --> 01:01:58.894 And then I fed it access to honeycomb, 01:01:58.894 --> 01:02:02.547 data dog, grafana, like whatever your tool is. 01:02:02.574 --> 01:02:03.574 All the things. 01:02:04.248 --> 01:02:08.261 And now it's like paging the right person at the right. 01:02:08.261 --> 01:02:10.022 ~ this looks like it's a problem over here. 01:02:10.022 --> 01:02:11.103 Like, you know who's the expert on that? 01:02:11.103 --> 01:02:12.984 Because I read all of the Git history, 01:02:12.984 --> 01:02:15.086 right? And I know that this is the person who knows that. 01:02:13.860 --> 01:02:15.346 Yeah. Yeah, yeah, yeah. 01:02:15.487 --> 01:02:16.487 Yeah. 01:02:15.960 --> 01:02:19.211 Let's call them, right? And I I don't know. 01:02:19.211 --> 01:02:21.322 I mean, it's some someone's probably someone's building that already. 01:02:20.388 --> 01:02:22.503 Hundred percent. It probably happening, 01:02:21.322 --> 01:02:24.724 But like I I think that it and right. 01:02:22.503 --> 01:02:25.250 yeah. Somebody's vibe coding it right now, 01:02:24.724 --> 01:02:27.105 Yeah, that might 01:02:25.250 --> 01:02:26.250 in fact. 01:02:27.105 --> 01:02:28.245 not be the version I think. Yeah, 01:02:28.245 --> 01:02:30.586 well, whatever. Vib coding doesn't I don't even know what that means anymore. 01:02:30.586 --> 01:02:34.778 But like huge amounts of data to process, 01:02:34.778 --> 01:02:38.569 right? Like this is it's such a fantastic application of of this tooling. 01:02:37.636 --> 01:02:39.094 Love it. Love it. 01:02:38.569 --> 01:02:39.569 And I think 01:02:39.194 --> 01:02:42.871 it it will it will force us to have to build it and it will make 01:02:42.871 --> 01:02:45.242 it possible to build it. And I think that's a that's like a weird. 01:02:45.787 --> 01:02:46.787 Yeah. 01:02:45.923 --> 01:02:48.740 Interplay of a lot of scenarios with AI. 01:02:46.785 --> 01:02:47.790 Yeah, yeah, yeah, yeah. 01:02:48.879 --> 01:02:49.879 Yep. 01:02:49.222 --> 01:02:51.427 Right now it's important, but now it's possible. 01:02:51.427 --> 01:02:54.646 Right? Yeah. Yeah. 01:02:51.458 --> 01:02:53.810 And we can afford it. Like you were talking about mutation testing earlier. 01:02:53.810 --> 01:02:54.810 It's like we couldn't afford 01:02:54.680 --> 01:02:56.341 it before and we maybe didn't need it. 01:02:56.341 --> 01:02:58.113 We do now. Yeah. Yep. 01:02:57.220 --> 01:02:58.220 Right, right. 01:02:58.193 --> 01:02:59.203 And suddenly we can afford it, 01:02:59.203 --> 01:03:03.106 which is great. Okay. What one more question on the what you're observing 01:03:03.106 --> 01:03:04.837 and then we need to sort of start, 01:03:04.837 --> 01:03:05.908 you know, windy down the hill. 01:03:05.908 --> 01:03:07.428 You've been very generous with your time so far, 01:03:07.428 --> 01:03:10.201 Rob. Thank you. ~ but but one more question. 01:03:10.201 --> 01:03:13.083 We we've talked a little bit about industry trends and what you're seeing 01:03:13.083 --> 01:03:16.385 and what you can grok about what's happening with hiring and economics 01:03:16.385 --> 01:03:18.737 and those kinds of things. And you talked a little bit about how, 01:03:18.737 --> 01:03:20.268 you know, everybody's committing now and 01:03:20.612 --> 01:03:22.073 Really, really good insightful stuff. 01:03:22.073 --> 01:03:24.315 Anything else that you're seeing around, 01:03:24.315 --> 01:03:26.016 you know, how teams are built, 01:03:26.016 --> 01:03:30.100 right? I think especially in the consultancy world that I've that I've lived 01:03:30.100 --> 01:03:33.443 in for a lot of my career, we got really used to like, 01:03:33.443 --> 01:03:34.894 okay, you're gonna need, you know, 01:03:34.894 --> 01:03:38.917 sort of call it six software developers and a project manager 01:03:38.917 --> 01:03:40.238 and a business analyst, you know, 01:03:40.238 --> 01:03:42.570 especially sort of pre-Eric Reese, 01:03:42.570 --> 01:03:45.493 you know, ~ lean thinking, like maybe after that it would have been 01:03:45.493 --> 01:03:47.754 a product manager and not so much a business analyst, 01:03:47.754 --> 01:03:49.596 but you got used to certain shapes. 01:03:49.912 --> 01:03:50.913 teams, right? And 01:03:50.469 --> 01:03:51.469 Yeah. 01:03:51.093 --> 01:03:53.544 and of those six developers, like you needed a couple of front end and 01:03:53.544 --> 01:03:55.816 a couple of back end, they all really needed to be full stack, 01:03:55.816 --> 01:03:57.337 but you still needed some specialists, 01:03:57.337 --> 01:03:58.584 you know, and and 01:03:58.042 --> 01:03:59.042 Yeah. 01:03:58.584 --> 01:03:59.738 maybe depending on the domain, 01:03:59.738 --> 01:04:01.339 you needed something else and something else. 01:04:01.339 --> 01:04:05.041 But you know, there was sort of a a shape that you could almost just stamp out, 01:04:05.041 --> 01:04:08.604 you know. Are you seeing that change any like 01:04:10.748 --> 01:04:16.190 Yeah. I yes. But I don't know what the destination is. 01:04:16.190 --> 01:04:19.350 I just know it doesn't feel right anymore. 01:04:19.350 --> 01:04:20.679 Right. So there's like you know, 01:04:20.679 --> 01:04:26.671 we already talked about kind of giving more product thinking to the engineers. 01:04:26.671 --> 01:04:28.383 And maybe that's what that was a path 01:04:27.108 --> 01:04:28.257 Yeah. Mm-hmm. 01:04:28.383 --> 01:04:30.703 we were on anyway, but it was just more clear, 01:04:30.703 --> 01:04:34.454 right? I think the if you think about 01:04:35.155 --> 01:04:38.386 the cycle, right? 01:04:38.386 --> 01:04:39.908 The life cycle of 01:04:40.357 --> 01:04:42.708 Let's call it a feature, doesn't really matter. 01:04:42.708 --> 01:04:44.199 Pick big project, small project. 01:04:44.199 --> 01:04:48.351 And the time horizons to execute on a thing, 01:04:48.351 --> 01:04:50.342 call that unit of work, you know, 01:04:50.342 --> 01:04:56.525 X, X amount of work. The handoffs between multiple parties within that 01:04:56.584 --> 01:05:00.247 X amount of work might account for 10%, 01:05:00.247 --> 01:05:01.247 right? 01:05:00.687 --> 01:05:01.687 Uh-huh. 01:05:01.188 --> 01:05:02.882 Maybe 20. I mean, I'm being really generous. 01:05:02.882 --> 01:05:08.881 It's probably like 80, but and then take X and make it achievable in 01:05:08.881 --> 01:05:09.952 one tenth of the time. 01:05:10.640 --> 01:05:11.640 Mm-hmm. 01:05:10.720 --> 01:05:11.720 Or like 01:05:11.501 --> 01:05:12.801 one one hundredth of the time. 01:05:12.801 --> 01:05:13.922 If 01:05:13.327 --> 01:05:14.327 Mm-hmm. 01:05:13.922 --> 01:05:18.845 that what was twenty percent time spent on handoffs would now become 01:05:19.426 --> 01:05:23.929 two hundred percent, right? Like we could probably use real numbers 01:05:23.929 --> 01:05:25.430 and model this out, but it's not that important. 01:05:25.430 --> 01:05:29.393 200% of your time spent handing things off is completely unacceptable, 01:05:29.393 --> 01:05:33.255 right? So you want in my mind, 01:05:33.255 --> 01:05:38.159 you want to take more of the roles and capabilities of the team 01:05:38.539 --> 01:05:40.040 and consolidate them. 01:05:40.746 --> 01:05:45.998 Into fewer actors. Now that doesn't mean like all PMs don't have jobs now, 01:05:45.998 --> 01:05:52.660 right? Like people bring strengths to teams in ways that others 01:05:52.660 --> 01:05:53.980 are not capable of picking up, 01:05:53.980 --> 01:05:55.961 right? And sort of saying, well, 01:05:55.961 --> 01:05:58.502 well, everyone's just an engineer and they'll just do all the other stuff, 01:05:58.502 --> 01:06:02.813 right? Like I've I've designed apps before and been like laughed out of the room. 01:06:02.813 --> 01:06:04.974 So like someone needs to do some design thinking, 01:06:04.974 --> 01:06:06.184 right? And so 01:06:07.022 --> 01:06:10.815 I but how we structure that, like what is what is the ratio, 01:06:10.815 --> 01:06:14.425 right? Like it's like we kind of need less because the engineers are doing 01:06:14.425 --> 01:06:16.730 a lot themselves, but we need more because they can get so much done. 01:06:16.730 --> 01:06:21.374 Like, I think that's the part that we we can't quite figure out just yet. 01:06:19.748 --> 01:06:22.153 Yeah. Where it's gonna go. Yeah. 01:06:21.374 --> 01:06:24.506 It's gonna be it's gonna be something that will evolve and we'll see emerge, 01:06:22.153 --> 01:06:25.410 Yeah. Yeah. 01:06:24.506 --> 01:06:25.537 right? And different 01:06:25.537 --> 01:06:29.450 teams will be different. If you are a startup before people like that 01:06:29.450 --> 01:06:32.673 was always just the engineers doing it and you accepted that their design was crap, 01:06:32.673 --> 01:06:34.454 or they just lifted it off some site. 01:06:34.454 --> 01:06:36.596 I mean now they're using claw design or whatever to make it. 01:06:36.869 --> 01:06:42.273 It can happen. And so, like, everyone has to do more yet less. 01:06:42.273 --> 01:06:46.827 We can all get more done. Everyone's tools are assisted, 01:06:46.827 --> 01:06:48.438 right? Like the PM can generate, 01:06:48.438 --> 01:06:50.974 you know, perspective that can analyze the data. 01:06:50.974 --> 01:06:53.302 The designer can create designs that can do all these things, 01:06:53.302 --> 01:06:59.007 also augmented and faster. And so I think then 01:06:59.069 --> 01:07:04.571 I do think overall team size is gonna trend smaller. 01:07:04.571 --> 01:07:05.571 That's my gut. 01:07:06.448 --> 01:07:08.589 purely based on just communication overhead, 01:07:08.589 --> 01:07:10.392 right? Like ~ 01:07:08.940 --> 01:07:10.721 Mm-hmm. Yeah, right. 01:07:11.413 --> 01:07:12.865 was every, you know, whatever, 01:07:12.865 --> 01:07:16.439 every node I add is I I forget if it's like factorial or whatever. 01:07:14.800 --> 01:07:16.874 Two to the end. Yeah. Mm-hmm. 01:07:16.439 --> 01:07:17.439 Yeah, exactly. 01:07:17.180 --> 01:07:18.180 So 01:07:19.758 --> 01:07:22.969 We want to reduce that, which then means, 01:07:22.969 --> 01:07:28.610 you know, principles like team topologies and how do I create independence between 01:07:28.610 --> 01:07:30.491 teams so they can execute, you know, 01:07:30.491 --> 01:07:32.912 on their own value stream or whatever terminology you want to use. 01:07:32.784 --> 01:07:33.784 Mm-hmm. 01:07:32.912 --> 01:07:33.912 I 01:07:33.272 --> 01:07:36.653 think that becomes important. But then it's like, 01:07:36.653 --> 01:07:39.163 I could literally touch every system in five minutes. 01:07:39.163 --> 01:07:41.404 I like I I spent four weeks working 01:07:39.780 --> 01:07:41.259 Yeah. Yeah. 01:07:41.404 --> 01:07:43.794 on something in a programming language that I still don't understand. 01:07:43.794 --> 01:07:47.556 Because I know enough about programming that I can tell roughly if it's right, 01:07:43.918 --> 01:07:45.873 Right, right. Yeah. Yeah. 01:07:47.556 --> 01:07:48.556 but 01:07:48.528 --> 01:07:49.528 I 01:07:49.424 --> 01:07:52.104 Couldn't sit down and handwrite that because I've never taken the time 01:07:52.104 --> 01:07:53.148 to learn that particular language, 01:07:53.148 --> 01:07:55.586 but it doesn't like like the ability to kinda 01:07:54.222 --> 01:07:55.679 And the last thirty percent doesn't matter as 01:07:55.679 --> 01:07:58.110 much as it used to because machines are maintaining it now, 01:07:58.110 --> 01:07:59.436 not us. So yeah, yeah. 01:07:58.488 --> 01:07:59.508 Yeah. I mean, I 01:07:59.508 --> 01:08:03.369 know if it's well tested. I know if it does what I wanted it to do. 01:08:03.369 --> 01:08:06.880 I have security scanner. Like have all this other automation to make sure that 01:08:06.880 --> 01:08:08.041 I'm not making massive blunders. 01:08:08.041 --> 01:08:10.101 I have other people. We have all these review tools we build. 01:08:10.101 --> 01:08:14.972 Like we have the machinery to make that happen in a way that then allows me to be, 01:08:14.972 --> 01:08:18.754 you know, to move independently across a system as opposed to like I'm tucked 01:08:18.754 --> 01:08:20.304 in my little corner and I just do this thing. 01:08:20.304 --> 01:08:21.634 So ~ it feels like 01:08:20.622 --> 01:08:21.622 Yeah, yeah, for sure. 01:08:21.634 --> 01:08:24.635 the greatest non-answer. Like I it's absolutely changing, 01:08:24.635 --> 01:08:27.656 but like you can see factors that are pushing. 01:08:28.185 --> 01:08:33.837 In both directions, and maybe we end up after all of that push and pull with 01:08:33.837 --> 01:08:36.827 the exact same team that just functions really differently. 01:08:36.827 --> 01:08:39.388 I don't know. I I really think though, 01:08:39.388 --> 01:08:43.549 the ~ you know, I am the one person who does this part. 01:08:43.549 --> 01:08:48.029 I think that is gonna ~ be less. 01:08:48.029 --> 01:08:49.371 I think we're gonna see less of that, 01:08:48.899 --> 01:08:52.528 Yeah. Yep. A hundred percent. Hm. 01:08:49.371 --> 01:08:53.721 right? ~ and if you're really focused. 01:08:52.528 --> 01:08:53.831 Are we all generalizing more 01:08:53.831 --> 01:08:57.198 again? Interesting. It's a regeneralization almost. 01:08:57.198 --> 01:08:58.198 Yeah. Go ahead. 01:08:58.551 --> 01:09:01.263 Yeah, yeah. I mean, well, one, 01:09:01.263 --> 01:09:04.774 we we already talked about the fact that like knowing the intricacies 01:09:04.774 --> 01:09:10.477 of the curly braces, right? Like I remember my my one college course 01:09:10.477 --> 01:09:13.398 and programming, like I was why why? 01:09:13.398 --> 01:09:15.629 Why is there a semicolon on this line and not this line? 01:09:15.629 --> 01:09:18.229 Like someone make that make sense to me, 01:09:18.229 --> 01:09:21.401 right? Like, no, no kid is worried about that anymore. 01:09:21.401 --> 01:09:22.962 It's just not a thing, right? Like, 01:09:22.810 --> 01:09:23.810 Yeah, yeah. 01:09:22.962 --> 01:09:23.962 like it just 01:09:23.652 --> 01:09:27.524 it just doesn't happen. And so like that. 01:09:28.416 --> 01:09:33.296 Specific component is dialed back in terms of the role of an engineer, 01:09:33.296 --> 01:09:35.298 right? So the engineer has a lot more capacity. 01:09:34.200 --> 01:09:36.224 For sure. For sure. 01:09:35.298 --> 01:09:36.328 It might just be I'm 01:09:36.328 --> 01:09:38.698 going to build more, but it might be I'm going to think more strategically 01:09:38.698 --> 01:09:40.457 and I'm going to have like these questions about the customer. 01:09:40.457 --> 01:09:42.220 I'm going to spend that time thinking about product, 01:09:42.220 --> 01:09:46.531 which maybe up-levels the PM or the PM can support more teams, 01:09:42.234 --> 01:09:46.069 Something else. Yep. Yep. Mm-hmm. 01:09:46.069 --> 01:09:47.069 Yep. 01:09:46.531 --> 01:09:47.531 or the PM 01:09:47.060 --> 01:09:52.893 writes some code. Like, and a lot of that is going to be individual aptitudes 01:09:52.893 --> 01:09:55.203 and and skill sets, right? Like, 01:09:53.678 --> 01:09:56.822 Yeah. Interests and rewards and yeah. 01:09:55.203 --> 01:09:56.356 yeah, 01:09:57.064 --> 01:09:58.064 I think. 01:09:57.884 --> 01:10:00.605 You it's interesting because you talked about like stamping out the teams. 01:10:00.605 --> 01:10:03.686 Right? Well, we know it's exactly looks like this. 01:10:03.686 --> 01:10:07.847 But I I've always been I don't know about skeptical, 01:10:07.847 --> 01:10:12.548 but curious about that. Because on any given team like, 01:10:12.548 --> 01:10:15.439 sure, we could have two front end developers on this team and two on this team 01:10:15.439 --> 01:10:18.310 and two on this team. But this one over here is basically a designer. 01:10:18.310 --> 01:10:22.131 This one over here is great at getting the pixels that the designer told 01:10:19.866 --> 01:10:23.082 Yeah. Yeah. Yeah, yeah. 01:10:22.131 --> 01:10:24.057 him to get into place, right? So like they kind 01:10:23.082 --> 01:10:24.082 Yep. 01:10:24.057 --> 01:10:26.832 of look the same, but they don't. 01:10:26.832 --> 01:10:27.832 And like I 01:10:27.749 --> 01:10:30.560 You know, pick your f I watch a lot of soccer in my family. 01:10:30.560 --> 01:10:32.920 It's World Cup summer, so maybe that's topical. 01:10:32.920 --> 01:10:36.651 ~ NFL football starting, or maybe CFL's already underway. 01:10:36.651 --> 01:10:37.751 I don't know if you're a CFL, whatever. 01:10:37.562 --> 01:10:38.562 I don't know. 01:10:37.751 --> 01:10:39.492 Pick pick your favorite sport. 01:10:39.492 --> 01:10:40.502 Okay, you a basketball behind 01:10:40.502 --> 01:10:43.113 so maybe but like, right? Well, 01:10:43.113 --> 01:10:44.353 okay, so I live in the Bay Area. 01:10:44.353 --> 01:10:46.263 I guess we kind of invented small ball, 01:10:46.263 --> 01:10:48.514 right? Like everybody had a giant center, 01:10:48.514 --> 01:10:50.764 and then one day we're just like, 01:10:50.764 --> 01:10:51.835 you know what? What if we didn't? 01:10:51.835 --> 01:10:55.075 Right? Like what if what if the structure of our team, 01:10:55.075 --> 01:10:56.326 right, was 01:10:55.866 --> 01:10:56.866 Yeah. 01:10:56.326 --> 01:10:57.326 such that 01:10:57.246 --> 01:10:58.246 Yeah. 01:10:58.180 --> 01:11:00.322 That isn't actually the best way to play, 01:11:00.322 --> 01:11:05.368 right? Because we we these are individual humans with individual interactions, 01:11:04.570 --> 01:11:05.793 Yeah. Mm-hmm. 01:11:05.368 --> 01:11:06.368 the way they interact 01:11:06.078 --> 01:11:08.971 with other humans, the skills that they bring to the table, 01:11:08.971 --> 01:11:10.313 you know, whatever that might be. 01:11:10.313 --> 01:11:13.396 And so, like that, like, 01:11:13.396 --> 01:11:15.258 how do we take full advantage of that? 01:11:15.258 --> 01:11:16.339 Right. And then that becomes a thing, 01:11:16.339 --> 01:11:18.301 and suddenly everyone in the NBA is throwing threes. 01:11:18.301 --> 01:11:19.301 But like 01:11:18.832 --> 01:11:19.832 Mm-hmm. 01:11:20.223 --> 01:11:21.223 that. 01:11:22.198 --> 01:11:25.849 Is someone right in their particular place stopping to say, 01:11:25.849 --> 01:11:28.810 like, what are the strengths that I have at the table right here? 01:11:28.810 --> 01:11:29.810 And so 01:11:29.188 --> 01:11:30.188 Yeah, yeah. 01:11:29.711 --> 01:11:32.852 I think I particularly as we go through this evolution, 01:11:32.852 --> 01:11:33.852 and I'm just like, I don't know, 01:11:33.622 --> 01:11:37.624 it could be anything. Like we have multiple different teams at Circle 01:11:37.624 --> 01:11:39.625 CI and they're all shaped differently, 01:11:39.625 --> 01:11:43.396 right? Because that yeah. 01:11:39.642 --> 01:11:40.963 Differently. For sure. No, yeah. 01:11:40.963 --> 01:11:42.756 I I don't mean to overstate that. 01:11:42.756 --> 01:11:43.756 But but 01:11:43.456 --> 01:11:46.220 it but it does feel like that the like you said, 01:11:46.220 --> 01:11:49.363 ~ the whatever we were doing before is not quite right. 01:11:49.363 --> 01:11:54.529 ~ so so maybe maybe my preamble there was a bit of a you know oversimplification. 01:11:54.529 --> 01:11:57.332 But but but the basis of the question I think is still interesting, 01:11:57.332 --> 01:11:59.835 you know, which like whatever we were doing it this feels, 01:11:57.672 --> 01:12:00.444 Yeah, no well, it's super interesting because I 01:11:59.835 --> 01:12:00.835 yeah. 01:12:00.444 --> 01:12:05.576 think what we love in engineering is cookie cutter patterns, 01:12:05.576 --> 01:12:06.576 right? We're 01:12:06.000 --> 01:12:07.000 Yeah. 01:12:06.427 --> 01:12:08.858 like, let's, you know, this is the job of a PM. 01:12:08.858 --> 01:12:12.000 And you're like, okay, but that PM's not very good at that part of the job. 01:12:11.852 --> 01:12:14.023 No, yeah. Everybody does it their own way. 01:12:12.000 --> 01:12:16.843 But this this other thing, this other this team member who's like a junior 01:12:14.023 --> 01:12:16.556 Yeah. Yeah. 01:12:16.843 --> 01:12:18.704 engineer is actually really great at that thing. 01:12:18.704 --> 01:12:21.048 Could they do that instead of the PM doing it right like? 01:12:21.048 --> 01:12:24.647 And I think it it's it takes a it's taken me a long time, 01:12:24.647 --> 01:12:26.318 I would say in my in my career. 01:12:26.779 --> 01:12:28.150 To get to the point where I'm like, 01:12:28.150 --> 01:12:34.614 yeah, I'm like, I'm a CTO and other people are CTOs and they do the job away 01:12:34.634 --> 01:12:38.797 and I do the job differently. And that's not because one of us is a bad CTO, 01:12:35.992 --> 01:12:37.094 A different way. That's right. 01:12:37.094 --> 01:12:39.466 Yep. No. Yeah. 01:12:38.797 --> 01:12:39.818 but like 01:12:40.158 --> 01:12:43.220 at some point you you realize you're a member of a team. 01:12:43.220 --> 01:12:45.091 And ultimately 01:12:44.292 --> 01:12:45.292 I love that. 01:12:45.091 --> 01:12:46.562 the goal is the success of the team, 01:12:46.562 --> 01:12:48.584 not your success, right? And 01:12:46.842 --> 01:12:48.575 Yep. Yep. 01:12:48.584 --> 01:12:54.067 so other people on my executive team bring strengths to the table that I don't have, 01:12:54.067 --> 01:12:55.067 and vice versa. 01:12:55.308 --> 01:12:59.050 And my role might look a little bit different than the way someone else does it, 01:12:59.050 --> 01:13:04.253 right? As a CTO, like a very classic distinction would be someone who's can't shut 01:13:04.253 --> 01:13:08.796 up, right? Versus someone who like never wants to talk in front of other humans. 01:13:08.796 --> 01:13:10.687 That's those are two 01:13:09.370 --> 01:13:10.411 Yeah, right? 01:13:10.687 --> 01:13:13.808 totally valid and useful and contributing CTOs, 01:13:13.808 --> 01:13:14.808 right? And then you just ask 01:13:14.224 --> 01:13:15.224 That's right. 01:13:14.609 --> 01:13:16.300 the question, like, okay, in our organization, 01:13:16.300 --> 01:13:18.091 who fills in this role and who fills in this role? 01:13:18.091 --> 01:13:21.673 So I think you know, if if anyone were to take anything about out 01:13:21.673 --> 01:13:23.704 of this ramble about teams, like, 01:13:24.120 --> 01:13:29.183 Now is a great time to really assess who on your team brings what to the table. 01:13:29.183 --> 01:13:30.344 And that may be 01:13:29.562 --> 01:13:30.562 Yeah. Yep. 01:13:30.344 --> 01:13:32.125 across multiple teams in your organization. 01:13:32.125 --> 01:13:37.168 And how can you build a team that's gonna be successful given 01:13:37.168 --> 01:13:38.749 the constraints of this environment? 01:13:37.456 --> 01:13:39.796 I love it. I love it. 01:13:38.749 --> 01:13:39.960 Right. don't take my 01:13:39.960 --> 01:13:42.071 like it's gonna be two of these and one of these and one of these. 01:13:42.071 --> 01:13:43.212 Like, don't know. 01:13:43.844 --> 01:13:46.729 Yeah. So we actually have three questions left, 01:13:46.729 --> 01:13:48.732 ~ believe it or not, and 01:13:48.516 --> 01:13:49.516 Look at us. 01:13:49.113 --> 01:13:51.126 and we've come to the end of our time also. 01:13:51.126 --> 01:13:55.243 ~ so I'm gonna I'll call an audible and and just ask one of them. 01:13:55.243 --> 01:13:58.038 what's something that's keeping you joy right now? 01:13:59.502 --> 01:14:03.224 Giving me joy. Okay, this is I know you're from Toronto, 01:14:03.224 --> 01:14:04.494 so hopefully you'll appreciate this. 01:14:04.494 --> 01:14:07.475 This is totally random. ~ but Rush, 01:14:07.475 --> 01:14:11.267 the band from Toronto, right? ~ 01:14:10.298 --> 01:14:11.298 Mm-hmm. 01:14:11.267 --> 01:14:14.268 is on tour. I have to confess I have not gone to see them, 01:14:14.268 --> 01:14:15.839 but I've seen like every YouTube video. 01:14:15.839 --> 01:14:18.770 And ~ for anyone who doesn't know the band, 01:14:18.770 --> 01:14:20.701 they have this very famous drummer, 01:14:20.701 --> 01:14:23.042 Neil Peart, who like changed drumming, 01:14:23.042 --> 01:14:27.423 right, for the better. And he passed away in I think twenty sixteen. 01:14:27.423 --> 01:14:28.684 Twenty sixteen, twenty seventeen, 01:14:28.684 --> 01:14:29.684 somewhere there. 01:14:30.052 --> 01:14:31.672 So they're back on tour. Actually, 01:14:31.672 --> 01:14:33.633 it was later, my bad. But whatever. 01:14:33.633 --> 01:14:36.884 You can look it up. They hadn't toured since 2016. 01:14:36.884 --> 01:14:38.544 And then they found a new drummer, 01:14:38.544 --> 01:14:41.395 Anika Neele, I think is how you pronounce her name. 01:14:41.395 --> 01:14:44.916 And went out on tour. And the fan, 01:14:44.916 --> 01:14:50.618 it's like the hardest job in the world is to fill in the seat of this man, 01:14:50.618 --> 01:14:52.218 Neil. And she is 01:14:51.736 --> 01:14:52.736 Mm. 01:14:52.298 --> 01:14:56.849 absolutely crushing it. And the Rush fan base absolutely loves her. 01:14:56.849 --> 01:14:58.500 And 01:14:57.454 --> 01:14:58.454 nice. 01:14:58.500 --> 01:14:59.500 it is like 01:14:59.726 --> 01:15:01.467 When I first started seeing the videos rolling, 01:15:01.467 --> 01:15:03.138 because it shows up in my YouTube feed or whatever, 01:15:03.138 --> 01:15:04.138 I was 01:15:03.631 --> 01:15:04.631 Yeah. 01:15:04.018 --> 01:15:08.070 like, finally something just wholesome and happy in the world right now. 01:15:07.866 --> 01:15:08.866 Yeah, yeah. 01:15:08.070 --> 01:15:09.070 You know what I mean? Like there's 01:15:08.858 --> 01:15:10.322 there's so much angst out there. 01:15:10.322 --> 01:15:14.064 And like there's like on the internet in like YouTube comments, 01:15:14.064 --> 01:15:16.505 like it's so easy to be a bad person. 01:15:14.446 --> 01:15:17.751 Yeah. It'd be dark. Yeah. 01:15:16.505 --> 01:15:17.916 And the amount 01:15:17.916 --> 01:15:21.898 of joy generated by something that could have easily been a very, 01:15:21.898 --> 01:15:26.951 very bad outcome, right? There's all kinds of sort of skepticism and whatever. 01:15:24.142 --> 01:15:27.630 I love that. Yeah. Sure. 01:15:26.951 --> 01:15:27.951 ~ 01:15:27.751 --> 01:15:29.432 and it is like 01:15:29.636 --> 01:15:31.517 They're obviously having a huge amount of fun. 01:15:31.517 --> 01:15:33.128 The fan base is having a lot of fun. 01:15:33.128 --> 01:15:36.329 And you know, when you ask the way that you asked the question just landed, 01:15:36.329 --> 01:15:38.901 like it's lately I've been saying there's 01:15:38.160 --> 01:15:39.160 Perfect. 01:15:38.901 --> 01:15:42.662 like the one thing that's giving me great joy right now is just that people 01:15:42.662 --> 01:15:45.064 are so happy about this about this situation. 01:15:45.064 --> 01:15:46.064 So yeah, it's awesome. 01:15:45.252 --> 01:15:46.252 I love it. 01:15:46.015 --> 01:15:47.469 I love it. Okay. One more thing. 01:15:47.469 --> 01:15:49.666 ~ you mentioned earlier the name of the newsletter. 01:15:49.666 --> 01:15:52.355 so let's hear it one more time and I'll make sure it's in the show notes. 01:15:52.758 --> 01:15:56.932 ~ our the state of software delivery report ~ is a thing that we publish 01:15:56.932 --> 01:15:59.435 and then we have a newsletter actually called the confident commit 01:15:59.435 --> 01:16:02.328 and a podcast called the confident commit is like a umbrella for some 01:16:02.328 --> 01:16:03.499 of the other material that we put out. 01:16:03.499 --> 01:16:04.499 So you can look for 01:16:03.952 --> 01:16:04.952 Cool. 01:16:04.310 --> 01:16:05.310 too. 01:16:04.856 --> 01:16:07.010 Cool. Rob, thank you so much for coming on the show today. 01:16:07.010 --> 01:16:08.010 Really appreciate you. 01:16:08.483 --> 01:16:09.916 Yeah, you bet, Kyle. It's really fun.