[00:00:00.000 --> 00:00:05.340]   I think what may be changing right now, and I've yet to prove this yet, but I'm trying
[00:00:05.340 --> 00:00:10.620]   to. I'm thinking about what this looks like is code is cheap now. These elements can
[00:00:10.620 --> 00:00:14.300]   bust out these though, like very, very quickly. I've talked to a few people again last week
[00:00:14.300 --> 00:00:21.300]   around how someone vibed coded a bunch of stuff. But one thing that they did do is they
[00:00:21.300 --> 00:00:26.660]   wrote a budget test. They wrote the test in order to figure out what the specs are. They
[00:00:26.660 --> 00:00:31.340]   are able to blow out their entire code base and rewrite into a new language as long as they
[00:00:31.340 --> 00:00:38.260]   have the test in place. And it's refactoring dead. I don't know, like I get like this is
[00:00:38.260 --> 00:00:42.900]   very controversial, but it's like code is cheap, right?
[00:00:42.900 --> 00:00:48.260]   Cassie Schum is the vice president of ecosystems and interop at relational AI, where she specializes
[00:00:48.260 --> 00:00:53.200]   in designing and deploying AI driven data intensive systems built on relational knowledge
[00:00:53.200 --> 00:00:58.360]   address. Cassie brings over 20 years of experience as a software engineer architect and technology
[00:00:58.360 --> 00:01:03.080]   leader. Prior to a relational AI, she spent more than a decade of thought works, holding senior
[00:01:03.080 --> 00:01:07.600]   leadership roles and was a contributor to the ThoughtWorks technology radar. Cassie and
[00:01:07.600 --> 00:01:10.780]   I actually had the chance to work together at ThoughtWorks, sometimes even on the same
[00:01:10.780 --> 00:01:16.000]   account. So I'm particularly excited to have her here. I'm not at all jealous that she got
[00:01:16.000 --> 00:01:20.760]   to contribute to the technology radar and I didn't, but that's okay. Her technical background
[00:01:20.760 --> 00:01:26.120]   spans distributed systems, microservices, cloud platforms, and a growing focus on semantic
[00:01:26.120 --> 00:01:31.960]   layers, knowledge graphs and AI enabled decision systems. She's a passionate advocate for engineering
[00:01:31.960 --> 00:01:36.440]   excellence and increasing the representation of women and technology. I wanted to talk
[00:01:36.440 --> 00:01:40.920]   to her for a while now about how to move from LLM Heights to production grade AI and the
[00:01:40.920 --> 00:01:45.440]   critical data architecture needed for AI driven enterprises and to ask her instructions
[00:01:45.440 --> 00:01:53.240]   and knowledge graph just a data chart. So today, she's our guest on the engineering with AI
[00:01:53.240 --> 00:02:12.360]   podcast. Cassie, thank you so much for coming. My gosh, thank you so much Kyle for having me.
[00:02:12.360 --> 00:02:16.680]   This is really great. I'm glad you think so. I'm looking forward to it.
[00:02:16.680 --> 00:02:21.720]   So yeah, I've been watching you guys at relational AI for a little while and you guys are doing
[00:02:21.720 --> 00:02:26.240]   some really cool stuff. But like, how did it, how, we talked about your bio a minute ago.
[00:02:26.240 --> 00:02:30.200]   So, you know, what, what would you add and do you want to talk about sort of how you got
[00:02:30.200 --> 00:02:36.320]   to relational AI? Sure, sure. Yeah, I mean, I think it might be worthwhile to talk about
[00:02:36.320 --> 00:02:43.040]   how I got to relational AI, which maybe will answer the what we're doing in a way. So, I've
[00:02:43.040 --> 00:02:48.440]   been at relational for probably about three and a half years now, which I don't know, but
[00:02:48.440 --> 00:02:53.280]   just like that, and then also seems quite long as well with everything that's going on.
[00:02:53.280 --> 00:02:59.800]   But I ended up joining relational AI because when I met the founder, Moham, I was actually
[00:02:59.800 --> 00:03:07.000]   still at ThoughtWorks. And I was thinking about as you put in my bio, I was like, I think
[00:03:07.000 --> 00:03:13.320]   about enterprises, I think about how do you actually model real life, enterprises, domains
[00:03:13.320 --> 00:03:17.640]   and things like that. And then even more so, I thought what was an interesting proposition
[00:03:17.640 --> 00:03:23.040]   was how do you actually move this business logic that is spread across all these distributed
[00:03:23.040 --> 00:03:27.600]   microservices? And if you know they're not going to change that much, how do you actually
[00:03:27.600 --> 00:03:32.680]   put that closer to the data? And of course, the first thing I thought of was like with
[00:03:32.680 --> 00:03:38.520]   Sprox, like what do the stored procedures are going to be dug in about here? Like, no,
[00:03:38.520 --> 00:03:43.920]   the shape of things. So, you know, my skeptical ThoughtWorker in me was like, yeah, that sounds
[00:03:43.920 --> 00:03:49.480]   silly or what not. But once I dug a little bit deeper into what a knowledge graph was,
[00:03:49.480 --> 00:03:55.680]   I thought, hmm, this could be interesting. This could be something here. And so, with
[00:03:55.680 --> 00:04:02.480]   a lot of back and forth, I was convinced. And, yeah, join relational about three and a half
[00:04:02.480 --> 00:04:07.840]   years ago. And I initially joined as a field engineering lead because I was very close
[00:04:07.840 --> 00:04:14.240]   to the customers. I saw a lot of the issues that enterprise customers were having, right?
[00:04:14.240 --> 00:04:19.720]   So how do you model your domain? How do you take care of this sprawl that you have in terms
[00:04:19.720 --> 00:04:25.880]   of business logic and things like that? So it was really good to see up front what people
[00:04:25.880 --> 00:04:34.200]   were actually facing. So, yeah, now I'm excited. I moved to being more of the VP of ecosystem
[00:04:34.200 --> 00:04:39.720]   because now I'm looking to not just helping one or two massive customers, but how do we
[00:04:39.720 --> 00:04:46.440]   reach through ISVs, different developers, different modelers and things like that? So, I just
[00:04:46.440 --> 00:04:52.520]   want to get some more people, you know? So. Yeah, yeah, yeah. So ISVs, independent system vendors,
[00:04:52.520 --> 00:04:58.280]   and software vendors, yeah, whichever one that is. And at least rewind for a second. Like,
[00:04:58.280 --> 00:05:02.280]   when you're talking about, you know, bringing compute closer to the data and you reference
[00:05:02.280 --> 00:05:08.520]   Sprox and stored procedures, like, oh my gosh, no, right? Like, or did it? Did you guys actually,
[00:05:08.520 --> 00:05:12.440]   like, is that how this works? Like, of course, we're checking if statements. Okay, good.
[00:05:12.440 --> 00:05:18.920]   So, I think it was that I wouldn't be here right now. Absolutely not.
[00:05:18.920 --> 00:05:26.360]   Just making sure. Okay, I expected that that so I went, but I wanted to check. Okay, cool.
[00:05:26.360 --> 00:05:32.360]   So, so I mean, you and I were talking about this when we were preparing, right? And we're like,
[00:05:32.360 --> 00:05:37.800]   what the heck is a knowledge graph? I mean, isn't that just what Neo4j does? But I expect there's
[00:05:37.800 --> 00:05:44.120]   more to it. So, so what, what is a knowledge, you know, graph, like, and maybe even like,
[00:05:44.120 --> 00:05:48.280]   start with the data structure a little bit? And, and but we'll then talk about what do you guys
[00:05:48.280 --> 00:05:53.800]   mean when you're talking about building a knowledge graph? Yeah, sure, sure. Yeah, no, I, I agree.
[00:05:53.800 --> 00:05:59.080]   There's always this misconception between graph graph database and a knowledge graph in itself.
[00:05:59.080 --> 00:06:07.560]   And of course, Neo4j and different types of graph databases have been very popular for a long
[00:06:07.560 --> 00:06:12.120]   time. But I think it's you and I chatted about they never took off, right? Like everyone thought
[00:06:12.120 --> 00:06:18.120]   that that was a really, really great idea. But somehow it was just very, very hard to actually get
[00:06:18.120 --> 00:06:23.720]   it to the general public. There's a few reasons for that we found. So a lot of it is like, it's just
[00:06:23.720 --> 00:06:28.520]   super memory intensive, right? If you're trying to traverse over billions and billions of nodes and
[00:06:28.520 --> 00:06:35.480]   edges and things like that, Neo4j, like that, that was hard. We, we, we fell over in performance and
[00:06:35.480 --> 00:06:41.080]   things like that. So there's a bit of an infrastructure piece that has gotten a lot better over the
[00:06:41.080 --> 00:06:46.440]   last 10 years, the last decade has now enabled us to actually look at graph technology in general,
[00:06:46.440 --> 00:06:52.120]   whether it be knowledge graphs or Neo4j or whatnot. I think the thing that we're looking at is like,
[00:06:52.120 --> 00:06:56.680]   how do you actually separate that compute and storage? And so once you are able to do that,
[00:06:56.680 --> 00:07:01.080]   you're able to scale each of these things individually and these things get just a little bit easier
[00:07:01.080 --> 00:07:07.560]   in terms of memory and things. So that's, that's sort of the graph ecosystem, I think. And I, and
[00:07:07.560 --> 00:07:13.080]   there are better experts to talk about why and what and how, you know, there's differences between
[00:07:13.080 --> 00:07:18.520]   navigational pointers and being able to actually query on top of triples and tuples and things
[00:07:18.520 --> 00:07:24.280]   like that. So I won't get into the details of those things. But what I will say is let's,
[00:07:24.280 --> 00:07:28.360]   let's talk about the difference between knowledge graphs and then just a graph database. I think,
[00:07:29.240 --> 00:07:34.120]   I think the misconception is that people just often think knowledge graphs are just a fancy graph
[00:07:34.120 --> 00:07:40.440]   database. And so one of the things that as I was kind of talking about my intro that I really
[00:07:40.440 --> 00:07:47.080]   loved is what it actually is is like, it's when you're combining like the logic and the structure,
[00:07:47.080 --> 00:07:51.320]   right? So you have the structure of the database and the nodes and edges and how do you actually think
[00:07:51.320 --> 00:07:56.280]   about, you know, Kyle, you and I used to work together. We are two nodes and that's our edge,
[00:07:56.280 --> 00:08:01.400]   right? We also used to work at ThoughtWorks, that's another edge between us. I work at relational,
[00:08:01.400 --> 00:08:06.360]   we're doing this podcast yet another edge between us. So there are three different edges and
[00:08:06.360 --> 00:08:11.720]   it's now representing like what real life, I would say, a real Jome would actually look like.
[00:08:11.720 --> 00:08:18.120]   The knowledge graph itself is that logic that I just described. You and I are doing a podcast
[00:08:18.120 --> 00:08:22.600]   together. We used to work at ThoughtWorks together. We used to be on the same team together.
[00:08:22.600 --> 00:08:28.840]   Those are three semantic pieces of information. You are now baking into those edges
[00:08:28.840 --> 00:08:35.000]   into a knowledge graph. And so when you have the semantically rich, interesting information
[00:08:35.000 --> 00:08:41.720]   between all of these different nodes, gosh, you can imagine you're not just, you know, retrieving
[00:08:41.720 --> 00:08:48.680]   data anymore. You're actually inferring reason over these types of things. The connection between
[00:08:48.680 --> 00:08:54.440]   you and I probably looks a lot stronger given the semantics I just said. So I hope that helps
[00:08:54.440 --> 00:08:58.760]   in terms of thinking about what a knowledge graph is. No, it does. When you were describing,
[00:08:58.760 --> 00:09:03.560]   you know, your work there, you were talking about modeling and enterprise. And there was this
[00:09:03.560 --> 00:09:08.120]   part of my brain going like, yeah, I mean, that's kind of what we've always been doing in writing
[00:09:08.120 --> 00:09:12.360]   code. But this sounds like, no, like you're building like a singular model of an enterprise or at
[00:09:12.360 --> 00:09:18.600]   least connected models of an enterprise. That's pretty cool. Yeah. And I think what's interesting
[00:09:18.600 --> 00:09:23.400]   in just something to note is that, you know, we're in pretty big enterprises right now in supporting
[00:09:23.400 --> 00:09:28.200]   them. And what we're finding is if you're thinking about the knowledge graph in itself,
[00:09:28.200 --> 00:09:34.360]   everybody thinks about data. Of course, there's like thousands and thousands and thousands of tables
[00:09:34.360 --> 00:09:39.560]   that maybe an enterprise have. But if you think about the knowledge base, the ontology that the
[00:09:39.560 --> 00:09:44.760]   semantically or sits on, there's actually not that many concepts, right? Some of one of the biggest
[00:09:44.760 --> 00:09:51.960]   telecom companies that we work for right now has about 300 main concepts. That's pretty, you can
[00:09:51.960 --> 00:09:57.640]   handle that, right? Like that is something that can actually be the root of a lot of source knowledge
[00:09:57.640 --> 00:10:02.920]   and knowledge base and things like that. So it's very interesting stuff. I'm excited.
[00:10:05.240 --> 00:10:10.680]   Very cool. Yeah. Okay. So it's funny. When you when you talk about the relationships between us
[00:10:10.680 --> 00:10:17.000]   and you and I and all of this, my brain sort of goes to like, I don't know, like looking at CRM data
[00:10:17.000 --> 00:10:22.280]   and connections with customers, but also potentially interdepartmental customers, my head goes to very
[00:10:22.280 --> 00:10:28.360]   like person style relationships. But I imagine there's others like, what am I missing or do you
[00:10:28.360 --> 00:10:32.680]   want to deep dive into those? Like, yeah, help us think about that part. Yeah, there's, I mean,
[00:10:32.680 --> 00:10:38.200]   there is incredibly like, I can name a lot of different use cases right now. I think the main one
[00:10:38.200 --> 00:10:43.240]   that we just talked about is like how people are connected and you can see LinkedIn and Facebook and
[00:10:43.240 --> 00:10:48.680]   those types of companies are actually built on their own internal knowledge graphs, right? So they
[00:10:48.680 --> 00:10:53.560]   they have knowledge graphs has been around for a very long time, but really only these massive
[00:10:53.560 --> 00:10:58.280]   companies are able to maintain their own internal knowledge graphs. So making this exposed to
[00:10:58.280 --> 00:11:05.320]   that outside road is my goal, obviously. So I think that's one aspect of it. I think some of the
[00:11:05.320 --> 00:11:10.280]   use cases that we're looking at, the big one that we're working with right now is supply chain,
[00:11:10.280 --> 00:11:16.440]   right? So forget people for a second. I mean, if you think about just the normal manufacturing,
[00:11:16.440 --> 00:11:22.120]   supply chain use case where I have three or four warehouses, I have supplies in these different
[00:11:22.120 --> 00:11:26.680]   warehouses. I have trucks that go between these different warehouses and things like that.
[00:11:26.680 --> 00:11:32.840]   I've described three to four different concepts right now and then the relationships between them
[00:11:32.840 --> 00:11:37.800]   would actually describe either the distance, the time, like where they are, how many things they have.
[00:11:37.800 --> 00:11:43.480]   I mean, the number of semantics is endless when you think about even just these four concepts,
[00:11:43.480 --> 00:11:49.160]   right? And I think the power that you get out of something like a knowledge graph is with
[00:11:49.160 --> 00:11:54.040]   these semantics and with this kind of data that you have connected with each other, you can start
[00:11:54.040 --> 00:12:00.200]   to predict things. So as I said, it's not about just retrieving how many items are in one rare
[00:12:00.200 --> 00:12:08.120]   warehouse. Now I can apply things like solvers or prescriptive analytics to say, hey, if this warehouse
[00:12:08.120 --> 00:12:14.040]   closes down, how does everything else reroute, right? And so this is where mathematical optimization
[00:12:14.040 --> 00:12:19.000]   comes in and looks at these different nodes and edges and semantics that we place on there.
[00:12:19.000 --> 00:12:24.600]   And now we can predict like, okay, reroute all of these different trucks and go down this route,
[00:12:24.600 --> 00:12:31.000]   this is the most efficient way of doing it. And so that's a really great use case. And if you
[00:12:31.000 --> 00:12:35.240]   think about supply chain, if I look far enough back, it's like, oh my gosh, that's a graph again,
[00:12:35.240 --> 00:12:40.760]   right? And maybe I've been here for too long, but everything I look at is a graph now. It's like
[00:12:40.760 --> 00:12:47.160]   a graph graph. So it's always interconnectivity between anything and everything. So yeah,
[00:12:47.720 --> 00:12:55.080]   right. Well, into your point, so we're going beyond like, you know, what do I have on my DC,
[00:12:55.080 --> 00:13:00.040]   my distribution center, to like, what if kinds of questions? I mean, that seems powerful,
[00:13:00.040 --> 00:13:04.040]   right? Like, I mean, your piece has been around for a long time that it'd been able to sort of like
[00:13:04.040 --> 00:13:09.240]   try to get their arms around all this. And one could argue that they do or don't, right? You
[00:13:09.240 --> 00:13:14.600]   could have that argument. But I don't know that they're getting that deep into the what if modeling
[00:13:14.600 --> 00:13:20.440]   kind of things? And if so, it's probably pretty clunky. So that, that seems really cool. Like, how,
[00:13:20.440 --> 00:13:25.000]   how does that work? I mean, is there like a chat interface where someone's asking like, what if we
[00:13:25.000 --> 00:13:31.240]   shut down this warehouse? Or which, how should, how does that part go? Yeah, no, I mean, they weirdly
[00:13:31.240 --> 00:13:37.800]   knows it's like this question was planted. Is that we are actually working on our like UI interface
[00:13:37.800 --> 00:13:42.920]   to be able to answer some of these questions based off your data model. And this is where
[00:13:42.920 --> 00:13:49.400]   elements really play a huge part in our in our company right now because before, you know, you
[00:13:49.400 --> 00:13:55.240]   had to actually call some of these solver methods on top of your ontology or on top of your knowledge
[00:13:55.240 --> 00:14:00.120]   graph in itself and then get that result. And then you have an expert just kind of figure out what
[00:14:00.120 --> 00:14:07.240]   that result actually meant. Now we actually are creating sort of more of a platform where you have
[00:14:07.240 --> 00:14:13.080]   your knowledge graph. You can ask questions on top of it any dimensional type of semantics like
[00:14:13.080 --> 00:14:19.960]   historically what happened last year? You know, how many TVs did I sell in March of 2025, right? And
[00:14:19.960 --> 00:14:26.120]   so that's the kind of thing that we traditionally were able to do like from like Tableau and BI tools
[00:14:26.120 --> 00:14:30.840]   and things like that. You can use that same structure, that same knowledge graph to ask those
[00:14:30.840 --> 00:14:35.480]   previous questions. But now you can actually say, Hey, what do you think I'm going to sell
[00:14:35.480 --> 00:14:42.120]   based off of my historical knowledge in 2026? And how should I reallocate my supply chain
[00:14:42.120 --> 00:14:48.200]   based off of that projection? Now that to me, like it's those are really, really great questions
[00:14:48.200 --> 00:14:53.480]   that you can have like an interface like snowflake intelligence, open AI, cloud code, like because
[00:14:53.480 --> 00:15:01.560]   we are utilizing the MCP framework in order to expose these tools and these reasoners that we
[00:15:01.560 --> 00:15:11.400]   have already created years ago. So yeah, cool. Yeah, cool. So I can I can only imagine the extra
[00:15:11.400 --> 00:15:16.280]   advantage that you know, in terms of forecasting and projection and those kinds of things that
[00:15:16.280 --> 00:15:21.560]   you would have like you've got so many more paths open to you with with the proper knowledge graph
[00:15:21.560 --> 00:15:26.840]   instead of just sort of like relational data or BI, whether it's a lake house or an old school
[00:15:26.840 --> 00:15:35.800]   data warehouse or whatever. Okay, cool. Cool. So that's a lot of fun. Yeah. Okay. So how does this
[00:15:35.800 --> 00:15:41.720]   tie into the tools that we're all using? Like do you have a story there for us? Yeah, I think I have
[00:15:41.720 --> 00:15:46.520]   I have a couple of stories. So I think we have more of the external stories of when I sort of touch
[00:15:46.520 --> 00:15:52.600]   upon this a little bit of how we're utilizing AI, utilizing some of these tools that are now being
[00:15:54.360 --> 00:16:00.200]   accelerated is a great word like my gosh, even just looking back February of 2025.
[00:16:00.200 --> 00:16:07.480]   And so being available, changing with that is a huge thing for us for sure. We didn't have
[00:16:07.480 --> 00:16:11.560]   mold bot in January or maybe we did that. We did not. And I don't know if we still should,
[00:16:11.560 --> 00:16:22.120]   but yes, another good point. That's another podcast. But I do I do think so in terms of how we're
[00:16:22.120 --> 00:16:31.240]   using a tool more for customer adoption, I think has dramatically improved sort of like not only
[00:16:31.240 --> 00:16:35.800]   I would call it two different buckets, like sort of the analysts as an audience, like how do you
[00:16:35.800 --> 00:16:42.760]   Q&A on top of a knowledge graph and being able to utilize our LLMs and this chat function and
[00:16:42.760 --> 00:16:48.760]   these reasoning functions to expose through MCP is huge for us. So I think those are the tools that
[00:16:49.320 --> 00:16:56.040]   we're using within our product in order to drive more consumption, more engagement.
[00:16:56.040 --> 00:17:02.040]   And really give our audience like a holistic narrative, right? I don't have to jump to this
[00:17:02.040 --> 00:17:06.440]   pipeline or this pipeline or everything's kind of bundled up and I can just sit here and have a
[00:17:06.440 --> 00:17:11.560]   conversation with my knowledge graph about all with all these reason or capabilities and things
[00:17:11.560 --> 00:17:18.280]   like that. I think that's one aspect of it. I think in yeah, I think internally for our teams,
[00:17:18.280 --> 00:17:22.840]   we're using the tools, we've used them poorly, we're using them a little bit better now.
[00:17:22.840 --> 00:17:29.800]   And so being able to build this software has greatly dramatically changed as well. So just
[00:17:29.800 --> 00:17:36.760]   keeping up with everything and ensuring that our teams are understanding what this means for them,
[00:17:36.760 --> 00:17:43.000]   what is this means for the development lifecycle, etc. has been huge for us. So I'm happy to go
[00:17:43.000 --> 00:17:48.840]   to Madison. Yeah, yeah, yeah. Well, I mean, when you mentioned analysts and I've still got my head
[00:17:48.840 --> 00:17:55.240]   on supply chain from where we were, I guess I can imagine like a supply chain analyst sitting down
[00:17:55.240 --> 00:17:59.960]   with like a similar to the way I sit down with cloud code and say, well, by the way, this is something
[00:17:59.960 --> 00:18:04.680]   that I'm trying to get done. So I'm imagining that supply chain analyst doing a similar thing,
[00:18:04.680 --> 00:18:09.000]   you know, like we're trying to reduce our costs in such and such and we think it's the following
[00:18:09.000 --> 00:18:14.040]   factors, like what do you think? And it goes and says like, yes, and you know, so is that right?
[00:18:14.040 --> 00:18:18.600]   Am I getting it? Well, and I think about Q&A, is that how I should think about that? That's absolutely
[00:18:18.600 --> 00:18:23.960]   right. It can be just that simple, which is actually what we're aiming for. Because if we get
[00:18:23.960 --> 00:18:28.920]   complicated, then we lose people, right? It's like, wait, what do I need to do now? How do I get the
[00:18:28.920 --> 00:18:34.840]   answer, etc., etc. And so, you know, being able to have just that simple dialogue with,
[00:18:35.720 --> 00:18:41.320]   like, yeah, the interface to the knowledge graph is huge. Yeah, yeah, yeah, sort of start with
[00:18:41.320 --> 00:18:45.080]   our goals. And this is something we've learned. And maybe it's in your graph, maybe it's not,
[00:18:45.080 --> 00:18:49.000]   what can you add? How would you plan this? You know, what would, you know, what would, what are we?
[00:18:49.000 --> 00:18:56.040]   Oh, okay. Wow. Okay. That's really cool. Yeah. Okay. As you mentioned, you guys are using these
[00:18:56.040 --> 00:19:01.800]   tools in order to build the thing too, right? Yes. You and I, our background goes back to like
[00:19:01.800 --> 00:19:07.880]   making software, right? I can remember having meetings with you about like, you know, what, you know,
[00:19:07.880 --> 00:19:11.880]   what different pieces of tech that this tech lead was going to pick up and whether or not we had
[00:19:11.880 --> 00:19:17.080]   gotten it done. And you know, was the customer happy with us? And that was that's another story.
[00:19:17.080 --> 00:19:23.880]   But in the, so all of this stuff, like we, we didn't know these tools were coming, right? Back then.
[00:19:23.880 --> 00:19:29.160]   So from where you are, I don't know how much you can see, but to the extent that you've got stories
[00:19:29.160 --> 00:19:36.600]   about how teams inside of relational AI are using these tools, like, what are you seeing? Like,
[00:19:36.600 --> 00:19:41.560]   one of my sort of basic questions is like, do all the best practices stand? And I think for the
[00:19:41.560 --> 00:19:45.880]   most part, like, yes, testing. Oh my God. Like, probably we need more testing, not less.
[00:19:45.880 --> 00:19:51.960]   CI, you know, probably, but, but who cares what I think? Like, what are, what are you seeing in
[00:19:51.960 --> 00:19:57.400]   teams? When it comes to, what tools are people using? Let's kind of start there. What are, what are
[00:19:57.400 --> 00:20:02.520]   the favorite generative AI tools that people are actually using in work? Or what are you using?
[00:20:02.520 --> 00:20:10.280]   Well, you know, I use probably what my team and yeah, and our company uses, we are sitting
[00:20:10.280 --> 00:20:17.160]   square in the cloud code. We see the power behind their models and very developer centric, you know,
[00:20:17.160 --> 00:20:24.360]   pretty, pretty good. And it's interesting how I, if you asked me the six months ago, it could have
[00:20:24.360 --> 00:20:29.880]   been cursor, but as people jumped from jobs to job and cursor guy went back to cloud code,
[00:20:29.880 --> 00:20:34.200]   you know, we see cloud code getting. So I think it's, it's a race, right? And so we're,
[00:20:34.200 --> 00:20:40.280]   so I think the thing to think about is we're keeping an eye on what's the best models coming out.
[00:20:40.280 --> 00:20:45.400]   What are the, you know, the, the tools that make us the most efficient, but at the same time,
[00:20:45.400 --> 00:20:50.520]   we can't be mired in just keeping up with everything too. So there's a nice balance that we have
[00:20:50.520 --> 00:20:59.000]   to deal with as well. So I think that's one aspect of it. So, so yeah, we're using like, you know,
[00:20:59.000 --> 00:21:06.520]   IDs like VS code with cloud code plugins and things like that. I think when we actually look
[00:21:06.520 --> 00:21:11.880]   at the different agent technologies, we're looking at different agents that are helping us with
[00:21:11.880 --> 00:21:19.880]   story writing or agents that will help us with, uh, evolving works or specifically for us agents that
[00:21:19.880 --> 00:21:24.360]   help with modeling, right? Modeling is actually, I, I talk about this beautiful knowledge graph,
[00:21:24.360 --> 00:21:29.720]   but you gotta get there. You gotta model it into a knowledge graph, and that's not easy either. So
[00:21:29.720 --> 00:21:36.200]   being able to look at some of our agents and they can, you know, LOMs are really good at pattern
[00:21:36.200 --> 00:21:41.800]   matching, right? So if you give them a pattern, they're able to use that pattern and apply it,
[00:21:41.800 --> 00:21:45.880]   right? And especially with a human in the loop, you can sit there and validate and verify
[00:21:45.880 --> 00:21:52.840]   in all of these types of things. So we, we use them for various different things. Um, I will say,
[00:21:52.840 --> 00:21:58.760]   regardless of the tool of its cloud code, if it's cursor, if it's, uh, uh, any other model that we're
[00:21:58.760 --> 00:22:04.840]   using, um, one of the things I found, and I, and I know you'll appreciate this, uh, with our
[00:22:04.840 --> 00:22:11.480]   thought works background is our foundational's matter even more now. Like, and that, I don't know
[00:22:11.480 --> 00:22:17.960]   that surprised me or that just, that just sort of like came front and center. And I think like
[00:22:17.960 --> 00:22:24.760]   what I would call like this LLO move, right? Um, it just accelerated everything. And when you
[00:22:24.760 --> 00:22:31.160]   accelerate everything, then you see the outputs much faster. So just think about 10 years ago,
[00:22:31.160 --> 00:22:37.560]   if you have two different teams, one team doesn't test the other team tests, you don't really see
[00:22:37.560 --> 00:22:43.560]   that output very quickly. Like they have the same code, it's working, whatnot. But what we do see is
[00:22:43.560 --> 00:22:49.960]   years later, uh, which, which code base was, was sustainable, right? You know, and which code base
[00:22:49.960 --> 00:22:56.120]   turned into a big ball of mess. And so we could see that if we could sit there long enough to be
[00:22:56.120 --> 00:23:01.400]   there five years later, right? And I think like we have always talked about these foundational
[00:23:01.400 --> 00:23:06.520]   practices is like do this because in a long run, it's going to be good for you. And now with the
[00:23:06.520 --> 00:23:11.800]   acceleration of these tools, I'm saying do it now. And you're going to see the output pretty quickly
[00:23:11.800 --> 00:23:17.160]   if it works or if it doesn't work because of this acceleration. So I think foundations are,
[00:23:17.160 --> 00:23:21.800]   it's sort of proving us right, which I kind of love. It's like, oh, testing, we should be testing
[00:23:21.800 --> 00:23:26.920]   this all time. Yes, this is something I've been spouting for over two decades. You know, and
[00:23:26.920 --> 00:23:33.400]   someone's finally listening, you know, or how do you do spectrum in development? How do you actually
[00:23:33.400 --> 00:23:38.920]   look at those acceptance criteria? And, you know, everyone's talking about spectrum in development.
[00:23:38.920 --> 00:23:44.680]   How do you tell your agents to code appropriately? And it's like, oh, given when then, you know,
[00:23:44.680 --> 00:23:50.520]   acceptance criteria, things that give actual specification to the system. Crazy thoughts, right?
[00:23:50.520 --> 00:23:56.920]   So these again are practices that we talked about for years. But now really coming to fruition,
[00:23:56.920 --> 00:24:02.680]   which I'm really loving. And you're starting to see the differences of a foundational team and
[00:24:02.680 --> 00:24:07.720]   not a foundational team. So I think that's interesting. So to your point, like I definitely get
[00:24:07.720 --> 00:24:12.600]   the feeling that whatever you are, it's going to make you faster of what you are. So if you were
[00:24:12.600 --> 00:24:17.160]   putting messes into code bases, you're going to make messier code bases even faster, right?
[00:24:17.160 --> 00:24:20.600]   That's right. And if you are applying engineering rigor, then you're going to do that even faster,
[00:24:20.600 --> 00:24:26.440]   right? Yeah. So like that, that totally lines up with what you're saying. I've had a few people
[00:24:26.440 --> 00:24:32.600]   say, oh, you can't do red green refactor with it. I had one guest that went so far as to invent a
[00:24:32.600 --> 00:24:38.200]   framework that allowed him to do red green refactor, Jeff Langer talking about his AADV framework.
[00:24:38.200 --> 00:24:43.480]   But nobody's saying like, oh, we don't need unit tests anymore, right? It's just that that
[00:24:43.480 --> 00:24:50.440]   specific loop may be more challenging. Yeah. I think one of the other questions that I've got
[00:24:50.440 --> 00:24:55.880]   for folks though is sort of like, what's it doing to team makeup, right? Like, are we going to need
[00:24:55.880 --> 00:25:00.440]   as many developers as we did? Like, like, there used to be sort of a ratio, right? And you've got a
[00:25:00.440 --> 00:25:06.280]   BA and, you know, call it three pairs of software developers and maybe a QA person who's there for
[00:25:06.280 --> 00:25:11.000]   like strategic support, but they're not building the code base that developers are. And that was like
[00:25:11.000 --> 00:25:16.680]   the stamp that you could just like stamp teams out. Never exactly true. But roughly, you know,
[00:25:16.680 --> 00:25:20.680]   that was something that was repeatable. You know, maybe you could do four pairs with the BA.
[00:25:20.680 --> 00:25:24.680]   Maybe if you were up to five, six pairs, you might have needed two BA's. Maybe maybe a junior
[00:25:24.680 --> 00:25:30.040]   QA to help out the main QA, but, you know, that that was kind of the ratio. Is that changing now? Yeah.
[00:25:30.040 --> 00:25:39.320]   Yeah. I, I, I thought about this question quite a bit. And I do think typical consultant raises,
[00:25:39.320 --> 00:25:47.960]   it depends for sure. I think, but I will, I will follow up with some thoughts here. I think in
[00:25:47.960 --> 00:25:54.600]   Greenfield, like in a Greenfield project and things like that, if there are strong, foundational
[00:25:54.840 --> 00:26:01.320]   rigor that is either embedded into the process or embedded into the agents, if you will. Like,
[00:26:01.320 --> 00:26:06.760]   I'm finding, at least this is a observation right now, is that like boiler plates, very
[00:26:06.760 --> 00:26:13.000]   task-oriented type of things can actually be handled by the LMS. And I think if you direct it well
[00:26:13.000 --> 00:26:19.400]   enough and get a repeatable structure involved, then you're saving a lot of time. You're saving a lot
[00:26:19.400 --> 00:26:25.080]   of effort, right? And so, do you have a team that does a lot of boiler plate? Does that do a lot
[00:26:25.080 --> 00:26:30.120]   task-based type of things? I think you can actually make your team way more efficient by replacing
[00:26:30.120 --> 00:26:35.240]   those things with componentized sort of task-oriented agents that will do these repeatable things
[00:26:35.240 --> 00:26:43.320]   that are more deterministic and that you can control in a way, right? So, I don't think that
[00:26:43.320 --> 00:26:49.880]   removes the architecture, the design, the, you know, the good practice, like people have to install
[00:26:49.880 --> 00:26:54.920]   the good practices, right? And I have to ensure that the models are learning from that type of
[00:26:54.920 --> 00:27:01.800]   foundational practices and things like that. So, I would say that the makeup of the team,
[00:27:01.800 --> 00:27:06.920]   like these functions that you called out, like the analysts, the product manager, the developer,
[00:27:06.920 --> 00:27:12.360]   the architect, the QA, like those functions still exist. We need to have them. Whether they're
[00:27:12.360 --> 00:27:19.400]   represented in an agent or human or whatnot or a combination of all of those things. I think
[00:27:19.400 --> 00:27:24.280]   that's absolutely still necessary. I think the ratio may change. I don't know what that ratio is
[00:27:24.280 --> 00:27:29.480]   because I think it definitely depends on, are you dealing with this entire legacy code base? Because
[00:27:29.480 --> 00:27:33.720]   the functions are a little bit different if you're trying to reverse engineer things, write tests
[00:27:33.720 --> 00:27:38.120]   on top of things that never had tests on, and then rewrite that code using the agents. That's
[00:27:38.120 --> 00:27:44.040]   as a whole other team makeup that is very different than I am sitting here in a green field world,
[00:27:44.040 --> 00:27:48.280]   trying to build an interesting app and whatnot. And then there's some, you know,
[00:27:48.280 --> 00:27:55.640]   building STLC to kind of things that you can actually automate and things like that. And then,
[00:27:55.640 --> 00:28:00.840]   of course, as you know, there's always something in the middle, right? So there are some legacy,
[00:28:00.840 --> 00:28:04.840]   some not, and whatnot. So how do you decipher between all of those types of things? And I think
[00:28:04.840 --> 00:28:09.640]   human in the loop is the answer still to this day. Would that change in future? I don't know,
[00:28:09.640 --> 00:28:18.600]   but today still need a humans. 100% yeah. And like you, I do wonder if it's going to kind of change
[00:28:18.600 --> 00:28:25.880]   the kinds of humans. Like I wonder if we'll be rewarded by we, we pay a lot of my smart friends
[00:28:25.880 --> 00:28:30.920]   have always talked about like having a T shape person, just meaning the top of the T you have some
[00:28:30.920 --> 00:28:37.240]   breath, but then the tail of the T you also have some depth in something, right? Because if you
[00:28:37.240 --> 00:28:41.880]   don't have any depth in anything, then you don't like really understand how important depth can be,
[00:28:41.880 --> 00:28:49.480]   right? But you also can't have all of the depths, right? Like you are great at Python and Polars,
[00:28:49.480 --> 00:28:53.800]   I am better at getting Django to work right, you know, we're both of us are useful, you know,
[00:28:53.800 --> 00:29:01.800]   in the end, I feel like this is kind of amplifying that too, right? Like it quad code can be really good
[00:29:01.800 --> 00:29:06.360]   at Polars. Like do I need to be good at Polars? Like it's still good to have somebody on the team
[00:29:06.360 --> 00:29:10.920]   that has it though. So I don't know, like I feel like it's changing that makeup a little bit too,
[00:29:10.920 --> 00:29:16.360]   that it can bring some of the technical expertise is anyway that we used to have to instill in people
[00:29:16.360 --> 00:29:23.640]   or embed in people. Yeah, sure, sure. I love that you brought up the T shaped or I think it was
[00:29:23.640 --> 00:29:28.760]   Kent Beck who did the tear shape, like the paint drips, right? Like because there's many different
[00:29:28.760 --> 00:29:34.120]   levels of depths and things like that. I was actually just talking about this a couple of weeks ago
[00:29:34.120 --> 00:29:41.960]   with a few folks like Kent Beck and Martin around. How do yeah, how do we actually look like you do?
[00:29:41.960 --> 00:29:49.880]   We had a really nice conference. It was a very, it was a very, I was very lucky to be invited to
[00:29:49.880 --> 00:29:57.480]   such a thing, but it was very much a discussion around which parts of the the paint drops will the
[00:29:57.480 --> 00:30:03.080]   agents be able to take because they are repeatable paint or really these are things that happen
[00:30:03.080 --> 00:30:08.040]   all the time. And then there's these other depth pieces with there's innovation, there's
[00:30:08.040 --> 00:30:12.600]   experimentation, there's new algorithms that we still need humans to really understand and design
[00:30:12.600 --> 00:30:18.040]   and things like that. And that depth I think doesn't go away. And I think about this in terms of my
[00:30:18.040 --> 00:30:23.640]   company where we have prescriptive analytics folks, we have predictive analytics folks, we have
[00:30:23.640 --> 00:30:28.680]   these people that we still need and not even still need very much need because that's our differentiator
[00:30:28.680 --> 00:30:35.320]   right there, right? And so that differentiation sure an agent can help with sort of the infrastructure
[00:30:35.320 --> 00:30:42.760]   around it, but there's some key pieces that I would not leave to the LLMs to solve for us or
[00:30:42.760 --> 00:30:48.680]   then there is no differentiator in this world, right? So, but I do think that there are like
[00:30:48.680 --> 00:30:55.080]   patterned type things, these boilerplate type of things where we can hand those over so we can
[00:30:55.080 --> 00:31:00.600]   unload our cognitive load to that, which we don't want to hold in our brains and actually focus
[00:31:00.600 --> 00:31:05.320]   our cognitive load on the thing that matters. Like how do you actually design the system? How do
[00:31:05.320 --> 00:31:10.120]   you actually talk to the business and figure out what they actually want versus what we're assuming?
[00:31:10.120 --> 00:31:15.560]   You know, like going back to the human aspect of it, so. Yeah, 100 percent, 100 percent. Love that.
[00:31:15.560 --> 00:31:21.000]   Okay, so you mentioned, I don't know if you mentioned Spectro by name, but you talked about like
[00:31:21.000 --> 00:31:25.640]   writing stories and you also talked about the importance of having like acceptance criteria
[00:31:25.640 --> 00:31:31.800]   you're given when then, you know, kinds of things. Like, how can you talk more about how it's
[00:31:31.800 --> 00:31:37.000]   impacting product management and story breakdown, right? Like, and I know in my stamp it out,
[00:31:37.000 --> 00:31:41.560]   analogy, I talked about BAs, but not product managers. Obviously, that's been another big change
[00:31:41.560 --> 00:31:47.800]   over the last sort of 5-10 years. Like, how are you guys doing all of this? Like, do you find
[00:31:47.800 --> 00:31:52.680]   that they're still like a product team and an engineering team and are the tools they're using
[00:31:52.680 --> 00:31:57.720]   different or the processes? Are they using agents to write PRDs and are we using agent agents to
[00:31:57.720 --> 00:32:03.320]   read them? Like, how is all that going in terms of the product management and story breakdown?
[00:32:03.320 --> 00:32:08.200]   Yeah, sure, sure. I think, I think for us, the way that, and again, I think it'll differ for
[00:32:08.200 --> 00:32:13.000]   any other organization, so it sort of depends on what you guys are used to, but for us,
[00:32:13.000 --> 00:32:21.480]   we use our coding tools as still assistance, right? Like, it helps us open up the ideas, right? So,
[00:32:21.480 --> 00:32:27.160]   from a product management point of view, sure, we have these hypotheses, we have these, you know,
[00:32:27.880 --> 00:32:34.680]   theories about how, you know, these things can do certain things. But what we would do is we say,
[00:32:34.680 --> 00:32:41.640]   "Hey, LLM, go look around at the competitive landscape and give us some ideas on what our
[00:32:41.640 --> 00:32:45.480]   competitors are right now." And we may have an idea of what they are, but we may not have the
[00:32:45.480 --> 00:32:50.920]   full spectrum. And the beautiful part about our LLMs is, boy, they can they scour the internet
[00:32:50.920 --> 00:32:59.160]   websites. Yeah, much faster than we can. And much deeper. So it's good to utilize that to say,
[00:32:59.160 --> 00:33:03.000]   "Okay, give me the competitive landscape." And of course, there's hallucinations and things
[00:33:03.000 --> 00:33:07.800]   like that, but as long as you have a product manager siphoning through this and using it more
[00:33:07.800 --> 00:33:13.000]   of a source and understanding what we're trying to accomplish, I think I think I'm seeing
[00:33:13.000 --> 00:33:18.680]   our product management team being a lot more strategic and thinking more externally into, like,
[00:33:18.680 --> 00:33:24.280]   how our products can live in this world versus internally, oh gosh, how do we deliver this particular
[00:33:24.280 --> 00:33:30.120]   feature or function or whatnot, right? So I think I think I'm seeing that as a shift. And then,
[00:33:30.120 --> 00:33:35.880]   of course, like, once you boil down that product roadmap into specific feature sets and things
[00:33:35.880 --> 00:33:43.880]   like that, then again, it's not that we don't write the PRDs or the stories or whatnot, but we
[00:33:43.880 --> 00:33:49.800]   have the LLMs help us. And as I do as an architect, I'm like, "Okay, this is my assumption. This is
[00:33:49.800 --> 00:33:54.440]   what I'm going to write down. And this is what I want to give to my teams. I'll go check me in my
[00:33:54.440 --> 00:33:59.400]   missing anything." And I have it, like, I'm human, like, I'm like, "Oh, I forgot about that security
[00:33:59.400 --> 00:34:05.960]   thing. I should probably touch upon that." So again, it's just back and forth in trying to use it
[00:34:05.960 --> 00:34:11.240]   as another resource, right? Because the very powerful resource. So I think it's, it behooves us not
[00:34:11.240 --> 00:34:17.480]   to use them, but also you get in trouble if you rely too much on them, because we have to rely
[00:34:17.480 --> 00:34:22.920]   on our own expertise to make decisions based off of, like, how we siphon through these sources. So, yeah.
[00:34:22.920 --> 00:34:27.320]   Yeah. I love what you're saying about having it review work, though. Like, one of the great ironies
[00:34:27.320 --> 00:34:32.040]   of all of this is that one of the best ways to figure out, or if an LLM has steered you wrong,
[00:34:32.040 --> 00:34:38.760]   is to ask an LLM to check its work. And, yeah, that's just the world we're living in. Yep. Yeah.
[00:34:38.760 --> 00:34:44.280]   Okay. So, and you touched on how architecture and planning have changed a little bit, right?
[00:34:44.280 --> 00:34:49.560]   Like, having having an LLM review and architectural plan, what more can you say about that? Like,
[00:34:49.560 --> 00:34:56.280]   what else are you seeing there? Yeah, oh gosh. So much. But I'll try to boil it down a little bit,
[00:34:56.280 --> 00:35:02.440]   I think, back in the day, like, in the way that, you know, we've sort of thought about the agile
[00:35:02.440 --> 00:35:07.880]   iterative approaches. Like, okay, let's do some test-driven development. Let's start somewhere,
[00:35:07.880 --> 00:35:13.400]   and then let's actually start writing this test, writing the feature, refactoring, and that,
[00:35:13.400 --> 00:35:20.440]   in its own right, helps us design the code base, right? And again, I might say something a little
[00:35:20.440 --> 00:35:24.840]   bit controversial, because, you know, I truly believed in refactoring all the time, like, the
[00:35:24.840 --> 00:35:30.040]   refactor, make your code base cleaner, and things like that. I think what may be changing right now,
[00:35:30.040 --> 00:35:34.840]   and I don't, I've yet to prove this yet, but I'm trying to, I'm thinking about what this looks like,
[00:35:34.840 --> 00:35:40.680]   is code is cheap now. These elements can bust out these though, like, very, very quickly. I've
[00:35:40.680 --> 00:35:48.120]   talked to a few people again last week around how someone vibe-coded a bunch of stuff. But one
[00:35:48.120 --> 00:35:52.600]   thing that they did do is they wrote a bunch of tests. They wrote the tests in order to figure out
[00:35:52.600 --> 00:35:58.760]   what the specs are. They were able to blow out their entire code base and rewrite into a new language,
[00:35:58.760 --> 00:36:05.640]   as long as they have the tests in place, and it's refactoring dead. I don't know, like, I get,
[00:36:05.640 --> 00:36:11.640]   like, this is very controversial, but it, like, code is cheap, right? And so it goes back to,
[00:36:11.640 --> 00:36:20.840]   how do we change the way we think about, you know, yeah, test, test the red, green, red,
[00:36:20.840 --> 00:36:24.920]   green, red green, refactor. Red green refactor, yeah. All of those kind of things, I think, it doesn't
[00:36:24.920 --> 00:36:29.800]   go away. I think it changes how we think about it. So we are looking at, like, how do we actually
[00:36:29.800 --> 00:36:36.440]   refactor the tests? Like, how do we actually look and make sure that those specs are the gold plate,
[00:36:36.440 --> 00:36:41.880]   right? And so now that, you know, code is commoditized a little bit more now. So what does that look
[00:36:41.880 --> 00:36:46.760]   like in terms of your architecture, your design, your infrastructure? So again, I don't have the
[00:36:46.760 --> 00:36:51.480]   answers, but I have the questions for sure. And we're trying to prove some of these things out. So,
[00:36:51.480 --> 00:36:57.640]   yeah, I'm, I'm interested in where you're going very much so, right? Like, so Pete Hodson was on,
[00:36:57.640 --> 00:37:03.000]   and he, he talked a little bit about how, like, the carrying cost of code is still real. Like, you
[00:37:03.000 --> 00:37:08.600]   can't just like write too much code and have it not matter. And I, and I think that's true. And also,
[00:37:08.600 --> 00:37:14.280]   Jeff Langer, when he was on, he was talking about how, like, maybe the goal isn't so much, like,
[00:37:14.280 --> 00:37:20.280]   getting perfect code, but getting a really great test base that helps the agency what code to write,
[00:37:20.280 --> 00:37:24.120]   so that you could almost just rewrite the code base. You know, we, we used to talk about
[00:37:24.120 --> 00:37:29.560]   architectures that could deal with, replace them in, of any component really, really easily. Like,
[00:37:29.560 --> 00:37:35.000]   that you should design for, like, what did we call it? Like, the, the, disposability, like, being,
[00:37:35.000 --> 00:37:39.400]   like an important architectural fact. Like, yeah, this thing won't list forever, right? So we
[00:37:39.400 --> 00:37:43.160]   should build it with that in mind. And, and now I'm wondering if that's shifting like you're saying,
[00:37:43.160 --> 00:37:49.800]   to code, like, where if you, if you had a good enough spec, you could just rebuild it. So, yeah,
[00:37:49.800 --> 00:37:53.400]   it's interesting. Like, but as you say, that is not yet proven that that would work.
[00:37:53.400 --> 00:38:01.960]   Yeah. And the counter are human to this. Again, not a few conversations, even Jess Humboldt said this,
[00:38:01.960 --> 00:38:08.760]   like, these LLMs still don't know how to debug the code base very well, right? And so you go
[00:38:08.760 --> 00:38:13.960]   back to the Amazonian thought process of, like, you build it, you run it, can the LLMs build it?
[00:38:13.960 --> 00:38:19.400]   Yes, they're building something, and they run it, and they support it. Would you trust an LLM to
[00:38:19.400 --> 00:38:25.320]   be on call for this? And so I do believe that they may come, but today it's not quite there yet.
[00:38:25.320 --> 00:38:30.920]   So I think we need to remember that as well, right? So I'm with you. A colleague used something
[00:38:30.920 --> 00:38:37.080]   I had not heard earlier this week. I guess there's an old IBM quote about, like, yes, computers can,
[00:38:37.080 --> 00:38:44.360]   can do a lot of things, but they can't be responsible for the decisions yet. And I've done cuts,
[00:38:44.360 --> 00:38:49.720]   that's still true. Yeah, 100%. Absolutely. Okay. So we talked about product management,
[00:38:49.720 --> 00:38:54.200]   architecture. We've gotten into software development. We've talked about testing a little bit, like,
[00:38:54.200 --> 00:38:59.800]   in my own personal experience, and listeners are probably tired of me saying this now, but I don't
[00:38:59.800 --> 00:39:04.280]   get to use this stuff at work, which in this remains true. I've been now saying this for months,
[00:39:04.280 --> 00:39:11.000]   but I do get to use it a lot in a side project. So this side project, it's just me,
[00:39:11.720 --> 00:39:17.960]   and it's not pre-revenue, but it's sort of in that zero to one phase and definitely pre-market set,
[00:39:17.960 --> 00:39:24.120]   right? So a lot of the way I work with it is very much like I sit down and talk to it, like,
[00:39:24.120 --> 00:39:29.640]   I am a technical product manager, right? And I try to break things down into something that's
[00:39:29.640 --> 00:39:35.320]   fairly small, right? In order to kind of manage context. But in the end, I'm usually doing the
[00:39:35.320 --> 00:39:41.320]   testing, right? Like, I don't, maybe I'm missing it, like, maybe, I'm probably, it's probably something I
[00:39:41.320 --> 00:39:47.320]   need to add. Now, I have, I ask it to write a lot of unit tests, and I review those unit tests,
[00:39:47.320 --> 00:39:52.040]   and I make sure that those unit tests cover what I want them to cover. But that's, but really,
[00:39:52.040 --> 00:39:57.720]   I'm still doing a lot of the testing though, right? Like, I don't know who is testing. Yeah. Yeah.
[00:39:57.720 --> 00:40:02.840]   Like, if I just asked it to go change an important page on the website, I'm going to look at it with
[00:40:02.840 --> 00:40:08.520]   my eyeballs, right? Where we are not going to deploy it until that has happened. And, but there's
[00:40:08.520 --> 00:40:12.200]   this other part of me, which is like, well, Kyle, we've been talking about, like, good functional
[00:40:12.200 --> 00:40:16.200]   suites forever, and you shouldn't have to do that. And, but I don't know, I feel like I'm testing
[00:40:16.200 --> 00:40:21.320]   more now with this stuff, not less, but I can't tell. Maybe this is just sort of like me and my side
[00:40:21.320 --> 00:40:27.320]   project, and that's the approach. What are you seeing? Like, is, are we, like you said, the,
[00:40:27.320 --> 00:40:32.600]   the paint trips, some of those were going to be able to get an agent to do when, when you have larger
[00:40:32.600 --> 00:40:37.160]   teams than just Kyle, are you, are you finding that like that is working, where you can get the
[00:40:37.160 --> 00:40:43.960]   agents to test? Or yeah, tell us more about that. I think, yeah, I, uh, this is a good question,
[00:40:43.960 --> 00:40:49.080]   because I think one of the things that we're seeing is that people are moving a lot faster,
[00:40:49.080 --> 00:40:54.520]   but we're not validating fast enough, right? And then the models are changing at a rapid pace.
[00:40:54.520 --> 00:40:59.640]   So it doesn't surprise me that you're like, I'm going to test more, right? Because there's still
[00:40:59.640 --> 00:41:05.320]   a bit of a lack of trust in the LLMs to get it right. They are non-deterministic, which means to us,
[00:41:05.320 --> 00:41:11.640]   like, I can't have a repeatable answer, right? To this kind of thing. I think the way that we're
[00:41:11.640 --> 00:41:17.400]   trying to approach it is, and I alluded to this earlier, is that if you have more componentized
[00:41:17.400 --> 00:41:22.360]   bite-sized types of things, the LLMs do much better with the context windows and things like that.
[00:41:23.080 --> 00:41:28.520]   And so we still are the human and loops that need to break these things down, right? So I talked
[00:41:28.520 --> 00:41:34.280]   a little bit earlier about how do you actually have task-based agents? So instead of talking to your
[00:41:34.280 --> 00:41:39.800]   agents and say, go do all the things. Please go write a bunch of code and finish this feature.
[00:41:39.800 --> 00:41:47.320]   Let's go back to our, you know, STLC, right? Like, agent, you are only responsible for writing the
[00:41:47.320 --> 00:41:54.120]   story in this pattern, write the story, push it to JIRA. You agent are only responsible to your point,
[00:41:54.120 --> 00:41:59.720]   is actually writing the unit test, and then testing that unit test. That's your only responsibility.
[00:41:59.720 --> 00:42:05.240]   So I'm starting to see us a little bit more of an orchestration layer to the agents and humans,
[00:42:05.240 --> 00:42:10.680]   and trying to figure out what all of this actually looks like. I do feel that we are testing more,
[00:42:10.680 --> 00:42:16.360]   because there's just this inherent mistrust that it's going to get around. Even if it's right,
[00:42:16.920 --> 00:42:23.480]   99% of the time that 41% of the time is what we're anxious about. I think it's ironic, because if you
[00:42:23.480 --> 00:42:30.680]   put humans going back to like devs writing code and stuff like that, it's not 99% time. It's like 80%
[00:42:30.680 --> 00:42:36.840]   time they get it right. So it's interesting how our brain and how our trust factor in getting in
[00:42:36.840 --> 00:42:43.240]   there. So I believe there will be less testing on our part going forward. One C systems become
[00:42:43.240 --> 00:42:49.560]   a little bit hardened, and so we actually see that system hardened within our own ecosystem.
[00:42:49.560 --> 00:42:54.680]   But I hear you, because I have for us, I was like, check everything, check everything that LLM did.
[00:42:54.680 --> 00:43:02.360]   Well, yeah, and I'm probably not describing how I'm actually, my actual behavior very well.
[00:43:02.360 --> 00:43:08.680]   I think what I'm maybe not doing, spending enough time trying to write the story so well,
[00:43:08.680 --> 00:43:16.440]   that I can trust that it will have had a chance to get it right. On some level, the mistrust
[00:43:16.440 --> 00:43:21.320]   is not so much that it will have hallucinated in an alternate universe. So that still happens,
[00:43:21.320 --> 00:43:27.640]   it's a lot rarer now. But it's more like that I even describe it well. I asked for a UI change,
[00:43:27.640 --> 00:43:33.640]   like I want to see it. I don't want to, I presume this will make it more the experience better,
[00:43:33.640 --> 00:43:41.240]   but will it make the experience better? I'm almost checking my own work that it did for me
[00:43:41.240 --> 00:43:47.720]   in a way. And I think we're like vibing a little bit because I should maybe, maybe, and tell me
[00:43:47.720 --> 00:43:51.240]   if I'm right or wrong, it's like maybe you feel like you're testing more because the feedback loop
[00:43:51.240 --> 00:43:56.840]   is just so much quicker now. Yeah, previously, like changing that font on that thing, like maybe
[00:43:56.840 --> 00:44:02.360]   takes many more hours or even a day, whereas it takes a minute now, and you're like, oh,
[00:44:02.360 --> 00:44:07.880]   let me look at it, right? And so again, time is cheap here to do. So that feedback loop is maybe
[00:44:07.880 --> 00:44:12.600]   making us feel like we're testing a lot more because we are. We're testing faster. It's coming
[00:44:12.600 --> 00:44:20.920]   back to us faster. Yeah, because we're deploying more. I love it. I love it. Yeah. So when I say
[00:44:20.920 --> 00:44:28.280]   integrating software, what comes to mind for you with yellow limbs? To me, this is like more of our
[00:44:28.280 --> 00:44:36.440]   path to prod and integrating into actual real systems and things like that. I'll tell you that I'm
[00:44:36.440 --> 00:44:41.880]   I'm still seeing much more human in the loop integration. If you're doing again, Greenfield,
[00:44:41.880 --> 00:44:46.520]   I think that's easier. There's patterns and things like that. But in enterprise, we're seeing
[00:44:46.520 --> 00:44:51.400]   a million different types of pipelines, right? And in many different types of ways that people are
[00:44:51.400 --> 00:44:57.480]   doing things, there's a lot of like DevOps heavy lifting that we're still doing in order to like,
[00:44:57.480 --> 00:45:04.440]   I would say, integrate into the production pipeline, go through the still same compliance
[00:45:04.440 --> 00:45:09.800]   and security guard rails and things like that. Some of those things will be there for a long time,
[00:45:09.800 --> 00:45:15.720]   right? Because that's that's how enterprises run and and and think about things. I think the guard
[00:45:15.720 --> 00:45:22.280]   rails that they have in place may be antiquated and old and it's at the very, very end. Well,
[00:45:22.280 --> 00:45:27.480]   we couldn't do like as you know, going futuristic is like thinking about taking not getting rid of
[00:45:27.480 --> 00:45:31.880]   guard rails, but taking these guard rails and pulling them closer. Since we have a faster feedback
[00:45:31.880 --> 00:45:39.080]   loop now, how do we pull this in and integrate faster, right? And so that's sort of where my head
[00:45:39.080 --> 00:45:43.160]   is that we're not there, right? We're still integrating at the end with all of these different
[00:45:43.160 --> 00:45:47.720]   enterprises right now. But I think the next step to thinking about this is how do we enable
[00:45:47.720 --> 00:45:52.760]   the enterprises to move these things a little bit to the left if you all shift left. Yeah,
[00:45:52.760 --> 00:45:57.000]   yeah, 100%. I think that's right. I think that's a good way to think about it. The other thing that
[00:45:57.000 --> 00:46:02.680]   comes up for me with integration is this sort of like the context window limits like how much I can
[00:46:02.680 --> 00:46:07.160]   trust it with like the rest of the code base some days, it'll think, oh, there's a method over
[00:46:07.160 --> 00:46:11.160]   here that I should use. And it's like, no, no, that's not what that method does. And then other days,
[00:46:11.160 --> 00:46:15.800]   it ignores the method that it should be using. And it's like, so it's like, I trust it pretty well
[00:46:15.800 --> 00:46:20.520]   with whatever code that it wrote during the context session. But it forgot the code I wrote
[00:46:20.520 --> 00:46:26.360]   yesterday entirely. So I'm the literal act of sort of like integrating in terms of like making
[00:46:26.360 --> 00:46:31.080]   sure that the code base is cohesive. And you don't end up with like the same calculation repeated
[00:46:31.080 --> 00:46:36.040]   a hundred times. But like that we're sort of reusing things as a library. Once I can tell it,
[00:46:36.040 --> 00:46:40.120]   oh, you need to do this. It's pretty good at doing it. Yeah. But it doesn't tend to think of that,
[00:46:40.120 --> 00:46:45.000]   you know, and it and it doesn't do a great job of realizing like, oh, I should use that
[00:46:45.000 --> 00:46:49.880]   microservice that we already built. It'll just start putting it into what it's doing today. Like,
[00:46:49.880 --> 00:46:57.800]   oh, I can just build it so quickly. And that's a good point. It can, right? Yeah. It can.
[00:46:57.800 --> 00:47:04.600]   Yeah. But I mean, that goes back to modeling. That goes back to us humans and the group actually
[00:47:04.600 --> 00:47:10.520]   show what does a good domain driven design architecture actually look like? And how do you feed that?
[00:47:10.520 --> 00:47:16.680]   Also, that piece of information into the LLM as well. Like, yeah, I think about these agents again,
[00:47:16.680 --> 00:47:22.600]   being a task specific. You can think about some of the agents being doing specific as well,
[00:47:22.600 --> 00:47:26.520]   right? And trying to figure out what that actually looks like. So I don't know.
[00:47:26.520 --> 00:47:30.920]   Oh, nice. I like that. I like that. Well, and to your point about modeling, like,
[00:47:30.920 --> 00:47:35.960]   it's if you remember to ask it to help you with designing a model, it'll probably be really
[00:47:35.960 --> 00:47:39.800]   helpful. Yeah. And if you ask it to critique the code base and see if it follows that model,
[00:47:39.800 --> 00:47:44.280]   it can probably tell you. Yeah. But it won't necessarily remind you to do that, right?
[00:47:44.280 --> 00:47:49.800]   Like, yeah. Yeah. And there's that human in the loop bit. Yeah. Keep keeps coming back to that.
[00:47:49.800 --> 00:47:56.200]   Don't we? Okay. Okay. So you mentioned DevOps a few times and you talked about path-to-prod
[00:47:56.200 --> 00:48:03.480]   infrastructure. Do you see Gen I changing how Gen AI changing how infrastructure works? Yeah. Yeah.
[00:48:03.960 --> 00:48:10.920]   And going back to the same pipelines and sort of task-based type of things.
[00:48:10.920 --> 00:48:17.640]   And I'm not speaking from experience here, but well, maybe I am a little bit like,
[00:48:17.640 --> 00:48:24.120]   but they only let me do POCs. They don't let me push a prod anymore. But but one thing I will say is
[00:48:24.120 --> 00:48:30.200]   being able to spin up like a Docker container and throw it up into snow park container services
[00:48:30.200 --> 00:48:36.600]   quickly for me as an non infrastructure person, that was pretty easy. That's a deal.
[00:48:36.600 --> 00:48:41.880]   And to deploy and to figure out like what those endpoints look like and make them public and Lola
[00:48:41.880 --> 00:48:48.040]   and all this stuff. I think there's enough work out there that LLM see those patterns of how to
[00:48:48.040 --> 00:48:52.120]   actually deploy these kind of things. Do we need to check them a thousand percent? For sure. We
[00:48:52.120 --> 00:48:59.000]   need to check our pipelines. But it still sits in the bucket for me that it is quite repeatable.
[00:48:59.000 --> 00:49:05.640]   And eventually I think these things be very important. We'll have a lot of help from the LLM
[00:49:05.640 --> 00:49:11.320]   to help us go faster on creating some of our pipelines. So yeah. Yeah. I think I'm right where you
[00:49:11.320 --> 00:49:15.720]   are on that. Like I you're you're reminding me of my startup work where like yeah,
[00:49:15.720 --> 00:49:19.800]   it can make a Docker file for me faster than I can. And it can write the Terraform to get that
[00:49:19.800 --> 00:49:25.960]   online faster than I can. And like cool, do it. I'm pretty sure that if like a proper like AWS
[00:49:25.960 --> 00:49:29.560]   specialist looked at all my Terraform, they'd be like Kyle. This is what the heck. I mean,
[00:49:29.560 --> 00:49:38.040]   it works. But right. Yeah. Yeah. Yeah. So yeah. So there's maybe an element of like it can do it
[00:49:38.040 --> 00:49:43.080]   for us. And that is good. But like anything else human in the loop. Yeah. I could probably even ask
[00:49:43.080 --> 00:49:48.600]   it to critique itself. Yeah. Yeah. Yeah. Is that going to be the answer for everything? Oh my gosh.
[00:49:50.520 --> 00:50:01.880]   And then we'll go retire on a beach somewhere. I love it. Final. Yeah. Okay. Cool. Cool.
[00:50:01.880 --> 00:50:08.680]   So what about how you hire staff, trained mentor? Like how you choose people? What changes
[00:50:08.680 --> 00:50:15.080]   are you seeing there? I would say like I'm the first person to admit we've we've missed
[00:50:15.080 --> 00:50:23.240]   out quite a few times. We're trying to figure it out like everybody else. I'll share some thoughts
[00:50:23.240 --> 00:50:30.440]   around this. I very much was like, okay, my teams are just going to be so much more efficient.
[00:50:30.440 --> 00:50:39.320]   Like it's going to be crazy. And no, I still think you need to going back to like just a core. You
[00:50:39.320 --> 00:50:46.360]   need to still hire for aptitude and attitude. Definitely. And for me, obviously, integrity. Right.
[00:50:46.360 --> 00:50:52.920]   So I think the aptitude and attitude is super important when it come when it came down this route.
[00:50:52.920 --> 00:51:00.040]   Right. I would have a lot of like sort of senior been around for a long time kind of engineers.
[00:51:00.040 --> 00:51:08.920]   I don't care. I like this is like this is my thing. You know, and all this kind of stuff.
[00:51:08.920 --> 00:51:15.000]   So to me, that's lacking attitude and, you know, really embracing new tools and technologies coming
[00:51:15.000 --> 00:51:21.880]   through. I've had, you know, engineers who I hired with maybe not the aptitude and they're like,
[00:51:21.880 --> 00:51:27.560]   well, LM's going to do everything for me. Right. And boy, did I just get LM slot? I just got AI
[00:51:27.560 --> 00:51:32.280]   sloped like just terrible and worse. Like I'm like, just throw that out. I can't even I can't
[00:51:32.280 --> 00:51:38.920]   even use that, right. And so I think it goes back to hiring for the attitude aptitude and
[00:51:38.920 --> 00:51:45.240]   integrity is huge for me now because we need people who can adapt much more quickly like your job
[00:51:45.240 --> 00:51:51.240]   that was last year may not be your job this year. And you need to evolve to that. So being able to
[00:51:51.240 --> 00:51:59.160]   have that flexibility is huge for us. The other bit is really around being able to learn like being
[00:51:59.160 --> 00:52:03.800]   able to actually learn on the job and figure and just not be the smartest person in the room.
[00:52:03.800 --> 00:52:09.720]   Maybe the LM is the smartest person room today and being okay with that as well. So there's a bit
[00:52:09.720 --> 00:52:13.560]   of that that I've been dealing with when I'm training and hiring and all of these kind of things.
[00:52:13.560 --> 00:52:19.160]   And at the very end, it sort of circles back to what we talked about in the very beginning. It's
[00:52:19.160 --> 00:52:26.360]   those foundational mindset, right. Like I have found that I don't mean to brag, but like I have
[00:52:26.360 --> 00:52:32.760]   pretty good foundations and I get to code again because it's like, okay, like I don't need to
[00:52:32.760 --> 00:52:39.000]   learn a syntax to a language anymore and that takes up a lot of time and that's what my job doesn't
[00:52:39.000 --> 00:52:45.800]   entail, but I can instill these TDD. How do you actually write good acceptance criteria? How do you
[00:52:45.800 --> 00:52:51.240]   actually do these componentized type of design and things like that? So if you're able to teach your
[00:52:51.240 --> 00:52:57.880]   teams how to think and really use some of these foundational bits and then evolve with that and
[00:52:57.880 --> 00:53:04.840]   learn, then I think you're in a better state. But we did have some missteps here. Like just assume
[00:53:04.840 --> 00:53:08.920]   the LM's can do everything. It's gonna make us go faster. Velocity's gonna go through the roof
[00:53:08.920 --> 00:53:13.320]   and all these kind of things and you just have to start thinking about how to measure a little bit
[00:53:13.320 --> 00:53:18.280]   differently now of your teams and what they're doing and things like that. So yeah, still learning.
[00:53:18.280 --> 00:53:22.840]   You get in a perfect world where it does help you just go faster. Is it helping you go the right
[00:53:22.840 --> 00:53:27.480]   direction faster? Where are you going off of a cliff faster or are you going nowhere faster?
[00:53:27.480 --> 00:53:32.920]   Like yeah, yeah, yeah, yeah, yeah. I don't want to go off the cliff faster, you know,
[00:53:32.920 --> 00:53:42.920]   that should be slow so I can catch myself. 100%. But yeah, yeah. So when it comes to enabling
[00:53:42.920 --> 00:53:48.200]   teams with these tools, right? Like what's working for you? Because when you guys started
[00:53:48.200 --> 00:53:52.200]   three and a half years ago, like you said, there was no cloud code yet. So you've already had the
[00:53:52.200 --> 00:53:59.320]   deal with some chain. What what any moments working at relational AI? Yeah, I think we're still a startup
[00:53:59.320 --> 00:54:05.080]   and so we're a little bit lean and I think we could probably use a little bit more structure in
[00:54:05.080 --> 00:54:11.480]   terms of enabling people and teaching people how to use tools appropriately. So there's definitely
[00:54:11.480 --> 00:54:18.040]   more to go on that for sure. I do think like there's two parts enablement for me. One part is really
[00:54:18.040 --> 00:54:23.240]   teaching foundational processes, like you know, ways of working and things like that. And now it's
[00:54:23.240 --> 00:54:29.320]   ways of working with all these different tools. So a lot a lot to do there still. And I think because
[00:54:29.320 --> 00:54:35.560]   it's such a changing ecosystem, it's it's hard to keep up, you know, with that kind of thing. So
[00:54:35.560 --> 00:54:41.640]   we're trying to figure that out. I do think though, there's another part of enablement, which is
[00:54:41.640 --> 00:54:48.600]   knowledge, right? So the biggest part of onboarding onto an enterprise is startup or any company is
[00:54:48.600 --> 00:54:54.200]   like, how do you onboard with knowledge as quickly as possible, right? Your knowledge base sometimes
[00:54:54.200 --> 00:54:59.320]   is a confluence, sometimes isn't GitHub, sometimes it's here, sometimes in there. I will say the tooling
[00:54:59.320 --> 00:55:05.400]   that we have now the LM's as I said earlier is that it is fast at gathering knowledge, right? Like
[00:55:05.400 --> 00:55:11.880]   it can say, this is where everything is, right? And I do think we've we focus on like how do you
[00:55:11.880 --> 00:55:18.280]   actually aggregate knowledge to the LM's and really understand how decisions are made where,
[00:55:18.280 --> 00:55:23.800]   you know, this code base, you know, feature set is a words document it here and bring that all
[00:55:23.800 --> 00:55:29.400]   together. So that's part of enablement and onboarding. I think that's been accelerated based off
[00:55:29.400 --> 00:55:35.640]   the LM's, but a bit more work to do on how to use a tool, how to use it responsibly, how to use
[00:55:35.640 --> 00:55:40.120]   it very effectively. I think we're all still kind of learning how to do that. So it's hard to
[00:55:40.120 --> 00:55:45.480]   give those ways of working when we're like, it's here's the ways of working two months ago. Let's
[00:55:45.480 --> 00:55:52.280]   see how this goes going forward. So yeah, so have it solve that problem yet, but working on it for sure.
[00:55:54.120 --> 00:56:01.000]   Cool. Cool. This is great. Cassie, thank you. Okay, so there's sort of a few questions I ask
[00:56:01.000 --> 00:56:05.800]   towards the end and where I feel like we're getting there. Thank you for how much time you've
[00:56:05.800 --> 00:56:11.640]   invested with us. What's something you've learned, right? Like maybe something you thought you were
[00:56:11.640 --> 00:56:16.200]   right about, but you've realized recently, and this could be in your personal life, this could
[00:56:16.200 --> 00:56:21.240]   be a relational AI, this could be for anything. But what's something that you used to think you
[00:56:21.240 --> 00:56:24.760]   were right about and you've since learned to like, oh, wait, no, that's not how that works.
[00:56:24.760 --> 00:56:33.960]   Yeah, I think more of a general vibe and I'm very happy to admit I was wrong on this one for sure.
[00:56:33.960 --> 00:56:41.720]   I think when the first models came out and chat GPT came out, I knew something big was happening,
[00:56:41.720 --> 00:56:49.320]   but I guess I've just been maybe some PTSD from how enterprises and things just move slowly
[00:56:49.320 --> 00:56:54.600]   in the software world. I was like, we got time. We got time to figure that out.
[00:56:54.600 --> 00:57:04.120]   I was wrong. This is moving at a much faster pace than I had, I myself had anticipated both
[00:57:04.120 --> 00:57:08.760]   in relational and both in like, you know, it's in our everyday lives. It's, you know, when I talk
[00:57:08.760 --> 00:57:16.120]   to Google, it's thinking about stuff like more so than it's ever thought before. And I do think
[00:57:16.120 --> 00:57:22.760]   they're, I guess I would say that I was wrong how fast it's going in. So there's so there's
[00:57:22.760 --> 00:57:29.240]   so many implications of this that us as a society have not thought about, like, you know, my
[00:57:29.240 --> 00:57:35.240]   teacher friends, my university friends are like, how do we teach students right now? How do we know
[00:57:35.240 --> 00:57:39.480]   that they're learning? How do, like, how do we embed LM's that like, what does this actually look like?
[00:57:39.480 --> 00:57:43.080]   And this is moved so quickly, like, we're not able to catch up with,
[00:57:43.880 --> 00:57:49.400]   sort of the ramifications of this both from a technical perspective, but from a societal perspective.
[00:57:49.400 --> 00:57:56.440]   So, yeah, I don't go into answer, but I was definitely wrong about how fast this was going to move.
[00:57:56.440 --> 00:58:01.960]   So for sure. Well, you and me both. And I read a few LinkedIn posts talking about the stages of
[00:58:01.960 --> 00:58:06.840]   grief of like, I don't believe it. And like, so I think we're in good company. I think you're
[00:58:06.840 --> 00:58:13.320]   exactly right. Yeah. Interesting point to about education. Like often, we sort of go to like,
[00:58:13.320 --> 00:58:16.840]   what's it going to be like for new junior developers? Like, how are they going to become, you know,
[00:58:16.840 --> 00:58:19.720]   trained and et cetera, et cetera, et cetera. And I think that is a valid question.
[00:58:19.720 --> 00:58:23.480]   But I like where you're going, right? Like, how does education work now? Right? Like,
[00:58:23.480 --> 00:58:28.840]   well, how do you even know? Like, did the LLM answer that? Or did the kid and
[00:58:28.840 --> 00:58:35.880]   is that okay or not? Like, what, what are we even doing? Yeah. Yeah. And are we reacting? Like,
[00:58:35.880 --> 00:58:41.640]   there are some people who I'm hearing from, they're like, we just ban LLMs from the classroom. And
[00:58:41.640 --> 00:58:46.600]   it's like, you can't do that either because that's a major part of our life or ecosystem.
[00:58:46.600 --> 00:58:54.760]   How do you think more proactively and embedded into it and and make it sort of your assistance,
[00:58:54.760 --> 00:58:59.960]   right? And actually embrace that. Teach people how to actually use that. Someone will always
[00:58:59.960 --> 00:59:05.800]   gain the system for sure. And, you know, sure, all of these things. But I, you know, again,
[00:59:05.800 --> 00:59:09.400]   I don't know if I'm right or wrong, but it's one of these things where I'm having active
[00:59:09.400 --> 00:59:15.000]   conversations with some of my friends, you know, in this world. And it's, it's hard. I'm not saying
[00:59:15.000 --> 00:59:21.560]   like, like the entire industry education, different industries will be uprooted by this. And so it's
[00:59:21.560 --> 00:59:28.600]   how we, how we react to that. I think it's the question. Yeah. 100% true. Man. Okay.
[00:59:28.600 --> 00:59:31.720]   Lighter question. What is something that's giving you joy right now?
[00:59:33.000 --> 00:59:41.240]   Oh, Kyle, I'm having fun. Like, this is actually like a lot of fun. And I, I don't mean to put you
[00:59:41.240 --> 00:59:47.320]   in the same bucket as me, but I was a very, I was so excited when I first started my engineering
[00:59:47.320 --> 00:59:52.760]   career and, you know, I coded something and then something came up and I saw that button and I was
[00:59:52.760 --> 01:00:01.240]   like, wow, how cool is that? And then I went, I took a, you know, a turn to management and people
[01:00:01.240 --> 01:00:08.360]   and dealing with enterprise and legacy organizations. And, and recently I've been, you know, coding
[01:00:08.360 --> 01:00:13.960]   again because I have this beautiful assistant of mine that's so much faster. And I'll be honest,
[01:00:13.960 --> 01:00:20.360]   it's experimentation. All of these things proving out a proof of concept. It's something that I don't
[01:00:20.360 --> 01:00:24.600]   have to sign to one of my teammates anymore. I do in some things, but some of them I just take
[01:00:24.600 --> 01:00:30.280]   for myself. And I think that's really fun. And that may speak to how terrible I would code
[01:00:30.280 --> 01:00:37.480]   or I was or whatnot, but I'm at least I'm proud in having fun with that now. So yeah, yeah, I can
[01:00:37.480 --> 01:00:44.840]   totally relate to that. Yeah, yes, yes, yes, yep, yep. Cool. Cool. Cassie, thank you so much
[01:00:44.840 --> 01:00:49.560]   for coming on the show today and spending some time with us and engineering with AI podcast.
[01:00:49.560 --> 01:01:04.200]   Awesome. Alright. They're small, highly maneuverable and in some respects they think for themselves.