The conclusion I've reached though is that if we looked like we were accidentally creating greater quality, we would respond not by achieving that quality, but by lowering our
standards or increasing our haste until we manage to lower quality back to where it was before.
Because the level of quality we have in software is not it's not the maximum we know how to achieve, or at least I don't think it's been in any code base I've been part of.
It's basically it's lowers
the conditions we'll accept.
And if it gets lower and things start breaking, then people will get it so it's just above the waterline.
In an Engineering with AI First, we're having a guest back for a second time.
Most of our audience will remember Chris.
When he was last on the show, he led with "Human in the Loop is dead".
He's also been one of the most often downloaded episodes of all time.
And I don't think it's just because he said something spicy.
I I tried that with other guests.
There's something else to it.
We wanted to have him back today because of the book he's been writing.
So let's get started.
Chris, what's it called?
What are you doing?
What are you saying in this book?
Right, it's awesome to be back.
Okay.
So the book that I'm writing is called Agentic Engineering at Scale.
And basically the concept is I wanted to talk about all the stuff that is beyond personal productivity.
Like there's a lot of personal hacking going on, but then a lot of people want to know how to get stuff done in groups of people or communities of people.
maybe I can I can kind of say how I've divided things up in the book because I you know I wanted to I want to make it work for people who have real problems in real organizations.
Not because I don't find it really fascinating, you know, how you set up your right agents MD or exactly how you code as an individual, but well, selfishly that stuff's changing
really quickly.
So very hard to put into a book in a way that will not be obsolete in a month's time.
But secondly, I think the the gap.
that a lot of people I'm seeing ask are people who come in after hacking on the weekend, getting all sorts of stuff done, and then find that they just don't know how to make that
thing or that magic happen or apply in their workplace.
~ so
that's what I'm trying to tackle.
Okay, so this is not so much for the person who's vibe coded something and and now wants to get it in the app store.
This is more for the the the plucky engineer who wants to to help out the team he's on, he or she.
Yeah, so ~ I I guess the so as a I'm a consultant, I work for ThoughtWorks, and so often we give advice to our clients about how to do better in their software engineering.
And that normally means because they have to be big enough companies to wanna pay externals to give them opinions, that they're trying to get large things done with groups
of people coordinating.
And so that doesn't it doesn't decompose into just everyone go off into their own corner and vibe code their own app.
There has to be
Right.
have to care about quality that maybe people don't care about as much when they're prototyping.
They have to care about managing priorities and specs and things like that.
They have to worry about teams of people working together.
And maybe the biggest thing, which I opened the book with, which I think is a bit of a contrast, is they have to worry about legacy code.
So a lot of people you hear, you see on LinkedIn or whatever, they're basically talking about things that they started last night and maybe you've got a hundred GitHub stars
already.
Mm-hmm.
not thinking about here's this system that pays a thousand people's wages, how do I work with that in a way that respects it and doesn't break it, basically?
Yeah, yeah, yeah.
Okay.
Cool.
what what was your process like for the book?
Did you research and talk to people and let let's hear that part?
Well the selfishly the the kind of the the fortunate thing is that my job consists of coming up with opinions in this area and working with clients to solve their problems.
So I do get a bit of synergy out of my personal thinking time and my work.
So a lot a lot of what I ~ write about is maybe a synthesis of my own experience across many different clients, which is one of the privileges of being a consultant that you get
to dip your toe into many different waters.
but also the collective wisdom of my colleagues at ThoughtWorks.
So I'll you I
do a lot of sharing of drafts and asking of questions on the internal chats and and things like that.
I mean of course also I I do a lot of reading to inform or try to make my book semi informed ~ as well.
but
really it's trying to channel the experience that people have already started to have in these in this area.
Yeah.
It's no easy thing writing a book.
There turns out to be a a fair amount of work, ~ in in in getting the writing down,
publishing down, the all the stuff.
Yeah.
Well, a and especially I guess fascinating now is I I don't know whether people will be starting to assume that books have been written themselves by AI.
I've certainly had at least one article that I wrote by hand a while back have someone report to Reddit, I think, for being a AI written.
~ I'm not quite I well, I'm not quite sure what caused that and I don't particularly blame someone for thinking that because I guess there are there is a lot of mixture of AI
content or AI inflected content or
or something out there.
But it is kind of a an interesting thought that I might be just putting quite a bit of effort and quite a bit of part of my life into compiling a hundred thousand words and then
have someone assume that they ~ they could do it themselves with fable.
~ I guess we'll see though.
What was it?
The two, three sentence prompt, basically.
sound something like that, yeah.
Yeah.
That's all it was.
Yeah.
No, ~ it doesn't work that way, for sure.
I it's funny though, to your to your point, there are certain tropes that like just hit me in a certain way right now.
Like when if it's like if if
What actually matters here is I'm like, okay, yeah, that was yeah, but that was probably you know, it was anthropic model, it was Claude that you were talking to that generated
that.
And you know, you everybody at first was trained to look for the em-dash instead of the hyphen because like who even knows how to type that?
But ~ but it now it's it's certain other things.
But now I wonder, like, is it inflecting our speech?
Is it it's picking up on what we are doing and then it's doing it and then we're gonna follow it.
I I don't know.
So which came first in all of that?
Yeah, well I have actually an dash problem because the way I found easiest to write is dictation.
~ maybe because
through conversations like this I find it easier to maintain the flow while speaking.
Yet
while speaking and then reading a transcript, in speech we have a lot of, you know, meaningful pauses that lead on to the next thing we say.
So I've naturally, when I get my raw speech transcribed, it has quite a few m-dashes based on the the software I'm using.
So I'm now faced with the paradox of do I leave them in because that's authentically what I came up with, or do I try
to write some kind of AI tool to get rid of the dashes to pretend that it's more natural than actually
is so I'm I'm a bit confused as to how how I should do that that at the moment.
~ but I'm certainly stouting myself.
These these are these are the problems that we now have as a society to solve.
Yeah.
Cool.
Well obviously you don't want to give away the ending or or you know or any of that.
Yeah.
I can I can kind
of g maybe engage with the key the key arguments of it.
I think I think if someone got to the end of a technical book
and only only kind of s found out right at the end what it was, I'm not sure they'd actually invest as much time.
Yeah.
Well, but I'm I'm also I'm wondering
if there's a story in there that you know, to your point, a lot of this comes from, stories real and synthesized is there a story you would tell out of this?
I think there's an I have a narrative of order of concepts.
the big overarching theme is that control theory is a really useful way of thinking about how to get coding agents to do good work
Mm.
and that if you're gonna take that seriously, it's not just a matter of an individual.
coming up with the kind of ~ guards, guardrails, checks, sensors and things like that.
It's actually something that you have to start sharing across your organization, which leads to a whole bunch of
problems because we already found it kind of difficult to do platform engineering.
But I think I have become convinced that if you're going to achieve with agentic engineering at scale, so just getting the book title in there, ~ another another kind of
branding plug, ~ you will have to start thinking about sharing
common expectations of what go good code looks like in a way that coding agents can access
Mm.
at scale and then you'll have to worry about the knowledge management and the human collaboration that gets to the consensus on those artifacts.
Right.
a team that I serve ~ has taken to having a a confluence page that that sort of is a is a little bit of a a starting point around like, okay, please don't just go write your own
crazy terraform willy-nilly.
We we use a YAML file that our platform team knows how to ingest, here's here's the ropes, there's the bathroom kind of stuff.
~ and that way, since it's in the confluence, like ~ the first thing we do in ClaudMD is go read it and it's like, okay, now none of us have to tell it again.
You know, and we can kinda keep it up to date.
Yeah.
So I'm imagining that kind of thing, but you may mean other more nuanced things in addition.
So I I I divide up the different kinds of and I'll say harness for now and maybe you can substitute in whatever word is popular at given times.
You know, we can we can refresh this as we get different jargon as we go along.
but I divide harnesses into four kinds and I think you need
a balanced diet.
So ~ maybe I think the two most important distinctions of different kinds of harness is whether it's something that tells you yes or no.
So whether it's giving information back to the agent and the agent has to make its own decision or whether it actually gives a red or a green signal.
So a unit test keep tells you no, it's not good enough, or yes, it is good enough.
But something like using play wright from the browser gives the agent access to look at the site you're working on, but it doesn't tell the agent whether it's acceptable or not.
The agent has to kind of apply the judgment.
So I think that's kind of that's one of the distinctions.
The second distinction, just a link to what I said about
control theory is ~ feed forward versus feedback.
So maybe I should unpack that because feed forward is a somewhat unfortunate nilogism.
But feedback is I've done a bit of the system, let me use this to perceive it and work out what's going on.
Either to tell me whether yes, no, if it's a judgment based thing, or just to let me get information or context about what's going on, kind of observability.
So that's got a feedback which people are familiar with.
Feed forward is effectively
guidance.
So it's you know, in control theory, feed forward is the stuff that tells you where you should be aiming for without you having to give it a go and and evaluate where you are.
So if you kind of think of those two things as axes, because I'm a consultant, everything becomes a quadrant, that you get kind of things that are normative, tell you yes or no
that are feedback, things that are normative that are feed forward, and you get the informative versions of feed for forward and feedback as well.
And so that combination I think is really important.
So to your point about guidance, I would call that a guide if you have something about the golden path and your confluence.
So that's informative feed forward.
So the agent can use that to do a better job.
It's not going to tick off the agent for getting it right or wrong, and it's not going to allow the agent to evaluate what it's done, but it is actually one of the important things
in order to to have a good system of input to the agent.
And I and I would say a lot of people who are struggling at the moment tend to be overindexing on one kind of harness or the other.
So they might be trying to lean
entirely on feed forward.
I'm not gonna give the agent any way to mark its own homework or understand what it's done.
I'm just gonna tell it what good is and hope that it listens.
And it might listen,
but it might also go off track and not be able to self-correct.
But alternatively,
I see people focusing far too much on feedback, which is leads towards brute force kind of
situation.
So they don't give the agent guidance on what a good design looks like.
So they don't give it the agent a design system or they don't give it a templated infrastructure for your company.
They just say go for it and like see how you go.
And that results in cycles, which is good, but too many cycles and it doesn't converge on a good answer as quick as it would if you'd started it near to where it should have been.
And don't think these things learn on the job in the same way or as quickly as humans do.
So yeah, that sounds like it could be particularly painful.
Yeah.
Well, yeah, learning on the job or just, online learning in general is, some people would say is one of the big architectural problems with the current generation of LLMs.
~ and
it's actually, you know, it's not what the everyday public thinks is happening.
The everyday public assumes that if they ask a question of an LLM, the L LM will learn at the same time because that's what we do.
But as we
know, if we you know if we want to feed information we've learnt back in, we have to do it explicitly with some kind of agent memory mechanism.
Exactly.
Yes.
We have this business of memory saving now and and those kinds of things, which is kind of trying to approximate it.
I'm not sure it's exactly the same thing, but it 'cause that just results in more feed forward.
But maybe that's fine, I guess.
But i you've got this concept of like the context window is only so big though, right?
So
I assume all this feed forward is eating into that budget in the end.
So I guess progressive context disclosure, maybe an overly bureaucratic term, is what people usually say for trying to stage the input so that the agent doesn't just read the
whole world through a straw.
So for example, if you're giving your agent ADRs, you should be giving it an index page with clear kind of titles so that it knows when it should actually go one step further.
And when I've seen people profile the context usage of their agent.
accidental overreading of context is probably the number one thing that burns tokens.
They didn't really realize it, but they've got a kind of a series of hyperlinks that the agent is preloading because it's been encouraged to go from the agent MD to this file to
your Confluence to another page on Confluence to a manual.
and it it does that maybe on every session or almost every session and that causes a bunch of irrelevant stuff to yeah junk up the context
window like you say.
Yeah, yeah, yeah.
And it gets dumber as the context window fills for sure.
So that this is to be discouraged.
I don't know about your experience, but I think the common figure that people say is that if you're more than fifty percent full, ~ it starts to have trouble with knowing what to
pay attention to.
So you start
to get erratic results.
Yeah, right.
I I often think of it as it doesn't know what to forget to to keep room in the in the but but yeah, you're right.
It's also the just the problem of what should I pay attention to?
This thing's got so much in it now.
Yeah.
Nice.
in my mind I it's it it manifests in maybe early on in your agents MD or even in your session you told it, please remember this, this, this, do not do this, this, this.
And once you get more than fifty percent full, you start at the playing Russian roulette with whether those instructions are still going
to be brought to mind at any one time.
Yeah.
Through the magic of time the time travel that podcasting brings, people listening to this episode, if they listen to the last one, it was with Birgitta Bockler also at
Thoughtworks, ~ and she talked about how we say these are guardrails.
Those are the things that keep your car on the highway, but
sometimes your car just goes straight through and the guardrails are just gone in this world.
So are they guardrails at all?
Yeah, feedback and feed forward definitely seem like better words for what's happening there.
Yep.
and to be clear, I partly got those terms literally from Brigita, who I talked about this before.
So I I don't want to claim to have superseded or or improved necessarily on her thinking.
I think she's done a really good job of distinguishing between different kinds of things that we we have in harness engineering and spec driven development.
Definitely.
She's she's given us some more words.
This is so exciting about ~ about your book, Chris.
what can you say about like when it comes out?
And i is this gonna be just Dead Tree or or I don't know.
Will there be a movie made?
Like do I get to play the starring role?
Probably not.
But
Yeah, I mean I guess ~ I Dead Tree is not the only format that you would release a book in these days.
I do have a promise of that being the case.
I have a an O'Reilly animal for the front cover, which is Australian fauna, which I asked for.
~ they didn't they didn't ~ they didn't agree to f get me a deadly Australian animal, so they found one of the cute ones that is not deadly, but still that's that's pretty cool
for for kind of a
eco-nationalist like myself.
~ it's supposed to come out in Q1 next year.
I'm about to say something that I'll probably regret.
I'm a little bit ahead of schedule in terms of the writing.
~ though I've discovered it's a really good embodiment of ~ the difficulties with improving the speed of coding, not necessarily improving the speed of the machine.
I'm working in a collaborative effort with a bunch of people who do other things in their day job with publishing and they have other schedule.
So me
dictating or typing slightly faster doesn't necessarily alter when the end product is, much like using a
coding agent doesn't necessarily get you live faster in a real project.
But yeah, the the schedule is ~ Q one next year, but the early release is in the works at the moment, so that'll be on the O'Reilly platform in electronic form with I think the
first four out of twelve chapters soon.
Cool.
Aren't you gonna like keep us keep us waiting to figure out which Australian charismatic fauna that we've chosen?
just assumed you wouldn't know.
So that I really like to use the the ~ obscure ones.
the animal is a honey possum, which is basically the size of a mouse, but it has an especially long nose because it's design well, I was about to say designed, I didn't mean
that.
~ it's iterated towards being able to suck out honey from banksia and protea flowers, which are these kind of like enormous flowers of Australian flora.
So yeah, it looks pretty it looks pretty cute is the idea.
And yeah, I don't think
I know of any fatalities at all involving them, but I guess we can we can always hope.
Yeah.
Tiny possum basically.
there's an analogy in there somewhere.
Th the long straw something, I don't know.
Yeah.
Yeah, I mean
I I I I think ~ I I've learned obviously well I I've done a lot of technical reviewing in the past.
So I have I've been, you know, reviewer of a lot of books is my first time actually being the the in the driver rather than the backseat driver.
~ but one of the things I learned is that, you know, publishing companies want to be publishing companies, not people who spend all their time explaining to adults why they
can't have the elephant or whatever.
So, you know, in the end they don't try to come up with really direct mappings of
cover animal to subject matter because I think that ends up being just like coupling
in software, that ends up being ~
a bit onerous even though it might seem neat at the time.
Yeah, a bad idea for your publishing process to try to find like the exact right
change what chapter seven's about and they have to like get you a new animal.
I don't know.
Like it yeah, it doesn't
Right.
That too.
This is cool.
Super exciting.
so I'm wondering did you did you check out the radar?
Like what were your thoughts?
it was a lot of fun for me to go back and and review what you and other guests had said, ~ and put this together.
~ yeah, what were your thoughts?
to start with, thank you for putting it together because I I I think ~ my kind of meta observation is that we're using words in really incommensurate different ways.
So all your work you're doing of kind of trying to organize the thoughts to actually start the process of us at least being able to even argue at each other rather than past each
other, I think is a pretty important function.
So thank you for doing that.
~ I think for me I was quite attracted
Perhaps because I was interpreting things through my own lens, thro the the things that were, I guess, talking about feedback in some way.
I saw a lot of things through that lens.
and that maybe in to tie it back to the work I was doing of of trying to reconcile feedback and feed forward, is that I I I can't
say I scientifically remember exactly the proportion of things, but I wondered whether we were focusing too much on the inner loop and not enough on the guidance of how to get to
close to what the answer is before we ~ iterate.
So I kind of I I liked it both for helping me to explore the different ways people have thought about those things, but also, you know, I was kind of in a way using it to map out
what parts of the conversation we haven't had yet.
Got it.
Now when you say interloop versus guidance, I'm not sure I can exactly like I could make three guesses as to what you mean, but but help me fill it in.
Well, I I I'm probably not even using my own terminology in a standard way, but what I meant specifically was the experience of an individual developer.
they've got the coding agent, they've got things set up, and they want to give the agent a way to tell whether it's doing a good job.
I think I I seem to remember quite a few of those.
So in my mind, that's very that's in actually one of the most useful quadrants of the one of the the four, but it's ~ feedback and normative.
So how do I get a linter?
To tell the agent that it did wrong and it has to fix it?
Or how do I you know how do I otherwise control the agent?
I think that is really important because anything that's in there is part of the discretionary process that you can set the agent off to do, to tie it back to the earlier
comment.
I'm not sure I phrased it exactly this way, but ~ of human in the in the loop being dead.
But if you want the human in the loop to not be there, you need to give
enough scaffolding that the agent can try, fail, try better, fail better, succeed.
And a lot,
I think, of our attention has been naturally at that question of in the small, including your guests, because that's a problem we have to solve before we can even earn the right
to encounter other problems about how we get to good results.
Yeah.
I'm seeing interesting like patterns in terms of how to make that safer, right?
and whether it's like, the Swamp Club, which I I learned about only like a few days ago and and still need to dive into, but ~ it's apparently aimed at sort of DevOps scripting
and helping the agent become well, repeatable ~ essentially I assume that this means it's going to become a tool that the agent uses instead of like letting the agent GPU it and
there 'cause it might do it at
the second
Wait at a different time.
~ but but yeah, tools like that, tools like code mods, you know, like building something that goes and does the refactoring for you so that it doesn't have to imagine how to do it
each time.
These not only save on tokens, but also make it safer for us not to be in the loop, I think.
Yes, I think if don't use inference to solve something unless you have to is I
think a reasonable rule.
Although I must also say that like I think some of our enthusiasm for these tools is partially due to wanting to retain the kind of control we have when we use tools like
refactoring tools and compilers.
So
~ I've certainly heard a lot of people talk about how they think it's a problem that agents are not deterministic.
And so their solution to that is well, how do I introduce determinism much in the process?
So I completely agree actually that if a code mod can do it, that's a more efficient way of doing it and it's a less error prone way of doing things.
But I don't I think it's a strength in a way that agents are non deterministic.
It's just a a strength that
we are not used to exploiting.
Right.
I I I in when I'm working in with organizations and I'm trying to make sure that the engineering staff in that organization has access to good models, I often find myself
saying, I just want my smart friends to have the good tools.
But I feel like this also applies to the agent.
I want my smart agent friends to have access to the good tools, whether it's Swamp Club or Code Mods or Linters, obviously.
But you know, sort of all of this category.
And I guess these are all in the feed well, many of these, not all, are in the feedback.
side.
But some of this is is it almost something different.
Like if we're expecting the agent to reach for that tool in the same way that it would reach for L S or Bash or whatever, is that feedback or feed forward?
That or is that something in the middle?
so I think it might be not I don't want to introduce three dimensions because ~ even even in electronic books you can't really represent it.
But I think I think the kind of mech mechanistic versus the inferential is a a little bit of another distinction because I think you
could I think you could introduce it for for other things in in terms of the way the agent invokes it.
but I I I think that a lot of the advantage to the it's not just that it's mechanical, it's that you'd need the agent to do less work.
If you can express the possibilities of what you want the agent to do, is what effectively is a smaller language or a smaller set of levers.
Basically, you're asking for less information to be outputted by the agent to make a good choice.
So you're not asking it to wade through a billion invalid possibilities to find the one good one.
It kind of matches up with the idea of the golden path from Netflix and that you mentioned earlier, which is that if you give it a more constrained set of options of how it can act.
you're putting less of a burden on it, you're making it more likely it'll choose well.
I think I'm learning that the story I told about go to confluence to find out how we deploy is something Netflix talks about called the Golden Path.
Get you wanna tell that story?
well, sorry, I think I think it was originally they taught the the the concept of the golden path was so so Netflix were ~ earlier adopter of microservices, expanding very
rapidly, ~
aiming to have really high talent density.
But they also then realized they didn't want everyone to be spending their high talent density in building infrastructure from scratch, right?
So
so how how do you square the kind of anarchist freedom principle with the kind of
centralization urge to have consistent platform.
The way they
did it, and it's obviously been copied by others because they're an admired company, have done many interesting things with tech, is that they have a golden path that says if
you're using Java and you're using this AWS service, then we support that really well.
We document it really well.
But it's not you can't leave the path.
It's that if you're on the path,
you get first class service across all of our tooling and other things.
So ~ you know, it's a it's a way of
in a way lowering the team's cognitive load because they don't have to evaluate every container running service that AWS has for themselves.
They know that this is the one or these are the two that Netflix supports and that they'll get great tooling out of.
But if they really do have a unique need, they're not constrained from going off road, basically.
yeah, yeah, yeah, yeah, I I can remember these conversations coming out of Netflix, you know, sort of five, ten-ish years ago.
Like I think I was in Cincinnati, so that would make it ten years ago.
~ at a at a guess.
and yeah, that that sort of conversation about like, yeah, you can use the you know, this framework instead of that what framework, but you know, as you say, it's it's not the
golden path, it's not as well supported.
And that's that's a neat way to split that.
It seems like interestingly, ~ in my own observations with my own clients, we're kinda not doing that with AI.
And maybe that's ~ like a little bit of a fear about the non determinism and the security impact.
I am aware of companies that have already seen well, I mean, we've all had the Shy Haludworm, you know, warrior's, you
know, come in and and and w inject scripts into our systems and so on.
So
May I get why people would, you know, let's let's not go with the anarchy path this time.
But but I'm just dreaming this up right now.
Does that match your experience?
Or are do you see organizations which are trying that, hey, let's experiment, let's allow experimentation, but this is the golden path versus what I'm seeing, which is more like
no, there this is the path, you know, blinders on.
Yeah.
So I don't think organizations have got to grips how they can, as a collective, ha exercise any kind of opinion one way or the other.
Is my is my experience.
So ~
teams build their own skills and share them among the team is actually the most common level of collaboration I've seen, with exceptions of maybe some specialist services like
token gateways or whatever, most of the time there is no collaboration of agents working to do things.
So it's the equivalent of completely
Atomize teams.
The the most I've seen is that some companies do have systems for sharing skills.
So someone will come up with a really good code review skill and then they'll share it with other teams.
Most commonly it'll get passed, maybe unofficially.
There are some organizations where there's like a central repository for them.
But even at that stage, and you know, we're talking about really basic stuff, not infrastructure templates or design systems like you're kind of hinting at with the golden
path.
They're just
Here's a way to review a pull request or whatever.
~ They get put on a central place, but it tends to rot at about the same rate as a wiki page as well.
~ Unfortunately.
So ~
in terms of the choice, does the organization say agents must code in exactly this way versus encouraging the agent?
I think actually by default, we're not restricting the agent at all, not because organizations have decided that they wish to see self organization right reign, but
just because the governance functions of technical organizations haven't got traction on how to exercise their function with agents.
Now I think
actually we're going to see that happen and in a very clumsy way soon, where, you know, there will be a set of guidance that your agent is expected to load and the CTO will be
able to like change it on the weekend and where previously the CTO might have wanted you to start writing unit tests or
move only to outside in tests or adopt this latest thing that they've heard.
They previously had to kind of rely on this chain of influence of telling the head of architecture and doing all hands and then if necessary people could just ignore them and
get on with their life.
But I think technical authoritarianism will actually be a a risk of having shared harness artifacts because it will
be possible to directly inject
You know, your your prompt injector will be your CTO, maybe is another way of putting it.
And if they have
great judgment, maybe that's good.
But I I can't say that, you know, being senior always gives you infallible tech judgment necessarily.
I I the immediate flashback for me is working in one of those organizations where the entire company, and in this case three hundred thousand people, the exact same JIRA
template across
all projects, whether they were accounting or finance or software development or and yeah, so authoritarianism gone wrong.
But maybe this will be better.
Maybe fingers crossed.
Yeah.
think some level of standardization is good.
That is actually the Netflix insight right?
It's better that kinda
we're a Java shop or kind of we're a dotnet shop or kinda we're a Ruby shop because that does
help us collaborate more effectively.
But the
excessive standardization is kind of pretending the world is simpler than it is.
I think it is very relevant because we're we're seeing ways of scaling decisions with agents that we haven't seen before.
And so
Yeah.
stomping over the local resilience of people taking context to into account and localizing decision with their own thing, that that might actually be stomped on a little bit by
having organizational level harness engineering.
And so I think we will see over correction and then recorrection in that area.
Cool.
So yeah, I one of the things you one of the things that you responded when I asked you about the radar, ~ you said, Well, Kyle, we didn't hear your voice.
~ and and true, that it w that was somewhat by design.
~ it's funny though, ~ on review when I was doing the work of putting together the radar, I thought to myself, hmm, because what I did is I fed, you know, the transcripts into
Claude, and Claude comes out and says, you know, Ahmed said this Ahmed said that, and Chris Ford said this and Chris said that.
But
It was giving them to me like by episode, but I'm not sure it actually knew who it was speaking.
So some of the things that it would attribute to a guest,
it might have been me that said it.
~ and so like certain things that I recognize as like, yeah, that's me that says that, is like AI is an amplifier.
I'm not trying to claim that one.
Like probably somebody said that to me first, but I know I say it frequently.
Yeah, no
doubt, no doubt.
So there is there is definitely some of Kyle in there.
and I I think that is one of the things.
If if your team is fast at making messes today, it's going to be faster at making messes tomorrow.
That's definitely something that I said more than once.
Yeah, thanks for checking out the radar.
Thanks for being part of it.
~ appreciate that.
I'm trying to think about what's changed since you were on.
It looks like it was back in April.
~ and I guess, you know, the the you were saying when we prepped the the March of the models, right?
Fable in in the paid world, ~ and and of course like Kimi Three and and others in the non paid world.
~ yeah, I mean what it it's been it's been an interesting time since since April for sure.
Yeah, I would say that the number one thing that's changed for me in my professional context is that the amount of people freaking out over costs has ramped up.
So, you know, we did have that faddish idea of token maxing at one point, right?
Where maybe, you know, literally consuming more of a resource was considered virtuous.
So we'll have a leaderboard.
I don't think that lasted very long and I guess it was a little bit of a stunt.
But we're seeing
At least I'm certainly seeing in my professional life organizations go the other way.
So, you know, that unlike us when we have our personal subscription, organizations are on, pay per token API plans of different kinds.
And even from the extreme of how do I make sure that no one in my organization accidentally spins up a recursive sorcerer's apprentice agent job and costs me $50,000
through to just widespread reported people
Having had a budget set and then blowing that budget roughly, I think quite commonly in one week out of a month is about the common ratio between what people expected and how
long it takes.
This
is causing a lot of consternation because people have remembered that we don't know how to measure software productivity, software development productivity.
So
when the costs were low, or maybe it was a few
teams who were trying things out and certainly they might have been prosecuting the C-suite's agenda because the C suite was interested in getting better at AI.
It was kind of fine.
But as more people blow more budgets, ~ people who are responsible for being able to defend ROI of things are finding that they can't easily work out whether it's worth paying
that much in tokens or not.
and so that's a little bit of a kind of parties over moment in a lot of places.
Yeah.
which is which is interesting.
I can remember hearing, you know, ~ C suite folks from various AI companies going on podcasts and saying, you know, if your engineers aren't using so many tokens per day, you
know,
you need to worry about that engineer's productivity.
It's like, wow, that was brilliant, you know.
I mean on some level there is maybe some value to it.
I mean I I kinda get it.
In our in our own organization, you know, so and so has a reputation of using a lot of tokens and no one's sure why.
And
other people, it's like, well, you're not using any, so are you learning fast enough?
It's not a useless metric, but ~ certainly like like you say, we we don't know how to be healthy about our metrics yet, I don't think, in software development.
So all metrics can go toxic ~ and problematic.
But but it really has been interesting seeing that swing around.
You know, people like my mother is sending me articles saying, AI is getting expensive.
Maybe you should worry about getting into AI now.
It's like, well, yes, but
I mean, lots of professions, the equipment is more expensive than the wages.
And in those professions, people argue about your raise less.
So the problem we have as engineers is is we are the biggest cost, and while we're the
biggest cost, it is worthwhile for people to argue the most about whether there's a slight increase in salary.
If we did get to the stage where Jensen, who I think you're kind of alluding to, ~ you know, is having people using hundreds of thousands of
tokens a year, then it becomes less worth the more we worry about the token cost, the less proportionately people might worry about the human wage costs.
So I'm I I'm not sure whether it's a dystopia for those of us who are already skilled software engineers, though it it might make the economics weird for for others.
Yeah.
Well well and you raise the the Jensen spector Well, to your point I raise the Jensen Spector but but to name him, I I keep thinking to myself, as long as he can make this
cheaper, as long as he can keep making the chips better for less money, you know, ~ then we are going to be fine.
~ yeah.
Which almost kinda brings me to another question.
~ so I've I've had this analogy bouncing around my brain for a little while and I haven't
managed to talk to anybody about it.
And so I'm gonna run it past you.
So you
Just between
you and me and and our ~ thousands
of listeners, yeah.
Yeah, yeah, exactly.
~ and and it is the analogy of FedEx and the fax machine.
~ and so some listeners will know the story, but for for those who don't, I'll tell it.
it's my understanding that when the fax machine, the humble fax machine, right, this essentially black and white printer combined with scanner, combined with modem.
that you know can sit on a table and can be plugged into a phone line and plugged into a power line, they can now call each other and send documents back and forth.
And you can drop, you know, a letter in the scanner side and it will scan it and then it will call somebody up and it will get printed remotely on the other side because print
jobs are essentially data.
And so we can move those over phone lines.
And it as I understand it, FedEx when they first saw these, thought, this is great.
You know, Chris is in Barcelona, Kyle's in Toronto.
We can put a fax machine in each office.
If Chris wants us to send something to Kyle really, really quickly, we can fax it.
And it won't look as nice.
It won't be the original, but we can get it to Kyle for so much less.
And like maybe we don't have to charge all that less.
And when fax machines first came out, they were 10 grand a pop.
So this maybe seemed like it was gonna be a thing.
But what they didn't realize is that they would end up being 80.
~ and so pretty soon everybody would have a fax machine, you know, and and big companies almost immediately realized well, yeah, we should just do.
Instead of paying FedEx to do this.
And eventually medium-sized companies, small companies, people, and then eventually the internet, and it was all all over.
I wonder if this might apply to AI.
You know, companies like Tiiny AI are building these
tiny little things that you can now run a reasonable model in.
And the better and better the algorithms get around compacting models down and you know and reducing the amount, making it more efficient for tokens to do their work or for
models to do their work.
yeah.
Like will this be the kind of thing that, you know, we're just gonna run it locally?
At that point, that's really gonna change the economics of this entire game.
So but what do you
think?
Like am I dreaming?
Like what what do you think?
Well, I I can certainly come up with a direct parallel from what I've seen.
So the problem there was that FedEx were thinking about competing against their competitors as though their competitors didn't also have fax machines, or also their
customers didn't also.
So they're kind of imagining a world where they could compare everything else saying static and then moving forward.
I do actually see that quite a lot in the software world where companies kind of simultaneously going, ~ code is code is cheap with coding agents, that's great.
But they're still pretending that
If that were true that code is valuable and that's
not an equilibrium that can hold.
So, you know, companies are thinking, okay, I can have higher margins because I can do what I used to do much more cheaply.
That's awesome.
But they're may they're thinking the fallacy that they're operating in a static world and they're comparing
their offering to what their competitors or substitutes were able to do before coding agents became popular.
So I think there's definitely a lot of like equilibrium settling that will need to happen.
It would definitely be ironic though that if the
you know, if the large L L providers having, you know, done the R and D and received a lot of capital to eat the world with L L Ms, were out competed by
locally artisanally home grown run things on everyone's laptop.
~ but I guess we'll see.
models are good enough and fast enough, once they're smarter than us, you know, like and and we can run it on, you know, our phone, ~ then does it matter if it's 10x smarter than
us or a hundred X smarter than us?
Like maybe.
and and this may not be for a while.
I mean things have changed fast in the
last two years, but maybe not this fast.
But so yeah, if but if it did happen, if it did get to the point where most people could get just fine results.
~ on their phone or on their PC, on their Mac, laptop, whatever it is.
I don't know.
Like I wonder what we're paying other people for the tokens for.
Yeah.
I guess so will our just fine remain static?
Because that that may sound a little bit like I the thing that Bill Gates didn't say about was it I don't know, like a it should be enough for anyone, right?
I don't think he actually said
that, but but but that's kind of saying, okay, well that amount of intelligence should be just fine.
I can't see I can't see why anyone would need more than that.
And I think I do have a little bit of faith slash pessimism, I'm not sure what emotion to have, about our ability to find ways to waste, consume
at least desire the additional capacity if it if it exists in the world.
I think the idea
of us going, Yeah, that's fine.
We've had enough, ~ unfortunately is not something that our civilization has done very often in the past.
So we'll see.
the hot thing is right now.
Whatever the thing that used to be hot, we want to optimize the heck out of that thing.
But whatever's hot right now.
talking about climate change, but you would no you were talking about more metaphorical, but yeah.
That's a whole other podcast.
Yeah, exactly.
Fair enough.
I w I I I won't try to sidetrack there.
But yeah, I I I th I think there there will be desire to increase consumption and things becoming cheaper increases consumption is the Jevons paradox that has suddenly become hot
~ again, right?
So
~ it will be interesting to see.
But I maybe there's a stage that we could consider in between the two of running locally and using the frontier
model providers, which is there are a lot of companies that are using
their cloud hosting providers to run models for their own in in house inference as a way of either an insurance policy against rising token costs from you know anthropic open AI
Google,
or even just a way of trying to do it cheaper because as long as they have the ability to capacity plan and manage utilization, they can just take the margin of the
cloud of the the token provider out of that.
So they so I think companies are definitely making they're trying to make sure that they have levers so that they aren't dependent on a supplier that might decide to massively
hike its costs.
Yeah, well, or or foreign government.
Like we we had a guest on Ahmed
Misbah and he's he's working on Arabic LLMs, right?
And I think it's it's a couple of things.
One is, you know, the language and being able to to yeah, there's an interesting computer science thing about what happens when we teach at other languages besides English.
~ but I think more to that, it's also just sort of sovereignty, you know.
~ and you know, Canada, where I live, ~ we have one of our telcos is standing up data centers.
know, with NVIDIA GPUs in them.
We're not we're not creating our own GPUs, but
but the idea of a sovereign AI, we do have ~ local model companies.
You know, there's a Canadian model ~ I'm not gonna remember the name.
That's gonna pain me.
but yeah and so the idea that we can't all be beholden.
We we saw what happens when the government of the US decides to tell Anthropic, I'm sorry your model is too something
and so you have to turn it off now.
I'm not sure we all want to be beholden to that in the end.
Yeah.
Yeah, if I understand correctly, there's rumblings in China of actually maybe not wanting people to use the Chinese models as freely in the West, ~ the way the way that's
happening.
So, you know, there could be
controls in that direction as well.
Or, you know, I mean I live in Europe
and ~ not wanting to depend on external parties, be they North American or Asian for critical infrastructure is a really hot
topic in Europe as well.
Yeah, yeah, yeah, for sure.
So I think that's gonna be an element of it.
And you've you've almost we're almost into like globalism now and you know, should you know, should countries depend on one another or should we be isolationists?
Like a we're definitely trying to solve the world's problems here, Chris.
Yeah.
Yep, yep.
We might need a a third
a third podcast to to completely finish the job, but you never know.
Yeah.
My my only counterpoint to your argument around Jevon's paradox and we always
want more of a thing.
I mean, yes, you're right.
And I could see it going that way.
Like it certainly has gone that way with say bandwidth.
Like why for instance, you know, my first ~ cool cable modem was nine hundred and sixty kilobit per second.
That's almost a megabyte, Chris.
my god.
A T one is a three thousand dollar a month thing, and this is almost that and it only cost me a hundred and fifty.
And now I've got a gigabit connection.
And if someone offered me a ten gigabit, I would take it.
What I knew what I'm doing with the other nine gigabits, I don't know.
But I would take it, right?
So maybe it goes that way.
On some level, it's gone that way because it was commoditized and because it was cheap, and because we could keep saying yes at this at roughly the same hundred bucks a month.
I think the argument on the fax side was with fax similes, we it didn't really matter.
We didn't need to be able to send, you know, 7,000 page faxes in thy in three seconds.
That was never of value.
~ that basically what the fax was doing was like, look, I got a wet signature, you know.
well it isn't wet, but I got a signature, you know.
And that's that's mostly what it was doing in the end.
~ you could send the other 25 pages in a Word doc in an email eventually, you know.
And so Good Enough had a different bound.
~ and and because and I think prices were were something about what kept that in check as well.
Like
if ~ we if we wanted to send, you know, that 7,000 page document in two seconds, we eventually did that on our gigabit line
Computers because again, that's where the commodity was.
That's where you could commodify and bring the price down.
So I don't so yes, Jevon's paradox, yes.
I mean, you know, I'm I'm a big electrification fan, and there is definitely a threat that the more we electrify, the more we use electricity, which might just mean we have to bring
the coal plants back on, and that's not what I'm after.
So I get it, I'm with you.
But I think there's another side of it though.
As long as we can keep optimizing the hardware and we can stay just ahead of the efficiency curve.
And we can commodify, commodify.
Well, eventually I will have that ten gigabit connection ~ to my house and it'll only be a hundred bucks a month.
They probably already have it in Korea, you know.
So
I think you're definitely right to think about diminishing marginal returns and what people will actually want.
I think that that that that's true and no no company is kind of owed a living by the world.
~ so yeah, I think you you do you make a good point and there may be people certainly part of the supply chain that are routed around like ~ like
FedEx was.
Yeah, yeah, yeah.
It's interesting times.
maybe the thing that I could add on to that idea about to what degree people will be willing to pay more for stuff.
I think I believe that that attribute of humanity might make it very difficult to improve the quality of software using coding agents.
So I think I do believe that if you were to do a great job with all the different kind of harnesses, the feed forward, the feedback, guardrails and whatever, that in theory,
someone who then maintains the same amount of energy and discipline can do a better job than if they were coding by hand in many cases.
So I don't you I I I I don't
subscribe to kind of a slop narrative to that implies that coding agent will automatically do a worse worse job.
I think you can do a better job with coding agent, especially
if, for example, you invest some of the time you save in improving the harness or doing more review.
The conclusion I've reached though is that if we looked like we were accidentally creating greater quality, we would respond not by achieving that quality, but by lowering our
standards or increasing our haste until we manage to lower quality back to where it was before.
Because the level of quality we have in software is not it's not the maximum we know how to achieve, or at least I don't think it's been in any code base I've been part of.
It's basically it's lowers
the conditions we'll accept.
And if it gets lower and things start breaking, then people will get it so it's just above the waterline.
So there there's a there's a paradox called Wirth's it's Wirth's law or Wirth's paradox, the after the computer science.
And that explains why using a computer doesn't get faster.
Like you think it should get faster to use a computer because computers are getting faster.
How could that how could that not be the case?
Well,
because the faster the computer gets, the more layers of crap we put on our stack until it slows just at the point where we can bear it.
You would think, you know, I've heard people say many times, Well, hang on a second, can we just stop developing more weird layers of abstraction for like one generation of
hardware and just take the win?
Wouldn't that wouldn't everyone love that?
And maybe they would, but certainly the revealed preference is that they don't really
give a crap of that and that they're willing, at least us as software developers are willing to
to to fritter away.
So I think we might have the same quality equilibrium problem where even if the techniques are available to produce higher quality software with better modularity, with more
stability, with fewer bugs with coding agents,
Yeah.
if we achieve that, we'll say, well, that means we can take a little bit less care next time.
That means that we can go a little bit faster next time until the quality drops to the level that people were kind of
Grudgingly accepting anyway.
So I think there's I think there's an equilibrium force there.
I like it.
If I if I borrow the analogy for a minute and
and riff on it, then what's coming to mind for me, ~ and I I've just now heard of Wirth's law, so I'm I'm probably gonna get it wrong, but I think it is sort of a counterpoint, I
guess, to to to Moore's Law.
And I always thought, like, yes, you know, you know, the ~ w CPUs get faster and faster and faster, so why don't they get slower and and I I always figured it was sort of two two
debts in the budget, ~ two things that that was going to.
Maybe maybe three.
Like maybe one is like, okay, you know.
faster can software can go faster, so we don't have to performance optimize as much.
So that's like going to the margins of the software development companies in theory.
They didn't have to spend as much money on performance optimization.
So I think that's a one bucket that it's going to.
I think another bucket that it's going to is like shiny features.
Like you and I right now are in different countries of the world and yet we're having a very high resolution conversation as though we were not.
That was not possible on my 486 DX33.
It just was not.
~ and so that's that's like a budget.
But I think there's this other budget, which is like technical architectural debt.
Like Linux still has POSIX in it.
and so so does Windows.
Like, because for a minute we all thought that we needed this layer of translation called POSIX,
which from what I can tell didn't ever do anything, right?
Like no one ever actually got any value out of POSIX.
I'm probably wrong.
Probably someone did.
But all the computers still have a POSIX layer in them.
So I figured like if Morris Law ever did stop.
That it would hopefully the budget would come out of that.
Like we would sort of go back to our technical debt, these old abstractions that don't matter anymore, and like let's strip them out of the operating system.
Are we gonna get a couple of milliseconds back?
I don't know.
But I ~ so how does that apply to this?
Not not sure.
But what do you think?
Well, I think well I think it it it it's a neat example because it works for both, you know, technical debt for quality and for speed.
I I I I guess I think we'll find our ways to screw up software in different and imaginative ways in the future is maybe my my conclusion.
So it
will be possible well was always possible possible with coding agent, it will be possible to maybe refactor or remove code where we have a clear idea of what the outcome should be,
but the human labor to do it would be
gargantuan but we can specify the problem and put a harness around it really clearly, sure, then we can go ahead and do it.
~
still you have to worry about compatibility and distribution and things like that that would be difficult if people are
using POSIX compatibility as as as as something as the interface that wouldn't be as easy.
so I it's it's not that the the same specific subsystems that slow us down or exactly the same anti-patterns that cause quality problems.
will be present.
So maybe in the future C based memory bugs won't exist anymore because we'll use coding agents to rewrite everything in Rust and and that'll be done.
But I think there'll be a substitution
of other quality problems that we will tolerate up until
our level of ~ acceptability of our employers slash the patience of our users and just at the point
before they throw the laptop down in disgust, we will
we will step in and just
Just keep it just good enough.
so I I I I know this is a kind of a pessimistic take, but I'm just trying to imagine the organizations I've seen tolerating software improving.
And I just can't say 'cause if there was a bunch of engineers that came in and said, Okay, great news, software is twice as reliable, let's say, I'm talking about kind of bugs and
things like that as it's been before, the you know, a lot of the response of organizations will be, Well, why did you spend so much time on it then?
~ and
Unless unless
maybe as consumers we can demand our share of the surplus introduced by coding agents to be paid to us in increased stability or performance, ~ I think it will get ~ eaten maybe
by like many other things into the you know, into the profits of the company that is sitting at the right point in the value chain.
Yeah, a hundred percent.
I I can remember, and this is now twenty years plus this is pre-thought works days for me, so I won't name anybody, but it's it's nobody in my life currently.
Going to the CEO and this is when I was first discovering test driven development and unit testing, all of this stuff.
And I remember going to the CEO and saying, Boss, we're gonna start unit testing everything.
Quality's gonna get better.
I it might slow things down a little bit.
I don't know.
And he's like, I'm not sure that quality matters that much, Kyle.
No, no word of the lie.
And he's not wrong.
You know, they invest or put money in because they want to see money out.
He wasn't wrong.
Now, luckily the unit test sped us up.
It didn't slow us down because, you know, it just helped f cut things off before they launched.
the there is the quality is free, which I think definitely does apply to a certain kinds of quality and unit testing
is great.
It's definitely a quality is free phenomenon,
yeah.
Yeah, yeah, yeah, yeah.
But not all of them are.
The other thing that's coming up for me is they originally like internet lines were sort of like the same telco lines.
~ and telco lines were built for telco levels of reliability that needed to be there because they were centrally managed and circuit switched instead of packet switched.
And people started just saying, Well, screw it, just make it an Ethernet line between two cities, you know, and it obviously can't be the same copper coax you
you'd end up having RF issues and so on.
But but still like drop the telco levels of reliability that telco expected because the internet was architected not to care as much about reliability.
And that that caused, I can remember buying a T1 line in Kingston, Ontario, and it was three thousand dollars a month.
And they, you know, I got a quote for, you know, for basically the same mega and a half from somebody else.
And it was like, I don't know, $300 a month.
And I remember calling up my bell rep and he was like
Well, you could, but the quality is gonna be terrible.
But Chris it wasn't.
Like it was fine, you know, like maybe ~ the the likelihood that that there would be a problem.
Well, that's not even true, because later I I don't know if it's the same year, but a bell technician dropped a a wrench in an open router and took out like most of Toronto for a
couple of hours.
So even my three thousand dollar T one line, and that's when I sort of gave up and said, Okay, this is dumb.
~ so I do think there is something like that going.
Around quality where there's this sort of like b if we if we we have to just understand now that everybody's gonna be doing four things at once, everybody's got a an app that
they just put in the marketplace, everybody's got a startup that they launched, we're gonna have more things, right?
Did as much care go into each one of them?
Probably not, because you could do it faster, which is why you did it in the first place.
But is that a bad thing?
Maybe not, right?
Like it might be like that hundred and fifty dollar T one line where it it really was okay.
In the end.
Yeah.
Some of the quality
serves humans' needs better better than if if it was high quality in a purist way.
That could be true.
Yeah.
And that's not to say it'll only go well.
I'm I'm sure in places it will go terribly and will tell interesting stories and it'll be funny.
Yes, well I guess, you know, with all sorts of feedback loops, if it takes a long time to figure out that you've done something bad, then
you're much less likely to gracefully adjust, you know.
A steering wheel of the
car where you can where you ~ can correct course correct instantly is fine.
If you were trying to drive a car with a steering wheel that had a lag of, I don't know, let's say a second, your chances of getting through any you know, if seconds not very
long, but
your chances of getting through traffic
unscathed would be very low.
And so
I might the analogy I'm making is that for things like, I don't know, accidentally lowering the quality of the software in your state or poisoning the well of open source,
for example, if it takes a
a while before the implications of our actions to come back and bite us, that doesn't
like it we're not necessarily going to take holistic long term rational decisions.
We might
find that we've messed things up before before we we know it.
I think it's it's it's generally thought of that we are bad at seeing those kinds of things as a species.
In fact.
Yeah.
Yeah, well that maybe if I could relate it to kind of ~ my my theme, I think we talk a lot in terms of feed forward about how we respond in a society.
~ we should do this.
Okay, everybody, let's let's do this.
We we imagine that is how as human beings we make decisions.
But I think, you know, feedback and things going wrong and being forced to do things when there's no alternative is much more how things change at a social level over time.
And so if we don't have the
if we're in a context where we don't have good feedback and maybe this relates to the climate change question the message before, if we you know, if our if our actions and our
con our consequences are separated by enough distance, ~ you know, it it might take us a a while to to to change what we do.
Yeah.
Fingers crossed.
We're gonna need all the luck we can get right now.
Yeah.
so.
Chris, thank you so much.
This has been great having you on a second time.
Thank you.
I I feel like I was you got pessimistic, Chris, this time round in in in various ways, but ~ I'm definitely really enjoyed the conversation and ~ hopefully we've increased the
quality of our ignorance together.
Amen.