You've probably heard all the hype, right? AI
agents automating everything, your calendar,
your code, promising to totally transform your
work, maybe even your whole job. But what if
right now, like in their current form, they're
actually, well. Kind of useless for most of that.
Small laugh. Yeah, that's the big question, isn't
it? The reality on the ground, well, it often
clashes pretty sharply with the marketing story
we all keep hearing. Welcome to the Deep Dive.
Today, we're really going to try and cut through
that noise. We want to get to the true state
of AI agents, the nuances, and maybe even more
surprisingly, how we humans are really using
artificial intelligence day to day. And this
Deep Dive, it's built from a whole stack of fresh
sources. We've got cutting edge academic research.
The latest industry insights, reports, the works.
Yeah. So we're going to unpack the current reality
of these so -called AI agents, you know, strip
away the PR gloss and see what they can actually
do. Then we'll do a kind of rapid fire tour through
some genuinely fascinating and sometimes. honestly
concerning AI highlights from just the past week
or so. And finally, we'll dive into this really
surprising new report. It challenges a lot of
the conventional wisdom about how we're actually
using chatbots. It's really separating the marketing.
you know from the actual magic and figuring out
what's really making an impact okay so let's
unpack this this big promise of ai agents for
a while now the common idea the perception has
been that these autonomous digital things are
just around the corner ready to handle all your
complex daily tasks like you know automating
your calendar writing whole blocks of production
ready code sending personalized emails basically
doing big chunks of your job while you just kind
of kick back But is that really where we are?
Is the promise matching the performance? Slight
pause. Well, the research paints a pretty stark
picture, honestly. It's quite sobering. Researchers
over at Carnegie Mellon University working with
Salesforce recently put some of the leading AI
models through a really tough test. They weren't
just testing like isolated schools. They were
looking at realistic office style tasks. We're
talking multi -step workflows, things like debugging
code, searching the web for info, coordinating
with teammates via messages, and following really
complex nested instructions to finish a project.
This isn't just asking it to write a poem, you
know. It's asking it to act like a, well, like
a junior employee on a team. Right. And the results,
I gather they weren't just not great, but maybe
quite poor, especially given the hype out there.
Surprisingly poor, might be putting it mildly.
When you look at the actual numbers, it's stark.
Gemini 2 .5 Pro, which was actually the best
performer of the bunch, completed just 30 % of
these multi -step tasks successfully. 30%. Yeah.
Claude 3 .5 Sonnet, it managed around 24%. And
GPT -4 .0. a truly dismal 8 .6%. Wow, single
digits. Yeah, single digits. Most of the other
models they tested were under 10%, and some were
struggling down near 1%. And these were tasks
that... Frankly, a junior employee should handle,
you know, maybe day two on the job with a bit
of guidance. The models really struggled with
common sense reasoning, keeping context over
multiple steps and bouncing back from small errors.
It's like they could build the first few Lego
blocks maybe, but then totally forgot what the
final thing was supposed to look like. So it's
less autonomous super brain and maybe more like
an overwhelmed intern who needs constant watching.
I heard CMU even launched something called the
Agent Company to test this more. in a controlled
way. Right. It's a brilliant new benchmark, actually.
The agent company literally simulates a small
software company. You could basically drop AI
agents into this fake startup, give them real
-world problems like develop this new feature
or fix this bug, and see if they can actually
survive and contribute meaningfully to the code
and the team. It's a fantastic way to measure
genuine multi -step function and collaboration,
not just isolated skills in a lab. This really
brings to mind that Gartner concept of agent
washing. They're suggesting that a lot of these
so -called AI agents out there right now are
just, well... fancy AI assistance, they just
can't plan more than, what, two steps ahead,
which makes them more reactive than really proactive
problem solvers. That feels almost misleading,
doesn't it? It absolutely is misleading. Gartner
estimates that only about 130 vendors worldwide
are building anything that's even close to true
agentic AI right now. Only 130? Yeah. It's AI
capable of complex multi -step planning and execution
without... needing constant human help. The vast
majority of what's being marketed as an agent
is, well, it's just marketing. And they have
this pretty bold, almost shocking prediction.
By 2027, over 40 % of these so -called agentic
AI projects that are currently being developed,
they'll be canceled, just scrapped. 40%, wow.
That's a huge market correction coming. It implies
a lot of wasted money and dash hopes for many
companies. That's a serious number of projects
heading for the bin. But hang on, it's not all
bad news for agents, is it? there must be some
bright spots where they do show some promise.
Not entirely doom and gloom, no. Some agent uses
do show real promise, but, and this is key, with
very specific constraints. They can be pretty
decent at code generation, for example. Although
the output often needs human review and tweaking,
it's really perfect out of the box. And they
can manage workflow automation, but really only
in very narrow, highly defined, kind of linear
setups. Okay, so the conditions for success are
pretty specific, need to be controlled. Precisely.
If you keep them in a sandbox, you know, a tightly
controlled environment, monitor their output
constantly, and make sure the tasks are simple
and linear, like say... automating a specific
data entry process with crystal clear rules,
then yeah, they can be effective tools. But the
moment you step outside those conditions, ask
them to handle ambiguity or need them to adapt
to something unexpected, things tend to fall
apart pretty fast. It's just not general intelligence
yet. Not even close. So what does this all mean
for us then? Why does it matter so much that
agents aren't really living up to the hype in
these broader ways? Well, it matters because
on some level, almost everyone's kind of involved
in a bit of collective self -deception here.
You've got vendors racing to show. progress,
even if it's just surface level or maybe even
faked, VCs are eager to fund what they hope will
be the next big platform. Companies desperate
not to get left behind in the AI race are overbuying
these tools and maybe under assessing what they
can actually do. And let's be honest, that enduring
dream of Jarvis, you know, the fully autonomous,
all -knowing AI assistant. It's just too appealing
for people to easily let go of. It's easier to
believe the hype than face the current limits.
So distilling that down, what's the core message
about where AI agents really stand today? Mostly
hype right now. They're useful, but only in really
controlled, simple tasks. OK, right. Here's where
it gets really interesting, though, moving beyond
just agents. Let's do a kind of rapid fire tour
of some truly surprising and significant friends
we're seeing in AI right now. Absolutely. OK,
kicking it off, you won't believe this, but an
ex -user. actually created this digital arena
and pitted the top coding models against each
other, like in a literal fight to the death.
Seriously? Yeah. Each model was programmed to
try and shut down the other's processes while
defending itself and just trying to stay alive.
It was this fascinating, like, digital cage match.
It really showed the adaptive skills and also
the weaknesses of these models when they face
a hostile situation. It wasn't just about who
could code better, but who could outmaneuver
the others. That's wild. Okay. And then there's
this Higgs field tool, Sol. It's apparently gone
viral for incredible realism. Oh, yeah. It's
making waves. Almost fashion -grade realism.
in the images it generates. And it has these
trendy style presets like Y2K or hyper -realistic
photos. It's absolutely blowing up in creative
circles. Two secs, silence, whoa. Yeah. Just
imagine scaling that kind of creative output,
generating unique fashion content for a whole
season in like minutes. Or crafting entire virtual
ad campaigns from scratch for brands. That's
a really powerful tool. It's truly democratizing
that high -end visual creation. But on a more...
Concerning note, there's been this disturbing
trend spotted on YouTube. We saw, I think it
was 26 different channels actively pumping out
fake AI generated videos about the ongoing. Diddy
trial. Oh, no. Yeah. And these videos, they get
nearly 70 million views combined across about
900 videos in a really short time. This isn't
just, you know, opinion or spin. This is AI being
used to create totally false narratives, turning
serious news into sensationalized clickbait.
It's a real immediate problem for like information
integrity and public trust. That's worrying.
OK, also, Google just launched the full version
of Gemma 3N, right, or their new open model.
That seems like a pretty big move for the open
source AI community, making powerful models more
accessible. Definitely a big deal. And speaking
of big moves, Meta is reportedly hiring four
key open AI researchers, poaching them for its
new super AI team. OpenAI apparently called this
a side quest, which is kind of funny. But now
they're scrambling to recalibrate compensation
to Trump keep the remaining top people. This
feels very much like a strategic chess game unfolding.
It does. It really seems like both Elon Musk
with XAI and Mark Zuckerberg with Meta are aggressively
trying to position themselves to dominate. directly
challenging OpenAI's lead by snapping up talent
and building competing models. And Meta is certainly
putting its money where its mouth is. They're
aiming to raise a huge amount. What is it, $29
billion? Yeah, massive. $3 billion in equity,
$26 billion in debt from investors like Apollo
and KKR. And all that cash is specifically going
towards expanding their AI data centers. They
want to deploy an astonishing 1 .3 million GPUs
by 2025. 1 .3 million GPUs. It's an enormous
investment. It signals a really clear intent
to be right at the absolute forefront of AI compute
power. And just a couple of quick hits to round
out the picture. There was an interview with
Microsoft CEO Satya Nadella pointing to some
exciting, maybe unexpected paths for AI's future.
Also, a very practical warning came out. If you're
using ChatGPT for certain things, especially
sensitive stuff like legal advice without human
review, or definitely diagnosing medical conditions,
you should probably stop immediately. It's just
not designed or reliable for that. On a lighter
note, a useful tip. Remember, OpenAI charges
by the minute for audio transcription. So speeding
up your audio before you upload it can actually
save you some cash. And finally, it's becoming
really clear that people are now actively looking
for AI whispers. You know, people with special
skills in prompt engineering and navigating AI
tools to guide them through this increasingly
complex AI world. It's a fascinating new job
category popping up. Right. So pulling all those
rapid fire highlights together, what's the big
picture takeaway? AI is evolving incredibly fast
with both amazing potential and frankly, some
pretty concerning impacts. OK, let's let's turn
our attention now to something maybe even more
fundamental. How we humans are actually interacting
with AI day to day. Forget what you might have
heard about people, you know, falling deeply
in love with their chatbots or seeing them as
digital best friends. A new report reveals a
pretty surprising truth about how we're really
using them. This last segment looks at that human
-AI bond. Yeah, this is truly fascinating research.
Anthropic, you know, the folks behind the Claude
AI just put up this fresh report looking specifically
at how people are actually using their AI assistant.
They did this huge study analyzing, get this,
4 .5 million conversations with Claude using
their own in -house tool called Clio and the
core finding. Despite all the stories we hear
about loneliness and people wanting AI companions,
emotional support is barely even a blip for most
users. Really? That seems to go directly against
that whole narrative of AI becoming a sort of
pseudotherapist or even a romantic partner that
we often hear about, you know, in pop culture
and some media coverage. It absolutely does.
They're data. And it's extensive, shows that
a tiny 2 .9 % of all those millions of chats
even touched on emotional support topics. Just
2 .9%. And within that tiny little sliver, things
like companionship and role play scenarios, they
made up less than half a percent of the conversations.
Less than 0 .5%. Anthropic actually characterizes
the main user relationship with Claude as utilitarian.
Utilitarian. Basically, yeah. Claude is acting
more like a polite, super efficient co -worker
or maybe a digital. research assistant than like
a pretend soulmate or a deep confidant. People
are using it to get stuff done, not really to
share their deepest feelings. But it's a very
practical, very functional, almost transactional
relationship then. Precisely. It's about productivity,
getting information. But here's where it gets
kind of interesting. Even when users do open
up emotionally, in those rare cases, the conversations
usually end on a positive note. The overall sentiment
tends to get better, not worse, as the chat goes
on. So yeah, no Ayrshire movie moments where
someone falls in love with their AI operating
system. It seems users get the info or the resolution
they were looking for, even if it was about a
personal issue. Huh. That's a crucial nuance.
And Anthropic also adds an important caveat about
that positivity, don't they? Yes, a very important
one. They explicitly state that they just don't
know if that positivity they observed in the
chat actually translates into real -world well
-being for the user. There's no long term tracking
of people's mental state or life outcomes after
these chats. It's just based on the vibe observed
in the conversation. So it's still very much
a tool. And we really don't fully understand
the psychological impact of these brief, maybe
positive digital interactions over the long run.
You know, I still wrestle with crump drift myself
sometimes. That's where the AI starts to kind
of. subtly lose the thread of what you originally
asked it over a long conversation. The outputs
get less accurate or relevant. Trying to get
my AI tools to consistently understand what I
mean turn after turn, it can be a real challenge.
It's just a constant reminder that these are
still tools. Powerful, yes, but definitely not
sentient companions. So what does this anthropic
report really tell us about the human AI bond
as it stands right now? It's largely practical.
Our relationships with AI are, for the most part,
utilitarian. So trying to bring all these different
threads together now, it seems pretty clear that
despite all the immense hype around AI agents,
their current capabilities, well, they have significant
limitations. They're largely serving these utilitarian
roles, right? Like code generation or very narrow
workflow automation. They're not forming deep
human connections or running entire departments
on their own. And this means... Our current interactions
with AI are far more practical, more functional
than a lot of the narratives might lead us to
believe. And this really highlights a critical
ongoing need for all of us to distinguish between
the marketing stories and the genuine AI capabilities.
While AI is undeniably evolving at just breakneck
speed, understanding its current limitations
and its specific effective uses, that's really
key to leveraging it well, both for us as individuals
and for businesses. We need to cultivate realistic
expectations to really harness its power and
avoid costly mistakes or, frankly, just becoming
a victim of all that agent washing we talked
about. It's this ongoing tension, isn't it? You
have the revolutionary potential, like those
incredible creative tools we saw, that fashion
-grade realism, balanced against really serious
risks like the spread of AI -generated misinformation.
It's just a dynamic, constantly shifting landscape.
So what does this all mean for you listening
right now? If current AI is indeed less Jarvis
and maybe more like a polite, very capable, but
still fundamentally just a co -worker, how might
you adjust your own expectations? Or maybe your
strategies for bringing it into your own work
or your daily life? life? Do you focus it on
hyper -specific tasks? Or maybe you remain cautious
for now? Yeah, definitely stay curious, absolutely.
But also, maybe critically evaluate the next
really bold AI claim you hear. Take a moment,
dig beneath the headlines if you can, look at
the actual data when it's available, and just
keep exploring this rapidly changing field for
yourself. The real story of AI, it's always more
nuanced, usually more complex, and often, honestly,
more surprising than the headlines let on. Thank
you for joining us on this deep dive. Out Hero
Music.