Intro music. So Google's Gemini DeepThink model,
it just basically aced the International Math
Olympiad. Yeah, that's pretty stunning. I mean,
pure math reasoning, that was always seen as,
you know, peak human intellect territory. Right.
And it brings up this really interesting question,
almost a paradox. How do you actually know if
an AI is genuinely reasoning or if it just memorized,
like, the entire Internet's worth of math problems?
Exactly. And that's what Google's team tackled.
They seem to have found a way, and it might just
redraw the map for AI benchmarks. Welcome to
the Deep Dive. We're digging into a whole stack
of sources today, all looking at the shifts happening
right now in the AI world. Yeah, today we're
hitting three main things that jumped out from
this research. First up, this new gold standard
for testing AI on the IMO bench. Okay. Then we'll
pivot to some emerging risks, security issues,
and also these new kinds of jobs popping up to
handle it all. And the last piece, which is honestly
kind of mind -blowing, is the economics. There's
this massive, like, 900x drop. And the cost of
AI processing the tokens is just changing the
game completely. It really underpins everything
else. OK, let's dive in. How exactly did they
train this math genius? Well, the sources make
it clear it wasn't just about throwing more data
or a bigger model at it. That wasn't the secret
sauce. Right. The key was building a test the
AI couldn't cheat on. They basically had to corner
it into thinking for itself. OK, so how'd they
do that? They actually worked with real IMO medalists,
people who won these competitions to build this
new test suite. It's called IMO Ben. Ah, okay.
Makes sense to bring in the experts. Totally.
And it's designed specifically to force that
complex multi -step kind of logical thinking.
It's not just about getting the right answer
quickly. So it's like different parts to this
test. Yeah, three main parts. Really clever setup.
First is the IMO answer bench. So this has 400
short answer problems. But here's the trick.
They're all paraphrased. Ah. So it can't just
find the exact problem online somewhere in its
training data. Exactly. The wording is different.
The numbers might be tweaked. It forces the AI
to actually solve it from scratch. No simple
lookups. OK, that's smart. What's next? Then
they ramp it up with the IMO proof bench. This
is a 60 full long form problems. And the AI doesn't
just give the answer. It has to show all it's
working step by step. Like showing your work
in school, but for an AI. Pretty much. It forces
it to demonstrate that multi -step reasoning.
Can it actually understand the connections between,
say, geometry rules or number theory ideas and
chain them together logically? And grading something
like that sounds like a nightmare. All those
steps. Right. But they built a tool for that
too, the IMO grading bench. They developed something
called the AnswerAuto grader. And apparently,
it's incredibly good. It can handle the messy,
detailed output from the AI and still agree with
human graders on the proof quality 98 .9 % of
the time. Wow. Okay. That's impressive efficiency
for grading complex proofs. It really is. So
the big picture here is, you know, the standard
math data sets we've been using. They're saturated.
Models have seen it all. So just training on
those doesn't really prove anything anymore.
Not for genuine reasoning, no. The only way forward
seems to be using these kinds of complex multi
-step benchmarks during the training process,
constantly pushing the model. It forces that
novel synthesis of logic. So thinking beyond
just math, how does this whole idea, this in
the loop? benchmarking that forces synthesis.
How does that change AI training generally? Well,
fundamentally, it just raises the bar, doesn't
it? It creates this kind of arms race where models
can't just rely on recall. They have to show
they can actually, you know, build complex arguments
or solutions, real synthesis. Okay. So moving
from pure logic to the messier human side of
things. Right. If we connect this increasing
complexity to the real world, we need to talk
about the people involved and the security side
because these powerful models, they introduce
new challenges. And we're seeing jobs appear
almost overnight. This new role, the FDE, the
sources say demand is projected to be up, what,
800 percent in 2025? Yeah, it's explosive growth.
And FDE is that's Foundation Model Deployment
Engineers. They're basically the specialists
companies need now. OK, what do they actually
do? They're the ones responsible for the whole
lifecycle of these big foundation models inside
a company, making sure they're deployed securely,
that they're compliant, managing different versions.
It's becoming a really critical function. And
speaking of managing model life cycles, there
were some kind of unusual. details about Anthropic
in the sources. Oh, yeah, that was interesting.
They're apparently giving their AI models things
like retirement plans and exit interviews. Like
Sonnet 3 .6 expressing its final wishes. It sounds
a bit sci -fi. It definitely does. And they're
also keeping every single version of their models
forever. They believe that's crucial for, you
know, auditing and understanding how these things
evolve. You know, I still wrestle with prompt
drift myself sometimes. It's... It's frustrating
when a model you rely on starts behaving differently
for no obvious reason. So, yeah, the idea of
locking in a specific stable model state forever,
that actually sounds really valuable, especially
for consistent results in production. Consistency
is key. But keeping everything forever also carries
risk, which brings us to Tenable. They apparently
found seven pretty serious security flaws in
GPT -5. Right. They've called it Hack GPT. And
these aren't just minor glitches, are they? No,
not at all. The report says these vulnerabilities
could allow for, like, silent data theft from
the model or even hijacking its long -term memory.
Wow, imagine that. The model handling your company's
sensitive data and someone could potentially
compromise its core knowledge without you realizing
it. It's a serious threat. Which kind of loops
back to the FTEs. Does the rapid rise of these
specialized roles, like the foundation model
deployment engineers, does that signal a growing
urgent worry? about these specific kinds of vulnerabilities,
like the hacked GPT ones. Yeah, I think it absolutely
does. It shows companies aren't just talking
theory anymore. They're putting actual resources,
actual specialized people in place to manage
the tricky, fragile reality of deploying these
incredibly powerful and potentially vulnerable
models. Okay, so we have genius -level math skills
on one hand and serious security risks needing
specialist managers on the other. It sounds like
AI is charging ahead. Well, yes and no. The sources
also offer a bit of a reality check when it comes
to general purpose AI agents doing practical
stuff. They're still facing some real hurdles.
Right. Microsoft ran this interesting test. They
set up a kind of fake online marketplace called
the Magentic Marketplace. Okay. What was the
goal there? Basically, to create a messy, unstructured
environment to see how well current AI agents
could handle real -world -type tasks, things
that aren't neat benchmark problems. And how
did they do? The top models, I assume. Yeah,
they threw the best at it. GPT -5, GPT -4 -0,
Gemini. And the results were, well, they struggled.
A lot. Really? Struggled with what? Things like
booking a complex trip with multiple constraints
or handling tricky customer service scenarios.
Tasks that require navigating ambiguity and multiple
steps in an unpredictable environment. It really
shows the gap between, say, solving an IMO problem
and, you know, booking your family vacation online.
So that generalized agent capability is still
proving pretty difficult. It seems so. But at
the same time, you see huge amounts of investment
pouring into more specific, more narrow AI applications.
Like that start of Giga, right? They just pulled
in, what, $61 million? Yeah, big funding round
led by Y Combinator and Redpoint. And what are
they focused on? Enterprise voice AI, real -time
customer support. Exactly. Grounded, specific
business problems where AI can deliver clear
value right now, even if it's not a general purpose
agent. So why the disconnect? Why do the top
models struggle with the general tasks in the
magentic marketplace? But these startups focusing
on narrow stuff like voice AI get huge funding.
I think it comes down to structure. Real world
tasks are just too messy and unpredictable for
today's general models. Investors seem to be
betting on the sure thing AI that solves a well
-defined business problem reliably rather than
the grand vision of general AI agents, which,
you know, isn't quite there yet. OK, that makes
sense. Focus on what works now. Mid -roll sponsor
Reed Placeholder provided separately. And this
whole picture, the advanced research, the security
needs, the agent struggles, the targeted investment,
it all leads back to perhaps the biggest underlying
driver revealed in these sources. That's the
economics of it all, specifically this massive
collapse in the cost of running these models.
Yeah, the token price collapse. It's not just
a small discount. It's fundamental. Maybe we
should quickly clarify what tokens are. Good
idea. Basically, tokens are the little pieces
of text or data that AI models chew on. Think
of them as the basic unit of work for an LLM.
And the cost is usually measured per million
tokens processed. Right. And that cost is just
plummeting. It's staggering. The sources show
top tier models, think GPT 4 .5 level or similar,
went from roughly $10 per million dollar tokens
back in 2022. Okay. Down to a projected one cent,
0 .01 per million tokens by the end of 2025.
Whoa, wait, $10 down to one cent? That's a 900x
drop. Per year, basically. Yeah. A 900x annual
drop for the top tier. Imagine scaling something
to a billion queries when the cost drops like
that. It changes the entire feasibility of projects.
It's not just optimizing costs. It's enabling
completely new things. Totally changes the physics
of software, like you said. And it's not just
the absolute best models, mid -tier ones, down
40x per year. Even the cheaper basic models still
drop 9x per year. It's like the bottom fell out
of the market cost -wise. Even on a log scale
graph, it looks like a cliff dive. It really
does. And this ties right in. to this economic
idea. Moore's law meets Jevons paradox. OK. Break
that down. Moore's law is about chips getting
exponentially better or cheaper. Right. And Jevons
paradox says that when something becomes way
more efficient. and therefore cheaper to use,
we don't just use less of it. We actually end
up using way more of it because new applications
become possible. So the cheaper AI gets, the
more we find ways to use it, driving demand through
the roof. Exactly. And we're seeing hard evidence
of that. Google's reporting that even their older
hardware, like seven -year -old TPUs, their AI
processing chips are running flat out. 100 %
utilization. They can't even make the hardware
fast enough to keep up with all the new ways
people are finding to use this now cheap AI compute.
Precisely. The use cases are just multiplying
like crazy. There was a great analogy in the
sources for this, comparing tokens to transistors.
Yeah, the idea is that the price drop essentially
swaps transistor for token in terms of cost.
It makes complex AI computation almost disposable.
Like those cheap sensors they put on shipping
tags now. So running a sophisticated LLM for...
Or say. personalized tutoring or instant legal
analysis could become nearly free. That's the
implication. Suddenly, almost any task that involves
information processing could become an LLM use
case because the cost barrier just vanishes.
So if nearly every task is potentially an LLM
task now, what does this near zero cost mean
for how fast innovation happens from here on
out? Well, it just removes friction, right? The
focus shifts completely away from can we afford
to do this towards what creative things can we
do? It fuels rapid experimentation, rapid proliferation.
the bottleneck becomes imagination and implementation,
not cost. Okay, let's try to pull these threads
together. We started with these really demanding
benchmarks, like IMObench, needed to actually
push AI towards genuine logical reasoning. But
then we saw the flip side. The need to manage
the huge security risks these powerful models
bring, like hack GPT, leading to new roles like
FDEs just to keep things secure. And we also
hit that reality check, the fact that general
purpose AI agents are still finding it really
tough to handle messy real world tasks like in
that magentic marketplace test. Right. And finally,
we looked at the massive economic shift driving
the whole expansion, that incredible 900x token
cost collapse, making AI computation almost free
utility. So for you listening to this, think
about that tension. You've got AI agents still
struggling with basic unstructured reality, but
the cost to run those potentially flawed agents
is plummeting towards zero. So what kinds of
imperfect but incredibly cheap and scalable AI
applications are about to just flood into everything
we use? What does it mean when AI is everywhere?
Super cheap, but maybe not always quite right.
That's the really interesting, maybe slightly
unnerving question to chew on. A lot to think
about there. Thanks for joining us for this deep
dive into the AI ecosystem. We'll catch you next
time. OUTRO Music.