Okay, so imagine this AI, right? It just aces
complex math, writes code like a total pro, but
then ask it to do some, you know, creative design
and it just completely bombs. Beat. That's kind
of the weird paradox with Grok 4. It really is.
A fascinating one. Welcome to the deep dive.
Today, we're really digging into Grok 4 XAI's
new thing. Just came out, what, July 9th, 2025.
Exactly. And yeah, Elon Musk's company, they're
saying, you know, world's best AI model. Big
claims. Right. Big claims. You hear that a lot.
Cool. We're not just going to take their word
for it, obviously. We actually got our hands
dirty, ran some real tests, tried to see what's
really under the hood. Yeah, we did. So our mission
here for you listening is to cut through all
that hype. Get to the core of it. Figure out
what you really need to know about Grok 4. We'll
look at the scale, the benchmarks, some surprises
there. Definitely some surprises. The cost, the
economics, and then how you actually use it in
a real project. Right, the practical side. That's
crucial. Plus, we've got these three key real
-world tests. They really show where it works
and, well, where it kind of falls flat. Okay,
so let's start unpacking this. What is Grok for
exactly? The headline number is just huge. Yeah.
An estimated 1 .7 trillion parameters. Yeah,
1 .7 trillion. It's hard to even wrap your head
around that number. Think of parameters as like
the AI's brain cells or maybe connection points
in its network. Okay. More parameters generally
mean more power to learn really complex, intricate
patterns in the data it's trained on. Right.
So how does that compare? Well, for perspective,
GPT -4 is estimated around $1 .8 trillion, so
kind of similar ballpark there. Google's Gemini
Ultra is about $1 trillion, and Anthropix Claude
4 is maybe around $500 billion. So Croc 4 is
definitely up there with the biggest model. So
definitely not something you're running locally.
Oh, no way. Yeah, you're definitely not running
this beast on your gaming PC. It's entirely cloud
-based, needs massive infrastructure. Makes sense.
And XAI's claims, I mean, they're bold. better
than a phd level in every subject smarter than
almost all graduate students in all disciplines
simultaneously that's straight from elon musk
wow that's quite the statement it is it sets
a high bar so boiling it down what's grok 4's
biggest raw strength just based on that scale
its immense scale allows it to learn incredibly
complex detailed patterns stuff smaller models
might just miss okay scale is one thing but performance
is another This is where it gets really interesting
for me. The actual benchmarks. How did it do?
Right, the numbers. On something called Humanities
Last Exam, or HLE, it's designed to test broad
knowledge and reasoning across different fields.
Grok 4 scored 25 % just on its own. But, and
this is key, when it could use tools like a search
engine, it jumped to 44 .4%. Ah, so it knows
how to use tools effectively. That's different
from just raw knowledge. Exactly. It shows it
can leverage external resources intelligently,
which is, you know, way more like how humans
solve problems in the real world. That makes
sense. What about more specialized areas? Yeah,
for grad level physics and astronomy, it hit
87, 88 percent. That puts it ahead of Google
Gemini and Anthropic Glod in those tests. So
a really strong grasp of complex scientific concepts.
87, 88 percent on grad level physics. That's
impressive. It really is. But honestly, the part
that truly blew me away was the A score, the
American Invitational Mathematics Examination.
Oh, yeah, that's notoriously difficult. Extremely.
Grok 4 scored 95 out of 100. Whoa, wait, 95 percent?
on a 95 that's that's incredible that's not just
calculation that's deep mathematical reasoning
step -by -step problem solving precisely it really
points to that powerful step -by -step thinking
capability especially for complex math it's a
huge deal imagine an ai solving those multi -step
math problems with 95 accuracy that feels like
a massive leap and what about for developers
coding benchmarks Ah, yes, the Software Engineering
Benchmark, SWE Bench, crucial one. This test
is fixing real bugs, adding features and existing
code bases. Right, the messy stuff. Exactly.
Grok 4 scored between 72 % and 75%. That places
it right at the absolute top for tackling these
real -world coding challenges. Okay, so putting
these numbers together, how do they translate
to real -world impact? What's the takeaway? It
means elite problem solving ability, especially
for tough scientific encoding tasks. OK, those
numbers are seriously impressive. Elite problem
solving power. But, you know, power usually comes
with a price tag. Let's talk economics. This
isn't free, right? Not at all. Grok 4 is a premium
commercial product. You access it via an API
and you pay for that access. There's no free
lunch here. So the value proposition isn't about
being the cheapest option out there. Definitely
not. It's about being the best for specific high
-value tasks. Think about it this way. Maybe
it costs, say, 12 cents for a complex query.
Okay. But if that 12 -cent query automates a
task that would take a skilled developer, I don't
know, hours to figure out, like tracking down
a really tricky bug. Right. The return on investment
could be massive. Exactly. The cost of the developer's
time, the project delays. Suddenly, 12 cents
looks incredibly cheap. You're paying for that
elite level performance, that acceleration. So
bottom line, is it really worth the cost then?
For high value, complex problems, its superior
performance absolutely justifies the cost. All
right. Makes sense. So let's get practical. How
do you actually start using Grok 4 in, say, an
automation workflow? What are the paths? There
are basically two main ways people are doing
it right now. Path one is the direct connection
to XAI. Okay, how does that work? It's pretty
straightforward on the surface. You typically
use an AI agent node in whatever automation tool
you prefer. You grab an API key from the XAI
developer console, plug in your credentials.
Standard API setup. Right. Then you select the
model, which would be Grok 4 -0709 or whatever
the latest version is. You can do a quick test,
like sending hello, Grok, just to make sure the
connection's live. And then you can give it tools,
like web search. Yep. You can add nodes for tools,
maybe a propensity node for research or Tavoli
or others. Grok 4 is designed to figure out when
it needs to use those tools to answer your prompt.
Sounds good in theory, but does it always work
smoothly? Well... No, not always. We actually
tried building something we called an ultimate
assistant. The goal was research a topic using
two different tools, find a relevant contact
person, and then draft an email to them. Pretty
complex task. Multi -step. Yeah, definitely.
And we immediately hit a snag. We got this error.
Failed to parse tool arguments from chat model.
Meaning Grok 4 wasn't sending back its instructions
for using the tools in the strict format, the
JSON format that the workflow needed. So the
workflow just broke. It didn't know what Grok
was trying to tell it to do next. Okay. That
sounds frustrating. Could you fix it? We tried.
We even tried forcing it into a JSON -only output
mode, but that didn't help either. In fact, it
just stopped trying to use the tools altogether
then. So the direct connection can be a bit finicky.
Yeah. I still wrestle with prompt drift myself
sometimes, you know, where... The AI's output
just changes over time, even with the same prompt.
A consistent API is like a godsend when you're
building something real. Totally agree. Which
leads us nicely to path number two, using OpenRouter.
OpenRouter. I've heard of that. It's like a middleman.
Exactly. It acts as an intermediary, a routing
layer. And honestly, it's often a smarter and
much more reliable way to go. Why is that? What
are the benefits? Several big ones. First. single
billing you get one bill even if you use dozens
of different models from open ai anthropic google
xai whoever that simplifies things a lot okay
that's convenient second a unified api format
open router makes all these different models
talk and basically the same way through their
api so your code or workflow setup is much more
consistent even if you swap models So less chance
of those parsing errors we just talked about.
Yeah, precisely. And third, often the connections
just seem more stable and reliable through OpenRouter.
In your workflow tool, you'd use a generic chat
model node, connect it to OpenRouter, and then
just select Zagrok 4 from the list of models
they offer. And did that work for the ultimate
assistant? Like a charm. It worked perfectly.
Grok4, through OpenRouter, laid out its plan.
Research with Perplexity, then research with
Tavli, then it synthesized the info from both
sources intelligently, looked up the contact,
and composed this really well -crafted email,
even citing its sources. Wow. So it managed all
four tools seamlessly. Seamlessly. It was actually
quite impressive to watch it orchestrate the
whole thing. So if a developer is starting out,
which setup method should they probably prioritize?
Generally, OpenRouter provides more stability,
reliability, and honestly, just ease of use,
mid -roll sponsor read. Okay, we saw OpenRouter
smooth things out. But let's talk brass tacks,
speed, and cost for that complex ultimate assistant
workflow. How did it actually perform? Right,
the performance varied. The first time we ran
it, it took about one minute and 40 seconds,
which is, you know, pretty reasonable for that
complexity. Yeah, not bad. But the second time,
it took over three minutes. Hmm. That's quite
a difference. Nearly double the time. Why the
variability? Server load, most likely. Grok 4
is new. It's popular. It's powerful. Lots of
people are hitting the API. So performance isn't
always consistent. That's a really important
point if you're building something that needs
predictable response times. Absolutely critical.
You have to build your applications assuming
that variability might happen. You need error
handling, maybe longer timeouts, or ways to manage
user expectations if a task takes longer sometimes.
It could definitely destabilize things if you
don't account for it. Okay. Good warning. And
the cost for that run. You mentioned $0 .12 earlier.
Yeah, through OpenRouter, that specific workflow
costs about $0 .12 each time it ran successfully.
$0 .12 doesn't sound like much on its own. It
doesn't. But if you're running that kind of complex
workflow hundreds or thousands of times a day,
it adds up fast. The cost is driven by the amount
of text processed, the input prompt, the tool
usage, the back and forth, the final output.
They call these tokens. More tokens, more cost.
So how do you manage that? You can't just not
use it if you need its power, but you don't want
costs spiraling. The smart strategy is often
a hybrid approach. Use cheaper, faster models
for the initial legwork. Legwork. Maybe use a
smaller cloud model like Hypu or one of the smaller
Gemini models to do initial data gathering, maybe
summarize some long documents or filter information.
They're much cheaper and faster per token. OK,
so pre -process the information. Exactly. Do
the grunt work with the cheaper models. Then
take that condensed summary, that key information,
and feed only that to Grok for the really high
level analysis, the complex reasoning, the final
synthesis, or the difficult coding task. Ah,
I see. So you're using Grok 4 strategically,
only where its unique power adds the most value,
leveraging its strengths without paying for it
on simpler tasks. Precisely. That way, you get
the best of both worlds. The power of Grok 4
where you need it, but better cost efficiency
overall. So how can developers manage these performance
and cost variations effectively? Strategically,
use cheaper models for initial steps, reserving
Grok 4 for the high -value reasoning parts. All
right, we've talked benchmarks. Set up, cost,
performance. Now for the really fun part. Putting
Grok 4 to the test in some real -world scenarios,
we set up three distinct challenges. Yep. Wanted
to see how it handled different kinds of tasks
you might actually throw at it. Test number one,
a simple bug fix. Okay. What was the bug? It
was a common front -end issue. A scrolling problem
in a React component. We gave Grok 4 the buggy
code and asked it to fix it. It nailed it. In
under two minutes, it identified the issue in
the CSS, proposed a clean, professional solution,
explained why it worked. Pass. Solid pass. Nice.
So for straightforward, well -defined, logical
problems like fixing a specific bug, it's very
effective. Incredibly effective. Really shines
there. Okay. Simple bug fix dot check. What about
something more complex? Test number two. Complex
feature development. This was ambitious. We asked
it to add a memory feature to an existing chat
application. Whoa, okay. That's not trivial.
What did that involve? It required changes across
the board, database schema updates, creating
new API endpoints, building new user interface
components, modifying the core chat logic to
actually use the memory. A lot of moving parts.
Yeah, that sounds like a multi -day task for
a human developer. Easily. How did Grok 4 do?
Honestly. My mind was truly blown. Really? The
entire feature was built, designed, coded, integrated
in under five minutes. Under five minutes. Seriously.
I didn't write a single line of code myself.
I just reviewed what it produced, ran it, and
everything worked perfectly on the first try.
Wow. Beat. Five minutes. I was trying to process
that. Thinking about the usual back and forth,
the debugging, the testing for a feature like
that. What was the most surprising part for you
just watching that unfold? It felt. Almost like
magic. But you could see the logic. It understood
the existing code -based structure, which was
well -organized. That's important. And it just
methodically generated all the necessary pieces
and connected them correctly. It wasn't just
code generation. It was architectural understanding.
It's nothing short of revolutionary for adding
complex features to existing well -structured
code bases. Well, revolutionary. That's a strong
word. But based on that. Yeah. Okay. Mind officially
blown, too. So test one. Simple bug fix. Pass.
Test two, complex feature, absolutely revolutionary
pass. What was test three? Test three, new project
creation. After that stunning success with the
feature ad, we thought, okay, let's see if it
can build something from scratch. We asked it
to create a beautiful landing page for a fictional
product. Just from a prompt, make me a beautiful
landing page. Pretty much. We gave it some basic
info about the product, but the key instruction
was make it beautiful. And the result? Yeah.
Was it beautiful? No, not really. The website
it generated was functional. The HTML structure
was okay. The basic elements were there. But
visually, it was really disappointing. How so?
The styling was just... Very basic. Kind of bland,
even outdated looking. Nothing like what you'd
consider a modern, polished, beautiful landing
page. It completely missed the mark on the aesthetics.
Ah, back to that paradox we started with. Great
at logic and code structure, but struggles with
the subjective, creative side. Exactly that.
It failed here, we think, because it lacked specific
design context or examples in the prompt. And
beautiful is just too subjective for it. It doesn't
have an inherent sense of visual taste or modern
design trends. Right. It needs clear objective
parameters or examples for creative tasks. It
can't just intuit good design. Precisely. It
struggles immensely with tasks requiring that
strong sense of visual design or, you know, open
-ended creativity, unless you guide it very,
very explicitly. So looking back at these three
tests, what's the biggest takeaway? Grok4 excels
at structured, logical tasks, even very complex
ones, but struggles with open -ended creativity.
Okay, we've run the tests, looked at the numbers,
the setup, the cost. Let's try to bring it all
together. Is Grok4 worth it? Let's recap the
good and the bad. The good. Exceptional reasoning
ability, definitely. Powerful math capabilities,
as we saw with that aim score. Excellent tool
use, especially through something like OpenRouter.
And it can produce really high quality research
and analysis when guided properly. Okay. Sounds
powerful. Right. What about the downsides? The
not so good? Speed inconsistency is a big one
due to that server load issue. You have to plan
for it. Right. Higher cost, especially if you're
using it for high volume tasks without optimization.
That direct XAI integration can be finicky as
we found. Yeah. The JSON parsing issue. And that
clear struggle with creative. subjective tasks
like visual design it's just not its strong suit
so the bottom line what's the final verdict look
grok 4 is undeniably impressive it's incredibly
powerful it excels at complex reasoning math
coding within existing structures deep analysis
it's not always the fastest it's not the cheapest
but when you need that serious intellectual horsepower
for your ai automations especially if you're
a developer working on well -structured code
or you need deep analysis It's an absolute game
changer. It can do things other models just can't
or can't do as well. So what defines Grok 4's
ideal use case then? It's for serious intellectual
horsepower on complex logical problems. So we've
really dug deep here, uncovered Grok 4's incredible
strengths that logic, the math, the tool use,
but also seen its limits clearly. The speed issues,
the cost factor that struggle with pure creativity.
It's a powerful, almost paradoxical beast, isn't
it? It really is. And this whole deep dive looking
at Grok 4, it points to something bigger, I think.
A profound shift in how we might approach work,
especially complex knowledge work. How so? The
future you see feels increasingly agentic. It's
becoming less about just how fast you as an individual
can code or research or write. And more about.
More about how effectively you can direct and
orchestrate these powerful AI assistants, maybe
even a team of them, to achieve complex strategic
goals. It's like being a conductor rather than
just playing one instrument. That's a great analogy.
And you're saying this is becoming more accessible?
Yeah. The tools are getting easier to use, as
we saw with Open Router smoothing things out.
The productivity gains, like that five -minute
feature build, they're potentially massive and
very real. And the price, if used strategically,
can absolutely be justified by the value. So
wrapping this up, what does this all mean for
you, the listener, the learner, the developer,
maybe just the curious mind tuning in? It feels
like the landscape is shifting under our feet.
It really does. It suggests that... Understanding
these tools, learning how to integrate them strategically.
It's not just about adding a new skill. It could
fundamentally change your capabilities, maybe
10x them, like people often say. The choice seems
to be leaning towards embracing these tools,
figuring out how to leverage them, or risk getting
left behind as others do. That might be the stark
reality, yeah. Which leads to a final, maybe
provocative thought to leave people with. Go
on. How might you listening right now? How might
you start to rethink your own role, your own
workflow, in a world where AI agents like Grok4
can increasingly accomplish very complex tasks
autonomously? What does that mean for your unique
human contribution? That's a deep question. Something
to definitely mull over. We hope this deep dive
into Grok4 has given you a solid, well -informed
starting point for thinking about that. Thanks
for joining us. OTRO Music.