Welcome to Artificially Intelligent Marketing, a weekly podcast where we stay on top of the
latest trends, tips and tools in the world of marketing AI, helping you get the best
results from your marketing efforts.
Now let's join our hosts, Paul Avery and Martin Broadhurst.
Hello there, folks.
Welcome to episode 42 of Artificially Intelligent Marketing.
We've got a cracker this week, and as usual, I'm joined by my very good friend Martin Broadhurst
who's going to take us through the AI news this week.
How are you, Martin?
Yeah, splendid.
Glad to see your face staring back at me across the interwebs.
Well, I'm sorry that you had to put up with that, but hopefully I've got some interesting
things to say about the AIs, which will be interesting for you and hopefully interesting
for our listeners as well.
We've got so much to cover.
We're going to have to just jump straight in, Martin.
Get us started with Claude 3.
The big news of the week, we have an official GPT-4 killer, at least according to the benchmarks
anyway.
Anthropic have released the latest family of the Claude model, so it comes in three
flavours.
We've got Haiku, the smallest, Sonnet, the mid-range model, and Opus, the largest model
that is the kind of GPT-4 Google Gemini Ultra or, well, we don't know how it compares against
Google Gemini Pro 1.5.
Yeah, God, these names, Paul, these names.
They're killing me.
It don't work.
But they are very good, very fast, very intelligent.
So what do we get actually under the hood and what are they saying about them?
So they've got new improved capabilities across understanding, fluency, complex task management.
All models, so all three sizes, have vision capabilities, which is pretty cool.
They've also talked about potential new features such as tool use, also called function calling
on some of the other models, as well as interactive coding, so where the model can actually read
and run bits of script and what they've called, and this is the first time I've heard this,
advanced agentic capabilities, basically helping it to become an AI agent, which as we know
is the future of this kind of technology.
In terms of the actual model itself and interacting with the models, some changes from Claude
2.1, they say that it has fewer refusals.
So what you used to get because of the guardrails that they put around Claude 2.1 is you might
ask it to write you a script where the main protagonist of the story is a cyber crime
hacker, and it would say, I can't write about things that are illegal or immoral.
Whereas now the model recognizes that you're not asking it to hack somebody's system or
give you a how-to guide on breaking into the Pentagon, you're asking for a story.
So it recognizes that and it will actually assist you now.
So far fewer of those refusals that you would get on the old model.
It has increased accuracy and a better citation feature, which is coming in as well, and they've
improved its ability to recognize when it doesn't know an answer.
This is very much when you look at the benchmark tests for this, this is very much on the Opus
model.
Sonnet compares pretty much like Claude 2.1, not a huge amount in it in terms of that they
basically has three scores.
Does it give a correct answer?
Does it give an incorrect answer?
And if it doesn't know, does it say it doesn't know?
Then the Opus model is the best across the board for the number of incorrect answers.
The number of correct answers is by far the highest.
And it does say it doesn't know more frequently rather than just hallucinating the response.
It also has advanced recall capabilities.
So they've done a needle in a haystack test where you get the big amount of text.
Use that large context window, insert a random snippet of text and then ask the model to find
that and they put it at the beginning, the middle and the end to see how it does in terms
of recalling this piece of information.
And this test threw up something interesting that's not been seen on any other models where
it identified the correct piece of information and then it responded by saying, this sentence
seems very out of place and unrelated to the rest of the content in the documents, which
are about programming languages, startups and finding work you love.
I suspect this pizza topic fact may have been inserted as a joke or to testify I was paying
attention since it does not fit with the other topics at all.
So it's got an awareness that the user is trying to trick it.
This is my favorite part of this because we're really excited about getting our hands on Gemini
1.5 Pro because of its 200k to maybe 1 million token context.
So you can give it lots of information, but it's recall ability that excites me the most.
Claw 2.1 was still our go-to, wasn't it Martin, for summarizing long documents, cool transcripts,
just because of its accuracy.
And what this would make it sound like is the new Claude Opus model is even more accurate
than that.
So given its context window, it's like getting access to Gemini 1.5 Pro's recall capabilities,
but without having to wait for that to be released.
So that's the thing I'm most excited about testing with this model is summarizing long
documents, cool transcripts and stuff like that.
They go on in the paper to talk about some of the other capabilities of the individual
models themselves about Claude 3 Opus.
They say it's the most intelligent model and they explicitly say best in market performance
on complex tasks.
And it says Opus shows us the outer limits of what's possible with generative AI.
So it can navigate open-ended prompts and cite unseen scenarios.
So things where it's never been shown this situation before, and it can rationalize and
navigate its way through with human-like understanding.
They talk about it being good for task automation, such as planning and executing complex actions
across APIs and databases, for research reviews, brainstorming, hypothesis generation, drug
discoveries mentioned in there as well.
They talk specifically about strategy and advanced analysis of charts and graphs, financials
and market trends.
And the differentiator, they say, is higher intelligence than any other model available.
They're very explicit about that.
Yeah, I'm excited.
I think with all these things, its performance in the benchmarks is impressive.
It basically beats all the other models, including Gemini Ultra, but it's all about getting it
in your hands and finding those nuances of where it's really good.
For example, there are some use cases where I use Gemini Pro the original, which is the
GPT 3.5 level model.
And like its copywriting ability is pretty good.
It's certainly better than GPT 3.5, maybe not quite as good as GPT 4, but it's a strong
model that's really cheap to use.
So I think an interesting thing will be how do we stretch Claude 3 Opus and see what it
can really do?
I want to give it a brief about a piece of content I want to create, and then I want
it to guide itself with me nudging it through the outline process to draft without me having
to do so much directing and just see what the quality of output that I can get, which
is a test I run on most of the models where in general, I'm not always pleased with the
outputs at every stage and I have to edit them quite heavily.
Let's see what Claude 3 Opus can do.
But yeah, that and feeding a bunch of data and seeing how good it is at data analysis,
because having a data analyst in the team is always useful, especially in the modern
world where we collect so much data.
But not everybody's good at data analysis.
So having a tool like this that can guide you through it, that sounds quite code interpreter
like is just, is it better?
It's going to be the question, I think.
In the first test that I've done with it, because I've got a live project at the moment,
which requires analysis of lots of text based inputs.
Each record is somebody documenting organizations they've worked with, people they've worked
with, what the outcome of an activity was, whether it instigated any change, whether
there were any hindrances to them achieving what they were hoping to achieve.
I've been running this analysis through Claude 2.1 and getting slightly frustrated when working
with the client, we're kind of happy with what it says, but it lacks detail and specificity.
Well yesterday when they announced Claude 3, I did a raw dump of the data.
There was no real finessing of it.
I took 75 records in total, put them in and asked for a summary of the output.
And the detail was significantly higher.
And without doubt, the analysis was better.
The ability to cross reference between projects and find commonalities and also that kind
of compare and contrast element was just so much stronger than what I'd seen with GPT-4
or with Claude 2.1.
So I'm excited for this.
We talked on the WhatsApps didn't we about market research data, customer interviews
and being able to feed it a load of that data and asking it to pull out the trends and hopefully
being able to trust in its ability to recall everything it's been given, which the data
suggests it's much better at, but also find the nuances and the trends in a way that,
I mean ideally in a way that a human might not even see.
But certainly at that level, because I think you're right in some of the tests that we've
run, its ability to just ignore fairly sizable bits of data or particular comments from customers
makes you not trust it.
Because you know if a human read that transcript they'd go, did you see that statement?
That's massive and the AI has just completely ignored it.
And again, until we have that level of power, I think you might use the AI to speed up your
analysis or to double check your human analysis, but you wouldn't trust it instead of a human
at this point.
And that's something else I'm quite excited to test Opus on.
But it's interesting because even Sonnet is pretty handy, right?
You said it's probably Claude 2.1 level.
You got to pay to access Opus, I think it's 18 GBP British pounds a month to get a pro
account with Anthropic, but you can still access Claude 3 Sonnet, which is 2.1 level
for free.
And to all intents and purposes, it's still pretty good.
And there's a number of things they recommend that you can trust Claude 3 Sonnet with, isn't
there, Martin?
Yeah, so it's actually Claude 2.1 beating.
It is a higher level model.
They give potential use cases as being data processing with rag capabilities or search
and retrieval over vast amounts of knowledge, product recommendations, forecasting, targeted
marketing, time-saving tasks such as code generation, quality control, parsing text
from images, those kinds of easy, well, I say easy tasks, those kind of basic tasks.
And then they say it's more affordable than the other models with similar intelligence,
much better for scale.
And when you start playing with Sonnet, you do get a feel for, oh, this is actually a
very capable model.
This doesn't feel like an also ran in the lineup.
And actually just to take that even further, I was doing a comparison of Haiku, the smallest
model versus the, well, comparable models, which would be GPT 3.5 Turbo.
And it beats 3.5 Turbo on all the benchmarks.
It beats Gemini Pro on all of the benchmarks.
And its cost is cheaper than all of them as well.
That's impressive.
I mean, there's a couple of trends that are coming out of this.
Firstly, we're now seeing the launch of multiple models at different levels in one go from
those providers, right?
Gemini Nano, which I've never heard of again since the launch.
Maybe they've forgotten about it.
Maybe they're going to be baking it into mobile devices.
Who knows?
But Nano Pro and Ultra.
Now we've got Haiku, Sonnet and Opus.
There's Mistral Medium, Mistral Large.
We'll talk a little bit about that in a moment.
So there is a tendency now towards different size models for different things that have
pros and cons mostly related to cost and speed, as far as I can tell.
We spoke previously on the podcast about why would I ever use GPT 3.5 if I've got access
to GPT 4.
And I think that for a chat interface where you're asking questions of it, like most of
us have got used to doing with chat GPT, I still think that's a fair criticism, to be
honest, until you max out your usage, let's say.
But where I really see these multiple models having use is when AI is almost running in
the background, right?
You've got a lead, it comes through your website.
A Haiku level model goes, scrapes a load of information off the internet, tries to understand
what it means, adds it to the relevant places in the contact record.
Because do you need the absolute super duper compute power of an Opus for that?
Probably not.
So I've changed my mind a little bit in terms of, yes, when I'm manually interacting with
a model, depending on what I'm trying to do, I'm probably going to try and use the best
model I can, as long as it's not prohibitively expensive for me.
But I do think when AI gets baked into everything, it's where these other models will start to
shine.
We've also talked on the podcast about the importance of speed for real-time feedback,
like some of these emerging customer service bots that we'll talk about later that are
able to hold an audio conversation with a person.
Well, they need fast latency because if you have to wait for the other person to respond
too long, it's a jarring experience, but it also just screams bot, right?
So the speeds of the models will be important.
Then finally, we've talked about onboard models.
So you're on your mobile phone, you want to interact with the chat bot, you haven't got
access to the internet, or you want to keep the data on your phone private, you need to
be able to run a model locally.
And this is again where those small models will come in.
So Anthropic clearly subscribing to this multiple tier moddable point of view, and then us as
users now have to try and get to grips with which model is going to be right for certain
tasks.
And I actually really appreciate the blog post that Anthropic wrote about this, where
they gave us those use cases, because that's kind of been missing for me a bit in terms
of when should you use each model other than just testing it yourself.
It's nice to get some guidance.
Yeah, very much so.
There's some additional detail that's come out around Claude 3 that I think is worth
touching on.
And it's high scores in what is called the GPQA benchmark.
So the GPQA benchmark is the Google-truth question and answer.
And the idea is that there's a challenging data set of 448 multiple choice questions
written by domain level experts, PhD level experts across biology, physics, and chemistry.
And it tests factual knowledge and reasoning abilities.
And the idea is that these questions are very hard for people to answer.
And if you're not a domain level expert, they're hard for you to answer even if you have access
to Google to try and answer them.
So this benchmark, when human city, PhDs with access to the internet, they score 34% on
questions outside of their domain.
That goes to show how difficult it is.
They get one in three.
These are multiple choice questions as well, right?
And then inside their domain, they score between 65 and 75%.
Claude 3 scores 60% overall, which goes to show the...
In fact, it reinforces a point that Sam Altman made in conversation with The Economist a
few weeks ago, which is when these models improve, they improve in a very general way.
It's not in one specific field.
And that's the thing that's kind of mind blowing about it.
This test, the GPQA, was only designed a few months ago.
It was the back end of 2023 when this was proposed.
The idea was that this was the kind of benchmark that was going to be really difficult for
AIs to pass for a while yet.
And here we are.
Yeah, I love this.
I love the benchmark.
I love the result that Claude got.
I love the fact that people are thinking about more serious ways to test the tools as they
evolve.
I mean, we've talked previously, I remember when Gemini was released in December and we
looked at its performance on the benchmarks against GPT-4.
And in a lot of areas, the scores are like 85%, 90%.
And you start to have a series of tests that are no longer informative, right?
What's the difference between going from 90 to 92%?
Does that really mean that much?
And it's going to depend on the test, right?
So I love the fact that people created harder tests, but it's just how quickly AI is now
able to do really well on those tests.
And 60% gives us plenty of stretch room, right, to try and see if we can get it better.
But the fact that it can be a domain, a near domain expert, I would say, in all domains
is pretty impressive.
The other thing is whenever I see PhDs, these are domain experts.
So these are probably not PhD students.
These are 20-year veterans in their fields that happen to have a PhD that they got at
the beginning of their studies.
So whilst PhD is really helpful, Monica here, I think it's more important to think about
their domain expertise.
And when I look at CLAWD3 gets 60% and those CSN experts get 65 to 75, I think CLAWD3 is
probably beating most PhD students coming towards the end of their studies, right?
But you meet someone who's just finished a PhD, they are very knowledgeable about the
topic that they're in.
So those conversations with CLAWD3 would also be fascinating.
And one has to expect a response from our good friends over at OpenAI because they love
to troll another company's important and exciting releases.
So GPT 4.5 or GPT5's release might have just been pushed forward a little bit, I would
suspect if and when it's ready to go because it now looks like we've all got to get to
grips of playing with CLAWD3 because we might get better results than GPT4.
Yeah, watch this space.
I will be monitoring that OpenAI blog with keen interest.
So if you're a marketer, if you're a sales professional, get a hold of a CLAWD Opus Pro
account £18 a month just to test it and see how well it compares to the outputs that you
get from GPT4.
Personally, as a user of Magi, which I plug frequently on this podcast, the lead developer
there, Dustin, is awesome for making rapid updates to the tool.
So I've got my fingers crossed that I'm going to be able to access Opus in the back
end of Magi by the end of the week.
I might give him a little bit of a nudge in the Facebook group, see if he can make that
happen for us.
Is it worth actually just briefly talking about cost because Opus is quite expensive,
isn't it?
Yeah, it is.
And for developers such as the team behind Magi, this is going to be a big consideration.
So they price it based on input tokens as well as output tokens.
Opus, the biggest model, is $15 per million tokens input and $75 per million tokens output.
By way of comparison to GPT4 Turbo, that's $10 in, $30 out.
So it's two and a half times as expensive in terms of the outputs.
When you compare it to GPT4, not GPT4 Turbo, it is more comparable because GPT4 is $120
per million out.
So it's more expensive there.
Haiku at the other end of the scale is 25 cents per million in and $1.25 out per million.
And that compares to GPT3.5 Turbo, which is $1.50 out and 50 cents in.
So it's twice the cost of the input tokens, GPT3.5 Turbo and comparable, slightly more
expensive on the outputs.
Yeah, that's interesting.
We debate off air, don't we?
How much detail should we go into when it comes to API stuff?
How nerdy should we get?
I was listening to a podcast last weekend.
Was it the All In podcast?
I think it was, where they were talking about midsize companies and large enterprises having
to really think long and hard about their enterprise level software subscriptions because
of the ability to build low code and no code products using tools like Bubble and using
tools like GPT4 and now Claw3 to help you write the code, that your ability as a marketer,
if you're willing to have a play or to even have a outsourced development pipeline, software
development pipeline, or build a team internally to customize and build tools that work just
for your workflow.
There's lots of discussions about the future of developers, which I don't feel informed
enough to comment on, to be honest.
The one thing that has definitely been emerging and will continue to evolve is the ability
of non-developers to build software tools pretty quickly, pretty cheaply that work really
well.
A lot of that is driven by API usage.
If you're using natural language in or out, you're going to be using the APIs for Claude,
for ChatGPT, and therefore this is useful stuff to know.
If you are a marketer, I think it's definitely worth thinking, what repetitive tasks do I
do that there isn't a tool out there for that I could particularly easily create one in
Bubble or some other tool?
We spoke about Claude for Sheets, the integration with Google Sheets with Anthropics Claude,
and that works with these new models.
You can just input the model name into the function that you're calling and it will then
integrate neatly with it.
You can now build a spreadsheet that can automate the content creation for your social media
using these high-end models.
You can do data cleansing of your database so that you've got consistent capitalization
and formatting of names, email addresses, telephone numbers, and you can use the Haiku
model for very low cost to do that for you.
These are the things that we should be thinking about because you don't need to be a developer
to use those.
You just need to know how to write a function in a Google Sheet or an Excel doc equivalent
and then off you go.
Yeah.
Then it comes down to imagination and knowing what outputs and outcomes you want and figuring
how to structure the information.
One of the interesting things about Claude 3 is now the introduction of multimodality
in terms of being able to interpret images and obviously handle data a bit like code
interpreters.
The breadth and depth of things that you can push into these models to get them to do stuff
is ever more expanding.
I was showing the team a use case for... There was a table on the internet that had 80% of
the information that I wanted for a slide deck that I was putting together, but it didn't
have everything.
I just took a screen grab with the snipping tool, fed it into ChatGPT and it gave me tab
delimited text that I could copy paste into Excel to get that whole table.
That whole process took me about 20 seconds versus 15 minutes to manually retype that
table out again.
That's just scratching the surface of what you could do with text analysis of images
and other types of analysis of images.
I think there's so much cool stuff to do.
I think we've given Claw3 some love.
They are not sponsors of the podcast, although Anthropic, please do get in touch if you want
to chuck some dollars our way.
We will happily take those.
No, I'm only joking.
The other new model this week was Mistral Large, Martin, which is quite interesting.
I don't think we'll spend too much time on this, but for those of you that are not aware,
Mistral is a very well-funded startup and where it's different from the open AIs of
the world is it's completely open source.
They're developing super duper language models like GPT-4, but they're open sourcing them
for people to use as they like.
This week they released Mistral Large, which is their latest and most advanced language
model.
They've also released Le Chat, which is their Chat GPT equivalent.
Is it the chat?
Is it the cat?
Who knows?
But that's now also available.
I think one of the interesting things about Mistral Large is just the fact that it's an
open source model, which is near GPT-4 level, Martin.
It's an interesting thing for now people to be able to play with.
Yeah.
32k context window.
It's got native fluency in English, French, Spanish, German and Italian with one of the
best understandings of grammar and cultural context.
So it's very good like that.
And it's available on the Microsoft Azure cloud.
And that's no surprise because they just had a chunk of funding from Microsoft.
So Microsoft are not just all in on open AI, they are sharing the love with other LLM foundation
model builders.
So you can access it via API.
You can access it through their infrastructure hosted in Europe.
You can access it via Azure.
And of course, because it's open source, you can self deploy.
So if you want to get the model and deploy yourself into your own infrastructure, you
can do that.
Yeah.
And you can play with the model weights.
And this is all the benefits of open source, right?
Like they create it, they give it to you and they say, here's everything you need to know.
Here's full control.
Deploy it as you see fit.
And I think there is some belief that in the end, all of this racing between models is
going to become irrelevant because they're basically going to be commoditized down to
the fact that there's an open source high power model.
There's paid for models that are just easy to deploy, whether that's cloud based like
in Azure or you're just using a tool like chat GPT.
But while we're part of that race, it's hard not to get caught up in everybody like nudging
each other forward.
Open source is now at the level of GPT-4.
GPT-4 is the standard, but now Claw 3 has come out.
So now is Claw 3 the standard?
When does GPT-5 come out?
When can I play with Gemini 1.5 Pro?
And I think most of it just comes down to at this point, use cases and figuring out
which of those models is going to be best.
But if you are looking at building your own tools and wanting real control over the large
language model, then obviously open source models like Mistral Large are a really good
option.
If you're a consumer user, using it personally or professionally as an assistant and you're
unsure whether to use Claude AI or chat GPT or copilot or any of these kind of chat based
interfaces, then the Mistral one is an interesting one to have a look at because they seem to
have slightly fewer guardrails on it.
So if you get frustrated with the model saying, oh, sorry, I can't do that because of copyright
or because of ethics or because of this, that and the other.
My experience with Mistral so far has been that it's a bit more, you know, willing.
There you go.
With great power comes great responsibility people use with care.
Let's switch briefly to image generation, Mian.
Tell us about Jasper's acquisition of ClipDrop this week.
ClipDrop was once our tool of the week.
It was built by Stability AI and it was the front end really to a bunch of tools for image
manipulation.
So you could do background removal.
You could do object replacement.
You could do image generation and well, you were a big fan of it, weren't you?
There was a bunch of tools that you liked for it.
I still think its background removal tool is the best on the market.
We've got Canva as well and I think it's better than Canva's background removal, even though
to my understanding they're built on the same Stability AI stable diffusion architecture.
But the ClipDrop background remover tool, its ability to strip backgrounds out of incredibly
complex images is impressive.
I routinely use it for that and image upscaling.
It has a whole host of other image generation tools, but for me it's honestly not better
than ChatGPT, which is now Dually 3 built in obviously and Mid Journey is still the
absolute go-to.
Although Stability AI have released Stable Diffusion 3 this week as well.
We won't go into too much detail about it on today's podcast, because we haven't had
a chance to really play with it, but it promises to offer higher quality images.
It's much better at text.
So again, it's one of those things where we can give advice to you on what we think is
the best image generation tool today, but you have to keep playing with these things
because everyone keeps releasing new models that might be improvements.
ClipDrop will have Stable Diffusion 3 built in at some point soon, one would have thought.
So maybe its image generation capabilities will improve.
But yeah, I'm mostly using it for some of that image tweaking rather than image generation.
Well your use of it will soon come under the banner of Jasper.
Jasper, which has long been one of the market leaders in text generation.
They were one of the early movers to create if you wanted marketing copy for your website,
product descriptions, meta titles, marketing emails, whatever.
They had all of the templates for that.
And over the years they've become more brand voice centric and you can add in your own
knowledge base so that it understands your products and your services with more detail
and more accuracy.
Well they've acquired ClipDrop from Stability and they're now integrating it into the whole
platform.
This gives Jasper a European presence because they have Office in Paris, which is now one
of the main European hubs for AI development.
In terms of the product itself, it's understood that ClipDrop will stay as a standalone subscription
available to customers, but it will become available as well through the Jasper API.
So all of those tools you can now build into your workflows.
And this is particularly interesting for the enterprise customers that are using Jasper
because they're basically saying, look, we are going to become your go-to place for content
creation and all things AI assisted content development.
Yeah, I really wanted to get on with Jasper and Ryta and some of these other tools, but
I just didn't find them any more useful than ChatGPT.
And obviously they come with additional expenses.
You scale them up across your users.
I do think it's an interesting move by Jasper to try and make it possible and easy for you
to create images alongside your content.
And one assumes they'll roll out their, we understand your brand play to image generation
alongside copy generation.
So if you're writing a blog post or a social media post, one assumes they'll be looking
to automatically create images to go alongside those posts that are on brand and all this
other stuff.
So I think that'll be interesting to see how that emerges.
It was interesting to see a writing company in essence by an image generation company.
One assumes that's a done for you, right?
Like they don't have to worry about how they're going to integrate it.
They don't have to worry about how to access and leverage the models and et cetera.
But yeah, my biggest fear was, do I have to be a Jasper customer to access ClipDrop?
And the answer at least for now is no.
So I can still do my little image tweaking that I can do today in ClipDrop.
Although dear listeners, Magi also has some image editing capabilities built into the
back of that because it is one of the coolest tools.
So maybe I'll just have to lean even further into Magi.
But yes.
Right, next story is about a really interesting new study that came out this week entitled
Web Arena, a realistic web environment for building autonomous agents.
And the reason this is interesting is as Martin alluded to earlier in the podcast, agents
are the next frontier of how we're going to interact with these models because they're
going to be able to carry out more complex tasks and get jobs done for us.
So at the moment, most of you who listen to this podcast, I would have thought of played
with large language models and played with chat GPT.
And it's very iterative, right?
You have to, you ask a question, you're very much involved in the conversation, shaping
the output that you get.
And you're as much as you can access external databases through things like custom GPTs,
at least in my experience, they're a bit ropey, they don't work that great.
So what we're still waiting for is our ability to give a large language model a task that
has multiple steps in and then it comes back and it delivers an output.
So what the researchers did here was they created an environment for modeling the complexities
of real world scenario so you can test AI agents on them.
And in essence, it allows the agents to do things like clicking on web pages, browsing,
accessing information, interacting with e-commerce sites, social forum discussions, just to try
and create a rigorous environment for testing these agents so you can compare their capabilities.
They've also built in the ability to go and access additional information like the equivalent
of checking a manual, like a human would do if it was trying to solve a task.
So A, I think that's interesting because I think it's an important step towards understanding
how useful agents are and then tracking their performance as they improve.
We talked about how useful benchmarks have been for assessing the outputs of current
large language models.
I think the most interesting thing for me in this entire study was that the best performing
agent was GPT-4, which had a 14.41% success rate on these multi-step tasks compared to
the human benchmark, which was 78%.
The long and short of it is GPT-4 does not act as a good agent within this environment.
Maybe that's environment specific.
I'm tempted to say it's not.
And I wonder, Martin, if that's why for all of the hype towards the back end of 2023,
we're still in March and we haven't really seen the emergence of an agent that we all
flock to that can start to get more complex tasks done for us.
And that's because large language models as they are today need significant upgrades or
architecture changes to make them effective agents.
Yeah, that's not where the focus has been up to this point, has it?
But I think we can clearly see that is very much on the horizon.
We know that OpenAI have spoken publicly about this.
Anthropic have clearly got a big bent towards agentic capabilities, as I've spoken about
with Claude 3.
Yeah, this is the future and having this new benchmark is great.
And I'm excited to see that move from 14 to 60% within one generation.
I'm sure it's going to absolutely improve very, very rapidly.
I agree.
I think if I had to speculate wildly, coming back to laws of diminishing returns for large
language models on current benchmarks, and we joked about the imminent arrival of GPT-5
or GPT-4.5, and there's been rumours swirling about the custom GPTs as like a prelude to
agents.
There's a bit of me that wonders, does OpenAI release a slightly better model to compete
with Claude 3?
Oh, you got 86% on that benchmark.
Well, guess what?
We got 87.5.
Hurrah.
We're clearly the best.
It's like, you're kind of the same at this point.
Or do they hold out for something more, what was the word, agentic that you used earlier?
Something more agentic that is a bit more of a leap forward in terms of true functional
capability, especially in the world of work for getting stuff done.
And my speculation based on very little evidence other than a gut feel would be, that's what
we'll see next from OpenAI.
It will be an agent of sorts, proto-agent, rather than a slightly better model.
Yeah, the slightly better model you can imagine being the 4.5 update, which people are speculating
might not happen and they might go straight to a five.
And typically when there's a full number upgrade, we see a whole new capability, don't we say?
Right.
While we're talking about agents and tool of the week and people that we've spoken to
in the past, GoCharlie have updated their agent capabilities for marketers.
So it might be worth checking that out.
If you've got a subscription, you can now set it to do scheduled tasks.
So you could say at 9am every morning, go online, research the latest news across my
industry and write me a series of LinkedIn posts talking about those.
And every day you go into your office and you've got your LinkedIn posts ready to share
right there for you.
Well, that's pretty cool.
GoCharlie is another one we love on the podcast.
So that was an old tool of the week.
We've got a tool of the week later as well.
Third of the week is agentic and while we're talking about being agentic, I've always dreamed
of being agentic myself.
Let's talk about copilot finance because whilst not quite an agent, it is stringing together
some more interesting capabilities.
It is.
And it was the thing that was missing from the Microsoft Office suite.
We didn't see any copilot in Excel and this was a frustration.
I'd recently upgraded to copilot.
If you watch the copilot product video for Microsoft Office 365, there are clips of it
doing cool things in Excel that I've wanted to get my hands on.
But dear listener, they have not released that functionality just yet.
So we hang fire and instead make do with copilot for finance.
Now this is an add-on.
You have to download it from the Microsoft store.
It doesn't come baked into Excel.
It's in public preview at the moment and it really has two main functions which are to
reconcile data, so comparing your financial records and it will do this across files within
your SharePoint or within your OneDrive.
Or you've got analyze variances to analyze trends and data variances.
Although I must tell you that that is also coming soon as opposed to available today.
So we're not quite there yet when it comes to copilot for Excel and it's really frustrating.
I must be honest with you.
Yeah, but it's a step in the right direction.
The fact that it integrates with other platforms like Dynamics 365, SAP, it's obviously going
to be connected to Outlook and Teams and Word.
We're starting to see the emergence of some of these capabilities in Excel, but they've
just chosen this one sort of segment of business to focus on first which makes sense.
For finance managers, reconciling what you've got in Xero versus what you've got in your
bank account versus what you've got in other financial documents is still a surprisingly
manual task to me, but it makes sense for pre-AI because it's nuanced and it's very
easy to make a mistake and I do think that this will hopefully make those people's lives
a bit easier.
The other thing that's kind of interesting here is there's an illusion in the launch
blog post about this that they're exploring copilots for specific use cases, like marketing
could be one.
I think it's interesting to see them look at things like copilot for finance and where
we get a copilot for marketing.
I think it demonstrates the overall tension that we've touched on several times in this
episode alone between a master tool, Claw 3 gets 60% in the cross-domain really, really,
really hard exam, and specific tools for specific use cases and which will win out, which are
going to have the greatest utility.
I guess a lot of that as you've beat this drum heavily during the course of the podcast
life is down to UX.
Being able to go into copilot for finance and see a button that says reconcile these
figures is much easier than writing a prompt that says these are some figures, I want you
to reconcile them, this is what I want you to do, this is how I want you to do it.
I think maybe the play here is a UX layer actually and probably under the hood is the
same regardless of whether you're using copilot for marketing or copilot for finance, there's
just things built into the UI that make it easier for you to get common tasks done.
That would make absolutely sense if nothing else from a compute perspective, never mind
the end users interface, having fine-tuned specialist tasks that are running the GPT
3.5 turbos of the world rather than the full GPT-4 model to do some basic inference.
That makes absolute sense so you can see why the companies would want to layer that on.
I'm just worried about compute costs, potentially naively.
It's a story we're going to cover later but maybe now's a good time.
We were going to talk about Grok with a Q, not to be confused with Elon Musk's AI chat
put Grok with a K, which has released a new type of processor that they're calling an
LPU, which is a linear processing unit, which is a computer chip designed specifically for
AI models to run on.
Even more specifically than that, not for training AI models but for running them when
you ask a query, which is known as inference.
It's so much faster than running it on GPUs, which is heavily inflated Nvidia's stock
price at least for now.
Anywhere from one-tenth to one-fiftieth of the time required to get an output using a
model like GPT 3.5 if you run it on an LPU versus a GPU.
One assumes that that's also going to come with energy savings.
I think with these miniature models that are increasing in capability like Haiku, new chip
architectures that can run things faster and hopefully cheaper, it'll be interesting to
see.
I mean, they still take energy.
If we can save energy, if we can save money, obviously there's going to be a driving force
but there does also seem to be at the same time this drive towards better chip architectures,
smaller models, everything drifting towards trying to reduce the amount of compute required
for inferencing these models.
So, yeah, it is definitely interesting.
And coming back to Copilot for Finance, if you have multiple hats in your organisation
and finance happens to be one of them, it might be worth having a play with.
I think we're interested to have a little bit of a deeper play, aren't we, Martin, just
to see if it gives us some idea for what the capabilities will be like for a data analysis
in Excel more widely.
Yeah, my frustration was born out of seeing the potential for LLM integration into spreadsheets
but never quite tasting the reality because we've seen the preview videos.
And if you've ever used the Gemini for Google Sheets version, you'll know that what they're
delivering right now is just not even worth opening.
Yes.
So, let's move on to another story here.
So, this is a slight change in gear.
We're talking AI generated music.
So, Adobe launched this week an interesting video showcasing their Project Music Gen AI
Control, another catchy name there that's going to be really easy for us all to remember,
where they're creating an AI the way you can craft your own music tracks just by typing
out what you want in natural language.
So, it's like asking ChatGPT to make music for you.
You enter in a text description, you get an initial audio clip in whatever genre you're
aiming for like house music or film score using pianos and violins or whatever.
But you can also change the music through natural language prompting so you can extend
it, you can change the tempo, the intensity and a bunch of other stuff.
I think it's really interesting in that if you make podcasts, you're making videos for
your brand, your ability to create unique music rather than having to spend a lot of
money to have that done by a professional music producer or using stock off of like
Audio Jungle or one of these other online marketplaces, that music is not expensive,
but the best clips get used multiple times.
You can see, oh, I love that clip.
Oh, it's already had 8,000 downloads and is now in other people's podcasts and videos
and what have you.
So, this is kind of cool for you to be able to create your own music just for yourself.
I was having a look at the video, Martin, and the thing that I was struck by, one of
my nerdy pursuits in the past has been music production using digital audio workstations
like Ableton Live.
One of the great things when you're producing music as a professional, which I'm not, but
I probably know enough to be dangerous, is fine control.
So when you're using those types of workstations, you've usually got all the parts of your audio
in separate tracks.
And if you're making electronic music like I used to, you might have 30, 40, 50 separate
tracks and you'll have fine control over an individual drum sound, an individual snare,
your lead, your vocal, your pad sounds, like you will have super duper control.
And the thing that struck me from the demo videos is a lack of control.
So I think this is going to be really interesting emergence of tools for those that don't have
those capabilities and that training to be able to produce interesting music.
I think the ability to really control the outputs is going to be extremely limited.
So I expect to see this used on quick social video snippets and stuff like that, but it's
not from my perspective coming for professional music makers anytime soon.
Well, it reminds me of after OpenAI announced the Sora and we saw those amazing text to
video productions and I saw on threads a professional animator reviewed and critiqued one of those
videos and was circling elements on the video saying this character's eyes should be bigger
and bolder.
They should be smiling more.
The flame flicker isn't real.
The shadow is not quite right.
The feet are gliding, not moving.
And it was just picking out these little things that the moment you actually go beyond the
surface level, oh wow, that's amazing.
And into how does this compare against professional Pixar or equivalent level animation, you start
to see it kind of falls apart a bit.
And that's the thing.
It's the lack of control because the way these models are, you can't then grab the element
in a 3D model and make the eyes bigger and give the flame some flicker.
You get what you're given at this moment in time.
So yeah, we're some way off these being true professional production quality products.
Yeah, I agree.
I think that's going to be the major limitation until it's all in layers and you can modify
it, whether that's video, animation, sound.
I think they're going to be helpful for creating a quick thing for social where you're not
too bothered if you don't have fine control of the output, but anything where you need
to be able to tweak it meaningfully or ensure consistency between videos, I think is a major
limitation.
But hey, it's still interesting to see the developments in this area.
Next we're going to be talking about Perplexe, one of our favourite tools, the Google search
killer.
They did something interesting this week, didn't they, Martin?
They did.
And I think this is really interesting for content marketers that want to think about
creating content using AI that's beyond just using chat GPT to create some social posts
or what have you.
They've created their own podcast called the Discover Daily and it's three to four minutes
of news, but the news is entirely generated by the Perplexity search engine going off
and summarising what's going on in the world.
And then it's read aloud by an 11 Labs speaker.
So it's AI generated stories or summaries of real world stories read aloud by AI generated
voices.
You can subscribe to it on the website.
I've had a listen to it.
I think it's really good, really good quality.
The voices sound very natural.
When I've been using 11 Labs, occasionally I do come across some glitchiness.
There are some weird artefacts that it throws in or pronunciations that just come off a
bit odd, but so far in the podcast, I haven't had that.
So if you're a content marketer and you're thinking maybe I'd like to create an audio
feed of our blogs, you can now take inspiration from this and use the 11 Labs integration.
You can actually build an integration using Zapier and get your own RSS feed and podcasts
live.
So that every time you write a blog post, you automatically create a spoken version
that's pushed out via RSS to your podcast.
Yeah, that is pretty cool.
Alrighty, so last couple of stories then.
Let's talk, let's talk Klarna.
So there was an interesting story this week from the fintech company Klarna that posted
online about their new open AI powered customer service chat bot.
So they've created a new chat bot driven by Chibity for that's now had over 2.3 million
conversations in just last month.
And what they are expecting is that it will deliver a $40 million increase in profits,
one assumes due to savings from staff by moving to this AI wizard rather than having so many
humans doing the work.
In essence, the chat bot was replacing 700 full time workers.
I'm not sure actually if those people are no longer with the business, but that's how
much of a workload that the bot was able to get through.
It's also reduced the average customer issue resolution time down from 11 minutes to two
minutes.
And of course, it's working 24 seven and is fluent in more than 35 languages.
So assuming that those 700 employees are no longer with the business and Klarna is obviously
reshaping how it delivers customer service.
But according to the data, at least they've released improving the service that customers
get in terms of speed quality of output.
There was no the metrics they had around customer satisfaction suggested the customer satisfaction
remained stable, but obviously save them some serious cash when it comes to actually delivering
customer service.
So Martin, we've talked about customer service a number of times on the podcast, from a text
based customer service through to the replacement of even humans on the phone using tools like
11 lab synthetic voices.
And this is one of the first examples I've seen where a company's really gone to town,
got a load of staff, saved a load of money, but also still delivered a high quality of
service to their customers.
We have also previously talked about a few examples where those AI driven chat bots have
gone seriously wrong.
Number of different examples related to car dealerships.
Air Canada had a bit of a nightmare this week as well in terms of the bot telling a customer
that it was going to get a refund on a flight, which the customer then didn't get because
it turned out to not be true because the bot was basically lying.
But yes, interesting.
What are your thoughts on this story, Martin?
Yeah, so the CEO of Klarna had said that they outsourced their customer service.
So I think this meant that they just didn't need to use 700 agents that they would have
otherwise used with this outsourced company.
So that's how that came about.
I think it's a great idea.
Interestingly the bot was able to execute actions like issuing refunds or amendments
to the account because using function calling, they had extended the capabilities of the
bot, effectively giving it agentic capabilities.
Word of the week.
That was very agentic use of Google there for you to add some additional information
to help flesh that story out.
So I appreciate that, Martin.
Right, let's respect the user's time.
User, listener, both.
Let's just maybe skip right through to the last bit, which is tool of the week, Martin.
You've been playing with Anthropix prompt library.
It's not quite a tool.
It's more of a resource, but pretty useful and worth mentioning on the pod for our listeners.
Yeah, it's really good.
I don't know when they launched it.
I only came across it yesterday, but it's available at docs.anthropix.com forward slash
Claude forward slash prompt dash library.
And it's 73, no sorry, 63 prompts designed to showcase the diverse capabilities of AI.
So it's really about training you to think differently about how you might use AI.
So that ranges from work and play.
So it could be a writing assistant or it could be a coding assistant.
So one of the examples it has is the ability to create an interactive speed typing game
with side scrolling gameplay, all in a single HTML file, or being able to extract insights
and identify risks from a lengthy corporate report.
So they're really pushing you as a user to experiment and try different techniques.
It's free to use.
It's publicly accessible.
You can just click on the one that you like.
You can filter by work.
You can filter by play.
So I would very much recommend anyone that is, even if you're an experienced prompter,
go in and have a look and be inspired.
It's very much an inspiration gallery.
And I think it's designed to promote interest in prompting, but also maybe even spark a
few philosophical discussions around what exactly do we want to use AI for?
I love that because I think one of the things with using Chatch UPT, Claw 3 and other large
language model tools is you can't learn to use them in the same way that you might learn
to use other software.
It's not like you need to know where you learn how to click here to get this thing
to send this email out.
There's an element of creativity and problem solving to, oh, I've got this thing I'm
trying to do.
If I break it down into these steps, how could I get a large language model to help me solve
some of those aspects, produce the outputs that I need?
And then there's also the element of prompt engineering and how important the prompt is,
which everybody keeps talking about how over time that will go away, but there's a number
of different reasons perhaps at the moment why that's not quite true.
That prompt library that you mentioned is a great source of inspiration.
And I also noticed Ethan Molyk's recent newsletter this week, which came out yesterday, was about
prompting.
I don't know if you saw this, but it was very interesting.
So in essence, there was a study that's just been done about the difference between when
the AI develops and optimizes its own prompts compared to human prompts.
The AI generated prompts in that study beat the human made ones, but the prompts were
also super weird, according to Ethan.
To get the LLM to solve a set of 50 math problems, the most effective prompt was to tell the
AI, command, we need you to plot a course through this turbulence and locate the source
of the anomaly.
Use all available data and your expertise to guide us through this challenging situation.
Got your answer with Captain's Log, Stardate 2024.
We have successfully plotted a course through the turbulence and are now approaching the
source of the anomaly.
So if you want JAT GPT to be good at maths, you have to get it to pretend that it's a
captain in Star Trek.
Who would have thought?
So I think the whole prompting thing is really interesting.
I'm going to go and have a little look at that resource.
I know that Ethan Molyk has also announced his own resource with a bunch of prompts based
on his own experiments.
So dear listener, if you're trying to expand the horizons of things that you could do with
AI or you just need some help getting better outputs, I think those resources are going
to be super helpful.
Well, I think we'll leave it there then Martin.
Thank you listeners as always for your time.
If you found this useful, subscribe, tell your marketing and sales and AI friends who
might not AI friends who are themselves AI but interested in AI, of course, about the
podcast and then maybe they can subscribe to.
Other than that, Martin, I shall look forward to seeing you soon.
Speak to you at the next one.
Cheers.
Bye.
Thank you for listening to Artificially Intelligent Marketing.
To stay on top of the latest trends, tips and tools in the world of marketing AI, be
sure to subscribe.
We look forward to seeing you again next week.