Welcome to Artificially Intelligent Marketing, a weekly podcast where we stay on top of the
latest trends, tips, and tools in the world of marketing AI, helping you get the best
results from your marketing efforts.
Now let's join our hosts, Paul Avery and Martin Broadhurst.
Hello, everyone.
It's Paul Avery here, which means you're listening to Artificially Intelligent Marketing,
two extremely exciting bits of news today.
And in priority order, the first one is this is episode 21.
How exciting.
We're very excited.
And then the other one is it's Martin Broadhurst.
Birthday today.
Happy birthday, Martin.
You don't look a day over 25.
It's very kind of you to say I feel immeasurably older than 25.
The joints, all of them, every one of them, even my knuckles.
The brain is firing and it's been firing hard all week because of all the wonderful AI stories
that we've been pinging back and forth.
Yes.
Yes, very much so.
Well, thank you for wishing me the happy birthday.
I'm going to have a lovely day.
Not really doing any work except for recording this podcast.
And let's be honest, this doesn't feel like work, does it, Paul?
This is just a good friendly chat informing our good listeners.
But no, it's going to be a good day.
We've got some good stories to cover today.
I've had some good news this week, which was welcome.
I'm now on the editorial board of the Applied Marketing Analytics Journal.
So that's going to give me interesting access to good research papers that we can bring
into the podcast and inform our listeners.
Without doubt, that's going to be some useful content for people coming forward as well.
Lovely stuff.
We've been planning some training for some of our clients coming through that are interested
in finding out how they can leverage the power of AI to inspire their teams to be more creative,
be more efficient, be more effective.
So that's been quite fun as well, hasn't it?
It has.
Yeah.
I'm looking forward to rolling that out with more clients in the coming months.
Absolutely.
Right.
With that, let's jump straight into these stories this week then.
I'm quite heavy on the synthetic voice, synthetic humans this week, starting with PlayHT 2.0,
which is a new conversational AI voice model.
Now you may remember the name PlayHT because when you listen to the intro of the Artificial
Intelligent Marketing podcast, that voice that you hear was synthesized using the first
version of their tool.
But what's really exciting is they have now unveiled their new model.
And it's a real major leap forward in conversational speech generation, at least according to the
company.
It's been trained on over 1 million hours of diverse speech data, and it can mimic voices,
accents, and languages with really amazing realism, even after hearing just a few seconds
of sample audio.
So in essence, what this promises to be able to do is, I think it's six seconds that the
press release says, six seconds of training on your voice and then able to mimic it.
And this includes some examples, which of course we can't really easily test, of someone
speaking in a language such as Spanish and then having that voice used to generate English
but maintaining nuances of that person's Spanish accent, which was quite an incredible demo.
It's also, if that wasn't exciting enough, I'm not thinking it is pretty exciting, it
also introduces the ability to control emotional expression when synthesizing the speech.
So now as a user, you can directly use the model speaking tones conveying happiness,
sadness, fear.
What you basically do is in your prompt, you just say the text you want the synthetic engine
to say, and then you just say scared, and it will sound scared.
And again, all of the demos of this, of course, always cherry pick the best examples.
But if it has the power that it looks like it does in the demos, then goodness me, it's
going to be extremely powerful.
So one of the things that I think has been able to help improve this is compared to the
previous model, which was kind of limited in terms of how it had been built and its
ability to really generate long form synthetic speech outputs.
But thanks to Play H2 2.0, you might have to look at the name that honestly, because
that's almost like trying to say artificially intelligent marketing.
It's got far more parameters, 10 times as many parameters in the model.
And because of how it's been built, it promises real time low latency speech generation, which
again, is a game changer for a lot of applications like customer service and just general conversational
abilities with an engine that you needed to be able to produce synthetic speech.
So yeah, expect to see this popping up in places like virtual assistants, customer service,
even audiobook generation.
There's, I think, probably applications for sales folks out there in terms of creating
customized voice messages and maybe video, which we'll come on to in a bit moment with
a story from Martin.
For marketers, I think it's just going to make it even faster and easier for us to generate
things like podcasts, videos, ads with custom voices, making it easier to give a play my
content button so that you can have people listen to your blog posts and stuff on the
go if that's easier.
And rather than it sounding super robotic, actually sounding like studio quality voice
over work by a human.
In terms of how this technology is really emerging and developing very quickly, Play
HT 1.0 was released eight months ago.
Now we have 2.0 and it is a real leap or at least from the examples looks like a real
leap forward.
So the model at the moment is in closed beta, so you can't go play with this yet, but it
will be made accessible through the company's API and their online sort of Play HT Studio,
which is what we use to generate the intro to this podcast.
Did you have a look at this, Martin?
Yeah, well, I'm thinking about the training and the six seconds worth of training data
required, which is obviously incredibly short amount of time.
So well done them for getting that down.
But as you were talking about the idea of being able to do the emotional nuances, I
was just thinking, do we need six seconds of someone sounding scared to get that into
the model, get someone with a fear of spiders and just throw a tarantula on them and record
that and now we've got their scared model.
I imagine that we don't have to do that though, do we?
That would be good though.
Could you imagine like the quick brown fox jumps over the whatever, whatever, but you've
got to do it.
You have to give the full range of performance, right?
Okay.
If you want to your voice in Denzel Washington style, you got to give the quick brown box
like you've got to be able to, you just move here.
Yeah, you've got to capture those six seconds of if you want it to sound sad, someone's
got to come in and tell you that your dog's just died.
Yeah.
No, the way I understand it is they must have been able to develop in the model, the ability
to sort of understand the sonic signature of someone's voice in that short timeframe
to then be able to reproduce it.
But I think we're going to know a bit more about this when we get our hands on playing
with it, but it sounds pretty cool.
It's not the only game in town though, is it Martin?
What else do we see this with?
It's really not.
Okay, so Hey Jen have unveiled their version two as well.
And theirs goes beyond voice synthesis into full AI generated avatar clones, so video
basically.
The results are incredibly realistic with people questioning on the demos that were
put out on Twitter this week, whether the avatars were actual humans.
I can see why people are questioning it, I think both me and you said we watched the
video.
So on the one that was published on the founders Twitter account, he had two videos showing
side by side and we both watched it thinking that one of them was real and one of them
wasn't.
One of them was AI generated.
It turns out they were both AI generated.
It was a bit mind blowing as we were both trying to find out, we were trying to figure
out the real one, but neither of them were.
That for me is quite a good Turing style test of like, when your brain's doing that, it
basically is believing both of them are real and you're looking for the nuances.
Yeah.
It was a bit of, can they do it for everybody or did they have to like spend a billion hours
of compute power just for the founder?
That's the question for me.
Yeah, I think that's the big piece of the puzzle.
Like if everybody's sat recording video of themselves in, you know, badly the officers
on their office webcam, is that going to generate realistic looking avatars?
We shall see.
But they say that in just two minutes, the founders voice and accent were cloned and
they were obviously cloned to a T so they've done a good job with that.
So not quite the six seconds that we heard about a moment ago, but you know, two minutes
of video footage is not a lot.
That video, I would say would pass for most people.
In fact, I reckon 95% of people would believe those video snippets.
So this synthetic media generation audio video suggests that we're about to see a full explosion
of this kind of content in all sorts of different media landscapes.
Like you spoke about eBooks or audio books and podcasts, video streams, you name it.
And I think the use cases are myriad.
I was thinking about specifically for businesses producing, I don't know, let's say webinar
content, but you're producing some webinar content.
You want to promote it on social media.
A lot of the time you're probably going to stick together a little graphic and post it
on LinkedIn and say, sign up for this.
But we know that actually if you were to do a little video snippet, that's going to be
a bit more engaging.
But to make that engaging little video snippet for your webinar, you're going to have to
write yourself a little script, record that, and then you're either going to record it
as we are now on a webcam, you know, not the best quality conditions in terms of you might
have to record it several times because you cock up the script a couple of times or the
audio is not quite right or this, that or the other.
Well now with a synthetic product, you know, you can make a one minute video and it's right
the first time every time.
It looks like it's recorded on studio quality equipment and you can put it out on social
media and that kind of short, the cost to producing short content has dramatically reduced
or is going to be dramatically reduced.
And we're just going to see more of it.
And I think for marketers, that's going to be a massive boom.
I agree.
But I think, you know, with great power comes great responsibility.
And we've talked about it on the podcast, I had a conversation on the LinkedIn's a month
or so ago about the deluge of crap content that is only ever a heartbeat of way with
the explosion of capabilities of these tools.
So I think it's great that we can produce it easily, but when there's low cost and low
effort to producing content, I think you can often see a low bit of low imagination and
churn like just get out.
And I just, I worry that our ability as marketers to saturate channels, like people always say,
enjoy a particular channel until markets get hold of it and ruin it.
Right.
And I think there's something to that.
I think we've done that to a number of social channels and this just democratizes the ability
to do that because a certain amount of money or skill that you would have needed to produce
this type of content is going to be vastly reduced.
The other thing that comes to mind is on the sort of sales prospecting side, which is maybe,
you know, touches on marketing's role a bit in terms of supporting with content creation.
But I get a lot more outbound crappy emails than I used to.
I'm pretty sure they're all using the same software or the same agency because even the
email templates are the same.
It's like, oh, so dull.
Just leave me alone.
One great way to get around that was to create personalized videos for your prospects, right?
That showed you really went to the effort of creating something just for them.
People tried to find ways to gain that, right?
By having a thumbnail with a person's name written on it, but then the actual video was
like not customized for them at all.
Well with tech like this, sales reps are going to be able to do that at scale.
And again, without the necessary effort put in to make sure that the content is valuable
for the recipient, it's not going to work.
Like it will work for like four to eight weeks or whatever the timeline it takes for us as
humans to spot that this is just us being fooled into thinking people think we're more
important to them than we really are.
So it could be interesting to see how that plays out.
And I think for both our first two news stories, the suggestion to marketers would be just
because you can produce a lot of content at scale, don't really use it.
Just don't.
Yeah, just don't.
Just use it to produce good content.
If you've got great ideas and you can add value to your audience and actually scale,
opens up new avenues for you to generate value.
Brilliant.
If it's just churn and burn, goodness me, marketers are going to make the world a pretty
dreadful place for us scrolling through LinkedIn, looking for one half decent piece of content
that wasn't just churned out without any thought.
Just because you can doesn't mean you should.
I agree with that.
Right.
So with that synthetic peeps and voice is done, let's move on to chat spot for our new
story.
So users of HubSpot may be aware of chat spot, which is HubSpot sort of large language model
chat botty type tool that we've been following on the podcast over the last couple of months
since it was launched.
I think it's fair to say it launched with some great ideas, but you couldn't really
do much stuff with it.
Well, we're very happy to report that it is starting to really emerge and mature as a
tool that you could actually consider using.
So with this tool, it's had a recent makeover this week.
So it's got a bunch of new features and has different sort of layout and usability to
it.
One of the features really, really cool is the ability to summarize CRM records.
So with a simple command, chat spot can now provide an AI generated summary of a contact
or company's record and not just the stuff it scrapes off the web, but its ability to
actually summarize the notes that you as marketers and sales folks are taking in the system.
So the ability, for example, to figure out what web pages are needed, visited before,
you know, before you have your prospecting call.
So that's pretty awesome.
The interface itself has seen quite a few improvements, going to be a lot easier to
use and things like templates and chat have been separated just to make it a bit more
of an easy experience.
And the tool can also now summarize YouTube videos, which is interesting.
So you can drop a YouTube link into chat spot and get a summary of the video content, which
is a fairly powerful application.
I haven't had a real play with yet, but I want to dig into, have you had a play with
that yet?
Yeah, I tried it on a couple of videos.
They seem to do a good job.
All of these kind of systems offer these summarizations are basically just taking the transcript from
the page and putting that as the input.
So it is as good a job as many of the others that I've used.
And if you're a HubSpot user, maybe even a HubSpot free user, I have to check that, but
chat spots effectively free, right?
So you could absolutely try that and see how you get on.
It will also give you information on podcast stats.
So if you put in a podcast, it will give you a summary of the podcast.
And we checked in artificially intelligent marketing and it found us and it knew that
we'd published it and knew how many episodes there were.
So that was good, good stuff.
It was, it was also slightly sarcastic.
In what sense?
When it said it gave us a rank and it said, where do we rank?
And it said top non.
Yeah, it gave us a rank, rank, but that's okay with us because it's a podcast that's
on the up as you will know, dear listener.
And you will share this with all your friends.
In fact, we're on a mission now for chat spot to give artificially intelligent marketing
a rank that's not top none.
And you dear listener are now a part of this very important social initiative.
Thanks for your help in anticipation.
Why should marketers care about this?
Because as we've been talking about having these types of tools baked into your marketing
and sales CRM is going to give you the type of power to create personalized email programs
for individual leads.
And I really feel like this summarized CRM records tool is a precursor to leveraging
that type of data for other content generation.
So I think it's great as a sales rep, but I think that's what comes next.
I actually had a really practical use case for it this week.
So I've sent out an email campaign this week, or should I say back end of last week, and
I just had chat spot open and I just wanted to check how that campaign had performed open
rate click rate kind of thing.
Normally I would just jump into my dashboard and do that, but I had chat spot open anyway.
So I was just asked it, now it had gone and it told me, it gave me exactly the sort of
report that I would have got if I went into analytic.
And that was super handy.
Just having that there can interrogate the data.
I can ask it to then show me who engaged with it.
And the more I dug into the data and wanted to interrogate it more, it kind of fell down
in a few areas.
But as a starting point, it worked really well.
Yeah.
And it's not many months old.
I think they're doing a lot of work behind the scenes to improve it.
And I think with a lot of these sort of next level applications, it's the teasing of the
progress we're making rather than would we on the podcast today say marketing and sales
folk get chat spot.
You won't need anything else.
It's going to do everything you need.
We're still in the emergence phase.
We get asked a lot, don't we Martin?
What tool should I use for this?
What tool should I use for that?
It's so dynamic at the moment.
Whatever recommendation we make this week is going to be different probably next week
or the week after, or certainly in a month.
And as much as I realize this is not helpful advice, you have to keep an eye on the tools,
especially the ones that baked into software you already use like Salesforce or HubSpot
to just see what's emerging and what you could tap into.
And I think that's the key because I think if you are a HubSpot user, chat spot is arguably
reaching that tipping point of true utility.
Whereas for a lot of time, even the template chats that you could click a lot of the time
ago, I know what you want, but I can't do it.
Like it's literally that would be its response.
And it's like, well, if you know what I want, just do it.
But, you know, and I appreciate that getting these tools to work is a huge undertaking
and it's just good to see the evolution of chat spot.
And I do think I can see it starting to become more of my workflow now it's reached that
tipping point.
Yep.
Very much agreed.
What's next for us, Martin?
There's some new updates in the open source model world.
So Stability AI have released new Beluga models and they are setting records.
So they released two powerful new AI language models, Stable Beluga 1 and Stable Beluga
2.
They put this in a press release and a little detail I really liked about this story was
that they said previously code named Free Willy, the models were renamed after the gentler
and more harmless Beluga whale.
For those of you that have seen the movie Free Willy, my question, how aggressive will
he was?
I don't think we should cut that audio snippet out either because I don't think I'd do very
well online with that.
But yeah, I actually featured when you were out of Play Mart and the Free Willy models.
It's interesting to see them get rebranded.
Something's driven that outside of the nonsense in the press release, I would suggest.
Yeah, they seem to talk about just the reputation of, they put a line in saying that the reputation
of orcas as being these more aggressive animals.
These models were, they needed to be, well actually they specifically say that the fact
that orcas are called killer whales was not ideal and these models are designed like claw
from anthropic to be helpful and harmless.
So the Beluga whale was chosen.
Anyway, enough about the branding.
The Beluga models were trained on synthesized data inspired by Microsoft's orca model, hence
the Free Willy idea.
Stability AI filtered the training data to ensure fair benchmark comparisons.
It uses only 10% of the data size of the orca paper, but it has through their fine tuning
of this model, it's got really high performance, exceptional performance in all of the benchmarks
and they've outscored many of the other similar models on the open source scene such as GPT
for all and AGI of evals.
So yeah, in fact so much so that stable Beluga two currently ranks number one on the open
large language model leaderboard showing strong language understanding and reasoning capabilities
and they're the things that I always look for is that reasoning capabilities because
when you look at GPT four versus GPT 3.5, it was the reasoning that just took it so
much further along.
You go, wow, now I can do more with this model.
So it's a major advance as with all of these stories in the kind of open source world,
why should we care commercially as organizations?
Why should we care as marketers?
Well, better open source models and more competition in open source models is going to give us
more choice.
That said, this is only available under a research license at the moment.
Like Meta's recently released Lama 2 model, which was given the full commercial license.
So until they give it the full commercial license, it's just something to keep an eye
on.
But yeah, more competition in this space is going to give us more options for creating
chat bots and customer service agents and all of these kinds of useful things that I
think AI is going to do for us in the near future.
Absolutely.
You mentioned the LLM tool and leaderboard GPT for all.
I actually installed that a couple of weeks ago and it was quite interesting to play with
it.
Have you had to play with it, Martin?
I have, yeah.
And you can download all the different models and run them locally.
Yeah.
And I've got like 16 gigs of RAM and I've got like an i7 equivalent, I think.
And they ran, but they ran pretty slowly.
And I think it's really interesting seeing the explosion of this space because I do think
the future for a lot of applications is going to be on board large language models when
they figure out, as computing power on board continues to progress.
We've talked previously about how Macs M1 and M2 chips have things like architecture
built in them to enable this type of stuff.
In fact, I need to get GPT for all installed on my Mac, actually see how much better it
performs.
Yeah.
And I think this is where these types of models...
That was the moment that all these models really suddenly were a thing for me because
you can select from, I don't know, 10, 12 models to download and there's a summary to
try and give you a feel for.
This one's like massive download.
It's really slow, but it would give you great output versus small download, much faster,
but maybe not as creative or good at reasoning and stuff.
So the more that we take big models, make them smaller and somehow tweak the parameters
and make them as good, if not better, the closer we get to running large language models
on our computers, which for some organizations may be needed.
They just may not be comfortable having the data go into the cloud.
I can imagine healthcare applications where the regulations and restrictions on how you
use data, where you move data, where you store data.
That could be a bit of a game changer for them.
So I think there's also applications in the marketing space for those companies that may
not be able to ever use cloud-based tools for this because the sharing of data with
them is just too complicated to set up and takes too long.
Although one thing to note on that, and I think this came out of Maycon conference for
me.
I realized that not a lot of people realize that Anthropic and the Claude model is HIPAA
compliant.
Is that right?
Now, I also know, I'm sure you've mentioned that on a previous podcast actually, but yeah.
Okay.
Well, let's skip and talk a bit more about Anthropic's new AI model because they've released
an improved version of Claude Instant 1.2, so some more model news here.
In essence, this new model incorporates some of the strengths of the recently released
Claude 2 that we've talked about on the podcast, showing improvements in areas like maths,
coding, reasoning, and those types of things.
It also generates longer, better structured responses and follows your formatting instructions
for users of chat GPT-4 and 3.5 Claude Instant 1.2 is probably a reasonable option now at
this point, especially because it has a larger context window of about 75,000 words like
Claude 2.
This is pretty cool.
The model is also apparently less prone to hallucination and a bit more resistant to
misuse.
Why is this interesting for marketers?
I think for us on the podcast, seeing that Anthropic and to a certain extent, OpenAI,
are continuing to invest in developing and improving older models.
There was a bit of research that came out a couple of weeks ago that we talked about
on the podcast looking...
The argument of the research was that GPT-4 had got worse, but what kind of got lost in
the shuffle was it looks like GPT-3.5 had got better.
If Claude Instant 1.2 is a significant improvement over 1.1, these are smaller, faster models
whose performance is improving.
When speed is important, you can start to rely on them for some use cases, potentially
content creation use cases, summarizing call transcripts for interviewing SMEs and producing
blog posts and all of those types of things.
I think that's really interesting.
For people who are drifting into power use and playing with APIs like we do, Martin,
the ability to access cheaper models and get good outputs more quickly is important and
therefore it will be important to the software developers.
That could see powerful software tools for marketers that cost even less because these
smaller models are improving a lot.
Yeah, I found with Claude 2.0 that on one particular use case that I was...
It was a workflow that I was building out in Zapier and I just couldn't get it to run
in time.
I kept hitting timeouts because it was a little bit slow and Zapier was just like, oh, we
didn't get a response in time, the end.
So I switched over to Claude Instant, which worked, but the outputs weren't brilliant.
I had to adjust my expectations slightly on that.
So this is great.
I'm very pleased that we've now got a model which is faster and going to give us what
I want.
Absolutely.
We've been hitting those same timeout issues when we use a tool called GPT for Sheets.
Well worth having a look at and a play with for those of you that are into figuring out
how to better use more exotic approaches to improve your marketing outputs.
We use it to summarize interesting stories that we find where we just paste the story
into a cell in the sheet and then we ask Claude 2.0 or some other model in the next column
over to summarize it for us.
We've been getting timeout issues with GPT 4.0 and Claude 2.0.
Whereas, if we can now summarize them with Claude 1.2.0, it's going to be cheaper in
terms of API credits, but also faster and we won't hit that problem.
So yeah, interesting stuff.
I do love playing with different tools that access all of this stuff through API.
It really opens up your creative juices.
It does.
And when you start using the API, the speed of the, when it is quick, the speed is so
much better than using the chat GPT interface or similar.
Absolutely.
Right.
Next news item is with you Martin Longshot.
Yeah.
So Longshot is an AI copywriting tool.
It's one that I've been on board with since basically day one of their launch.
I got an early, got access, had a subscription to the tool since then, and you know, similar
to any AI copywriting tool, they've got templates for writing blog posts and blog introductions
and meta descriptions and social media posts and all of that kind of stuff.
And it does a very good job.
But what I really like about Longshot is that they've always been trying to address customer
needs beyond just content generation, because you can now do that anywhere.
In fact, they were the first tool that I really saw that came out with a fact checking tool.
They had fact GPT, which would do, you could get it to write AI generated content about
things that had just happened.
For instance, in the demo, they talked about the world cup final and being able to write
about the world cup final the day after it.
You didn't have to wait until the models powering it were trained on that kind of data.
So they did a very good job there.
And they've just released a new tool called BotShot.
And BotShot is where you can now create a bot, customer service bot or some sort of
bot that sits on your website or elsewhere that is trained on your data.
And it's a no code solution for doing this.
So we talk about being able to train large language models on your existing proprietary
company data.
Well, now you can do that in a no code way using the BotShot interface.
They released it last week and they say that you can have your own chat GPT for your website
and deploy it in under two minutes.
So if the workflow is as good as it looks, and I think the workflow generally that they
put together is very good.
This is an interesting tool and an interesting way to add new capabilities to your website
without having to really get into things like fine tuning and using vector embeddings and
things like that.
It's interesting to consider how they must do it to get around all of some of those issues.
Is it like customer facing only for like on your website or is it, can you use it almost
like an internal tool?
I think you can use it as an internal thing.
Yeah.
Ultimately, you would just choose where the bot sits and how, you know, the kind of address
that people would interface through it.
I think the way that they do it and my understanding of it is that you give it the data and it
manages the vector embeddings piece for you.
So they've built the workflow that just means that you don't have to be the technical person
that figures all of that kind of stuff out.
Okay.
One of my sort of business fantasies is the ability to codify the training that I've given
people who've joined our agency over the years to in essence create a pool chatbot that the
team can ask bot pool instead of real pool and hopefully get half decent answers out
of it in my stead, right?
Getting me up to focus on other tasks, which could be done through knowledge bases, pulling
a ton of emails out or, you know, other different ways.
And anything that's going to make that type of application a bit easier is very welcome.
And a lot of companies, obviously a lot bigger than Biostrata probably have massive knowledge
bases already that they can open this stuff up to.
So good story.
I think I've got to go and have a little play with bot to shot by long shot.
It's good job.
It's early in the morning, isn't it?
There's a few tongue twisters in here today.
Next story, we're talking Zoom.
So probably a lot of you are and have used Zoom, we're on Zoom right now, the popular
video conferencing platform, which has updated its terms of service to allow the use of some
customer data to train its AI models.
Now these new terms, which came in in July 27 specify that Zoom can use service generated
data such as product usage, telemetry and diagnostic data for machine learning purposes.
However, supposedly the terms state the audio, video or chat content will not be used for
AI training without the customer's consent.
So an interesting one on this one, because I think this was spotted in the terms and
conditions, like it took a couple of weeks for an eagle-eyed person to nab this because
of course we're reporting on this today on the 14th of August.
So that is, I think, indicative of the type of information that we're providing to software
providers that they're potentially using to train AI models on.
And we'll just come to that in a moment, because if you're using Zoom's two new AI features,
which came out in June, which is a meeting summary tool and a chat message composition
tool to use these features, you must sign a consent form allowing Zoom to use your data
to improve the performance and accuracy of these AI services, a la in essence, we're
going to use your content to better train our models.
So if you're opting into using these cool new features, that's the bit where it says
we don't use your data unless you give us permission.
That's the bit where you give them permission.
It's not, hey, we want to train our tools on your data.
No, it's packaged in a here.
Do you want this cool thing?
Oh yeah, I want to play with that cool thing.
By the way, we could use your data now.
Right?
So you've got to be really hot on keeping an eye on these things.
I think this is where, particularly in areas such as the EU, companies are going to get
really unstuck.
They're going to have to be really, really explicit about separating this out in the
terms and conditions.
So we know that for instance, with OpenAI, if you put data into chat GPT, they're using
that model to train the future models.
If you use the API, they're not.
They're keeping those separate.
And I think the likes of Zoom are going to have to be really explicit about, look, we
can give you this cool new feature and we will take your audio and your feeds and we
can do the summaries and we can do the meeting notes and we can do the sales coaching like
Gong or whatever it is that they're planning to do in the future while separating out the
are you opting into our further training enhancement of our own AI models?
Because I just think that regulators like the EU will come down on them like a ton of
bricks.
Yeah.
It's quite a messy one, isn't it?
From that perspective.
We've spoke about in the podcast before.
If the product's free, it's because you are the product.
But in essence here, even if you're paying for this product, you're still the product,
right?
Because a lot of us are paying for Zoom.
And I think ultimately, there's so much data needed to train these models that wherever
these companies can get access to it, it becomes a very valuable commodity for them to access
and that we're going to see more and more of this and probably has happened for many
years and now our data is being used and we're just maybe not as aware of it and now we're
becoming aware of it.
I think it's also going to make it very interesting to see how this plays out when Copilot gets
rolled out to Office 365 and Google's Workspace version because ultimately, the power of a
lot of those tools is going to be predicated on being able to in essence, natural language
search through all your own documents and have a chatbot like to create content off
the back of them for you.
While by default, they're going to have to access all your stuff.
How is that going to be managed?
Is there going to be a training element to that?
If you opt into using your information and allowing it to be part of the training set
for future runs, can you get a discount on your subscriptions?
Who knows, right?
Because it's very interesting to see how all of that's going to play out.
Agent Bench.
Tell us about Agent Bench, Martin.
So on the discussion last week or in the interview last week with Brennan Woodruff, we spoke
about AI agents and the capabilities of agents.
So this is large language models being able to complete tasks, being able to actually
do things, not just tell you the next word or the next token, but actually go and execute
things for you.
And how well AI agents work has never really been benchmarked until now.
We've now got Agent Bench, which is a new benchmark developed by researchers.
That's a twisting words like researchers.
What a complicated word for 10.30 in the morning.
Researchers at Ohio State University, UC Berkeley and Tsinghua University in China.
So Agent Bench evaluates large language models across eight different environments, all of
which simulate real world situations.
So they look at the ability to interact with operating systems, databases, knowledge graphs.
There's a digital card game, which they have to play.
There's some lateral thinking puzzles, and householding, which is kind of interaction
with the real world.
So it's almost like a 3D kitchen environment, I think that one is, almost like robotics
or controlling robotics.
I think it's kind of virtual household environment.
Web shopping and web browsing.
So they're the different categories.
And the goal is ultimately to test the large language models capabilities for coding, logical
reasoning, knowledge acquisition, and actual interacting with the world.
So the researchers benchmarked 25 large language models, both API accessible large language
models, such as GPT-4, Claude, Claude Instant, and then a bunch of open source models as
well.
And it should come as no surprise to hear that the API based models outperformed the
open source models significantly, but there was one standout model amongst the list.
Paul, I don't know if you want to have a guess at which model outperformed the others.
Couldn't possibly, yes.
It wouldn't be an open AI model, almost certainly.
That's where you're wrong.
It was an open AI model and it was their state of the art model GPT-4.
All you are blown away.
So it scored on the agent bench overall score, it scored 4.41.
And that compares to Claude, which scored 2.77 in second place.
So significant outperformance or overperformance there.
It smashed the rest of the field, let's be honest.
Yeah, it really did.
There's a graphic that shows how it performs in each of those areas and it absolutely dominates
in every area except bizarrely the web shopping.
GPT-3.5 Turbo outperforms GPT-4, but in every other field GPT-4 outperforms and scores really
highly.
I think generally speaking though, the researchers came away with the conclusion that none of
them, even GPT-4 with its great performance relative to the rest, they're not ready for
public consumption in terms of actually rolling out to the real world as autonomous agents
just yet.
But I think what it demonstrates is that it's coming.
We are on the precipice of some very capable autonomous agents coming to the market and
that's going to change.
I mean, it's interesting to think about it from the marketer's perspective, but just
as consumers, as knowledge workers, what this is going to enable us to do is, it's kind
of hard to imagine right now.
And I think for me, the AI agent space is where if I kind of fast forward, if I throw
it ahead two or three years time, that's going to be the real big difference to now and then.
We look back to, I don't know, let's say like 2005, what's really different.
Well, smartphones and apps is the kind of thing that you go, oh yeah, well we had these
flip phones in 2005.
In like 2030, I think we'll be looking back at 2023 and going, we didn't even have those
AI agents that are just constantly doing tasks for us behind the scenes now.
We talked about this a little bit last week.
HyperWrite, which started out as a writing support tool, really like Reowriter, Jasper,
all these other tools, but they've got their AI assistant now, haven't they, that's appearing
to gain more and more capability.
In essence, you ask it to do things for you like book a flight for you or find a recipe
or on the website, they give examples like organize my Gmail inbox and draft responses
for me, find engineering candidates on LinkedIn.
So really stringing together some of those tasks, which I think is interesting and actually
is a fairly handy segue to our last story of the week, which is about MetaGPT, which
aims to replace entire software companies.
So this is a really interesting one that we saw on the superhuman newsletter this week,
which is an AI system that basically works by assigning different language models, specific
roles like engineer and project manager, and you give MetaGPT a simple prompt and it activates
a chain of AI agents that collaborate and build sites and more.
So the example that was given was that a developer created an entire flappy bird game in just
10 minutes using the tool and only providing one line of code and MetaGPT did the rest.
So to your previous point, mine, this is an interesting emerging area of chaining together
tools to develop more comp and deliver more complex outcomes and a bit of a glimpse into
the future where creating apps and other digital products becomes so much easier and faster,
even for people who don't have necessarily the training skills or background for that
type of stuff.
I've been using chat GPT to create a little bit of script for me that nobody else in the
world would need, but I can't code.
I can't use Python, but now I can.
I still can't write the code, but I can get code generated for the specific task that
I need and executed and off I go.
Now, if I can do that, everyone's going to be able to do that very soon.
And I think this example with MetaGPT is a great example.
I love the fact that he's created these specific roles like project manager and engineer.
It's kind of company in a box, isn't it?
It'd be interesting to see how this performs on, because this is AI agent, but it's still
very specialized.
This is exclusively focused on the task of creating software.
But using the agent bench scorecard, I'd like to see how MetaGPT performs in the benchmark
tests because it isn't going to perform very well presumably on things like householding
and lateral thinking puzzles maybe, but would perform pretty well on things like operating
systems, databases, knowledge graphs.
Yeah.
I think as well, the focus here quite obviously, but it's worth stating on digital outcomes.
Right?
Until robotics catches up, we're going to be able to create perhaps autonomous chains
of agents that can do interesting things automatically and autonomously in the digital world.
But until we overcome that interface problem to the physical world, that's where it's
going to be restricted to.
That lots of things happen in the digital world, most knowledge workers these days operate
almost exclusively in the digital world.
So it's not like they won't have a massive impact.
They will.
But I think once the robotics catches up, that's when we hit the bottom of another exponential
curve from there.
I really love your point as well, Martin, about using GPT-4, which you can do today
to help you with code.
You're one of the best copy-pasters I know, asking great questions of GPT-4 and then copy-pasting
the results and getting stuff to run for you.
I do you a disservice.
I'm joking, obviously.
Fine.
You're a fantastic problem solver.
It's not.
Do you know what?
You're not far off though, because yeah, you can write a good prompt and you can go back
and forth and refine, but ultimately it comes down to just asking a couple of questions
and then copying and pasting.
And sometimes when it doesn't give you the output you want, literally just go back in
and saying, well, that didn't work because it said this and running those time and time
again.
Absolutely.
And if you think about it at the moment, most marketers and sales folk are relying on off-the-shelf
software and SaaS platforms to do the stuff they need to do.
But as these types of, well, as the approaches that you take already are empowering you to
create snippets of code to do things you want to do that are very specific to you, obviously
you're an early adopter and you're having to think around problems and be quite creative.
But once these become packaged in products or out of the box type tools, the imagination
becomes the only limit really in terms of as a marketer, oh, I wish I could do this
thing.
Why isn't there a tool out there that does it?
Build your own.
Ask your agent team to go build it for you and then give them some feedback when it comes
back and iterate it through.
So interesting times.
I really wonder how long do you think it will take, completely unfair question for you Ryan,
how long do you think it will take before we can start to use some of these agents to
really get useful stuff done?
I think we'll be looking at probably about two to three years when they're really fully
functional.
At the moment, we're starting to see the glimmers of it and we're going to see buggy systems,
trial and error, iteration over time, but I think, you know, throw it out to a three
years time and we'll have, it's probably a similar timeframe actually to the likes of
GPT-5 and the kind of multimodality coming in.
Yeah.
I mean, code interpreters interesting because it takes you through its reasoning and tries
different things.
And I don't know if that's like an artifact or like it's for show to make the human feel
like inside the loop of what it's doing or there actually needs to like reason with itself.
And it's like, oh, okay.
Well, to access, to analyze data you just gave me like that, we probably need to format
it in this way or run this type of analysis and it doesn't need to say that out loud.
One assumes obviously the showing the text part to me is absolutely for me, but I wonder
if that's how it actually reasons.
Like it goes through a reasoning process.
With you know, chain of thought prompting is a thing, isn't it?
You know, asking large language models to show their reasoning and show their working.
The documentation, if you've ever read the documentation of Claude on how to get most
out of the model, just look at the documentation and it says, ask it to show it's working,
but it gives you examples of how to put that into a prompt itself.
You can't just ask it a question and then at the end of it, put, show you're working
there's a specific structure that you have to put in, in order for it to then take the
steps which gets a better reasoned output.
So I would imagine that part of code interpreters set up is the same thing.
It's been been trained to go through step by step and show it's working in order to
get the outputs that it requires.
Yeah.
Well, we will see if your prediction comes true.
Personally, I hope we get it a little bit faster than that, but I think it's probably
more realistic what you said.
Right.
As it's your birthday, I think we should let you go off and enjoy the rest of your day.
As always, thanks for your time on the podcast.
And look forward to the next one.
Thanks for cracking stuff for our lovely listeners.
If you're not subscribed yet, hit subscribe because we're going to provide you with these
insights every week.
And of course, we're now on our mission to have Chatspot effectively recognize artificially
intelligent marketing as a podcast worth ranking.
So please share this with your pals, your friends, anyone in marketing or business land
that you would feel would benefit from some of the insights and news and information and
product that we talk about on the podcast.
Other than that, have a lovely week, everyone.
Thanks again, Martin.
Bye bye.
Cheers.
Bye.
Thank you for listening to Artificially Intelligent Marketing.
To stay on top of the latest trends, tips and tools in the world of marketing AI, be
sure to subscribe.
We look forward to seeing you again next week.