Welcome to Artificially Intelligent Marketing, a weekly podcast where we stay on top of the
latest trends, tips, and tools in the world of marketing AI, helping you get the best
results from your marketing efforts.
Now let's join our hosts, Paul Avery and Martin Broadhurst.
Hello listeners, welcome to episode 37 of Artificially Intelligent Marketing.
It's Paul Avery here, joined as always by the fantabulous Martin Broadhurst.
Mark Tino, how are you today my friend?
I'm living my best Friday with elves in the background for those who are watching the
video.
I'm counting down to Christmas.
Wouldn't normally be this Christmas prepared, except having a toddler in my life these days
has just brought the whole thing forward a little bit.
You are having an Elf of a Time.
Cool, that one doesn't quite work.
We're not going to cut it though, we're going to make you listen to those types of jokes
all the way through the episode.
Yeah, well I'm glad you're enjoying it, I'm loving your background.
So much so I'm probably going to edit my background in post so that when people see this they'll
go, oh wow, Paul made the effort on his background as well, which in real time I hadn't, but
they will never know that.
I mean they will now, but otherwise they wouldn't have known.
We got some cracking stories this week, haven't we Martin?
In fact, before we get into our first story I've got to tell you, as you know I was at
a conference last week, I was at the SAMPS annual US meeting in Boston and we were talking
about AI.
While I was talking I did a couple of short talks on different topics and then there was
a panel and in the panel I slightly bashed Google.
I was like, they're trying to catch up, they're a bit late to the party, yadda yadda yadda.
I sat down and one of my good friends, Jeremiah Wurst, Jeremiah, little shout out to you,
he turned to me, shook me his phone and he went, you know that Google have just launched
Gemini, why are you speaking?
So I am going to take personal responsibility for accelerating the release of Google Gemini
by giving Google a hard time on a panel with 70 people in the room that they clearly had
a representative at and said we need to get this out now because I'm fed up with the
folks at Artificially Intelligent Marketing giving us all sorts of shit about it.
So the share price dropping in real time based on your comments and they said release press,
please press release now.
But it was quite funny and it also did really serve to really highlight another point that
we made again and again during the day which is this stuff is moving lightning fast.
When I started my talk my comment was fair, by the end of my talk it wasn't fair and
I don't know anymore.
But that is AI in a nutshell and especially the applications and technologies we're seeing
emerge for marketing and sales folks and business in general.
So I think an overriding feeling at the event was it's extremely hard to stay on top of
all of this stuff and if you're listening to this and that's your feeling, if you feel
like you're drinking from a fire hose and you're completely overwhelmed, I don't think
anybody feels any different.
You are absolutely in the crew when it comes to that.
Even nerds who spend many hours a week on this stuff like Martin and I will come together
to do this podcast on a Friday and between us there'll be stories that Martin's seen
and I haven't and vice versa because it's just so hard to stay on top of things.
But that's why we give you this hour.
Hopefully we can do at least 80% of the heavy lifting for you and with some of that heavy
lifting in mind, Martin, why didn't you tell us about the biggest story this week?
That Google company who said we're no good at AI, they've announced a GPT-4 competitor
in the form of Gemini.
So Gemini is their multimodal large language model.
It actually comes in three sizes.
So they've announced Nano, Thro and Ultra.
Ultra is the big one, that's the one that is supposedly capable and well actually outperforming
according to benchmarks GPT-3.
I think it beats it on 30 out of 32 benchmark tests.
Thro is the one that they've released now.
It's effectively GPT-3.5.
It's available via API and BARD if you're not in Europe or the UK.
You can actually access it if you use a VPN, which I have done.
I've played around with it and yeah, it's similar to GPT-3.5.
It's fine.
Ultra looks really interesting because it's multimodal in a way that I found quite interesting
the way that they described it during the launch event.
So Demis Hesabis, the CEO of DeepMind, they've clearly been working closely with Google Brain
and DeepMind working together now.
They've brought these two organizations together.
Demis Hesabis said that this is a room multimodal model rather than an interface that gives
the appearance of multimodality.
So I think what he was referencing there was if you use ChatGPT, GPT-4 is multimodal in
so much as you can input text and image, but it can only output text.
But if you're using ChatGPT, you get the appearance of multimodality because it will output images
through the model Dali, which is connected to it, and you can input voice.
Again, that's actually using Whisper to transcribe it and then inputting it as text.
And it can output voice.
But again, that's not actually part of that same model.
It's a different model.
It's all connected up.
Whereas Google is saying that Gemini is truly multimodal.
And they actually give some interesting examples of this in the demo video.
For example, they have somebody speaking in Chinese and Chinese dialect and asking it
to translate into English.
And they said this would actually not work in the current models that are on the market
because the language, the input language is tonal and that would be lost in the Whisper
transcription of it.
So you would lose some of the nuance around the input of audio like that.
It looks really powerful.
The proof is in the pudding though, and that's what we're all waiting for to get our hands
on it because it isn't available to users.
Ultra isn't available to users until early 2024.
So it looks very exciting.
However, it was not launched without a little bit of controversy.
The launch video showed it doing certain things that seemed very impressive, I must admit.
So the example in the video was hand drawn images, little sketches on Post-it notes.
For instance, two cars driving down a hill.
One of them looking like an aerodynamic car, the other one like a block.
And it says, which of these would go faster?
And the model responds, the one on the right, which is more aerodynamic.
And then they keep editing the pictures by drawing new bits on them and speaking to the
model and it converses back and forth with them.
And it looks like this whole thing takes place without any text input.
It's just somebody speaking to the model, showing it pictures and changing the pictures
and saying, what about this?
What about that?
And the model responding.
Well, it turns out that whole segment was completely manipulated and not at all as described.
The images were actually uploaded with text prompts.
It was not a speech back and forth with images being shown in real time.
The whole thing was a manipulation.
And actually what people took umbrage with was that what the voice was saying as a kind
of input prompt, so what the human was supposed to have been saying, didn't match what the
text based prompt actually said.
So they gave more detail in the actual text prompt that the model got.
And people were quite annoyed about the manipulation there.
But that aside, this model is very impressive on the face of it.
It does have some very cool capabilities.
At the other end of the scale, going from Gemini Ultra down to Gemini Nano, this is
also very interesting because what they've announced here is a RNE model that you can
actually run on a mobile phone without requiring any internet connection.
And this is going to be rolled out to the Google Pixel 8 and the Google Pixel series
in the future.
So I'm very excited to see the evolution of these small language models for mobile devices.
Yeah, there's a lot to unpack there.
The tonality thing was very interesting and I imagined, and I would love it when people
can get their hands on this, being able to speak with a bot where it's listening to your
tone of voice and it can tell if you're stressed or scared or upset, right?
So like we've talked before about the different applications of bots in the future and how
when people like to rate doctor outputs as based on how empathetic they were, GPT-4 was
rated more empathetic than real human doctors.
Well, how much does that empathy increase when a model can actually understand the tone
of voice, not just the text, like content of the information that's being presented?
That's pretty interesting.
On the...
We fudged this to make it look better.
What a shame.
So desperate to show they were back in the game that they just reached 10% too far.
And because we know that, I think many of us are now...
I think many people are like, all right, I'm not going to reserve my judgment until I see
it in action, but you just lost my trust.
So I'm personally, I'm assuming it's as good as GPT-4 and maybe slightly worse, honestly,
not slightly better.
The other thing that's interesting is the paper they released with all of their performance
on the benchmarks.
I do love the...
There's been a few things floating around on the Twitters of, wouldn't it be nice if
we could all mark our own homework in that way?
Obviously, these are standard benchmarking processes, but there are a few little nuances
that people have untangled in there in terms of the prompting techniques and the number
of prompts used to get a better result for Gemini over GPT-4 that are, again, just a little
bit questionable.
So I'm looking at this and thinking it's probably GPT-4 level, but what I really want to know
is what applications are really going to benefit from a model that was trained multimodal first.
That could be a bit of magic in there because it's not clear to me because now I don't trust
any of the demos, how that will translate into real world applications that GPT-4 with
all of its add-ons like Vision and Dali can't do.
I want to see that.
I don't know.
It'll be fun.
There are other really impressive things in the demo as well in the chat interface.
So they showed, it was a chat-based, you know, GPT, chat-GPT style interface, a little barred,
call it whatever you want, where the interface changed because the model itself created new
UI elements in the chat.
So the example it gave was somebody uploading homework with four maths questions on it.
And they've handwritten the responses.
They've taken a photo of the paper, uploaded it, four questions in these blocks.
And then the chat says, which one of these do you want to discuss?
Click on the option.
And it's taken the photo and put on interactive clickable elements on top of it.
And then that allows you to interact with it in that way.
It also showed a demo where very similar to the copilot, I don't know what it is about
the big tech companies, that they love using birthday party planning as a default use case
for using chat LLMs.
But they talked about planning a birthday party for someone and trying to come up with
a theme and ideas.
And the chat does that, comes up with these ideas, but then brings up, it's almost like
in the, like when you do a Google search, it's got the image across the top or the box
of images across the top and you can scroll through it.
And then you've got the knowledge graph at the side.
So it kind of designs in real time, these different UI elements that you can interact
with to say, oh, I like that idea.
You've told me I could have a dinosaur theme for my kid's birthday party.
Click on that and then it expands the chat in a new way and brings up more elements.
So that was really interesting because that's all supposedly done in real time on the fly
by the model itself.
That is interesting because I imagine more of worlds where we speak to computers like
you and I are speaking now and then they do stuff for us.
Mostly because I've been conditioned by Star Trek to think that that's what the future
looks like probably.
But also there's an element of that's how humans interact.
So would that be the best way to interact with a synthetic human or an AI computer?
But I wonder if we're seeing some hints there as to where they think their web-based experience
that we currently call search might evolve into this sort of dynamic interactive tool
that over time they try and find clever ways to put ads in amongst it.
Because you know those generative overlays that you were talking about and in some areas
I think one of them was like a person's making a cake and they're like I want to see cat-based
cakes and a bunch of them come in.
You can scroll through them.
What if every fourth one of those is some sort of sponsored content or an ad to buy
the necessary materials to make said cake or I don't know right.
So yeah I think the jury's out.
I like to be, last couple of weeks I played the role of naysayer a little bit on the podcast
but the truth is we need a model in town that's as good as GPT-4 to keep innovation flying
out the door at open AI.
If you're a user, if you're a tech optimist right because they need that pressure on them
and I think now they have it.
The reason I like to bash Google a bit is they've had and continue to have some of the
brightest AI minds in the world and not like they kick-started this transformer you know
generative pre-transformer revolution and I just I want them to be in the game because
I think they create an ecosystem that moves at speed and that good or real that's kind
of what I want.
I want the ecosystem to move at speed provided you know the terminators don't descend on
us and or turn the whole universe into paper clips.
Well they're very clearly putting the heavy hitters on this because obviously Demis Asabis
is working on this Sundar Pichai was very much involved but also co-founder Sergey Brin
was actually named as a contributor on the paper that they published and I heard someone
talking about they were at an event recently where he was he was there and Brin was speaking
to some AI developers talking to them about some of the decisions that they made with
the development of this particular model but getting really really granular in the detail
like a level that you'd have to be really fully immersed and all over the detail to
even know to ask that question.
So I think that he's been actually coding some of the model.
Nice!
Well I think the summary here for marketers is there's a new game in town and there's
going to be two parts to this game. The first part is we're going to have access to a GPT-4
like model which has got some super skills. It's not super clear yet how we're going to
leverage that but if you think about Google Duet and these other tools that can access
your emails and your files one of the major criticisms we had of those tools including
Bard is that the model that underpin them just wasn't good enough to do anything interesting.
It got too many stuff wrong it was hallucinating all the time whereas you can imagine if you're
a Google workspace company and your Bard and your Google Docs integration and your Google
Sheets and your email integration is powered by Gemini Ultra now we're looking at some
superpowers right so that could be really interesting and a true competitor to Copilot.
So 2024 is going to be interesting for businesses and how they leverage generative AI across
their software suites especially regardless of if they're a Microsoft customer or a Google
customer they're going to have an AI model about the same level that's what it looks
like isn't it. Cool let's talk prompt engineering. So recently Microsoft has been exploring prompt
engineering and revealed some significant advancements in the capabilities of the LLMs
that we know and love like GPT-4. At the heart of this discovery was Medprompt which was
a strategy that significantly enhanced the performance of these models for specific tasks.
So the Medprompt it was initially developed for medical challenges as they probably suggest
but it's proven its versatility by achieving breaking record breaking scores on the MMLU
benchmark which is a test spanning diverse subjects from maths to history. So this is
a series of medical challenge type prompts that are now doing well in other areas by
fine-tuning the prompts given to GPT-4 researchers could steer the AI to understand and respond
with greater accuracy and depth. This method involves combining simple and complex queries
allowing the model to integrate multiple responses and choose the most confident answer. I know
you saw this this week Martin what was most interesting to you about this story? I think
some of the developments around also the techniques that they use in the prompt engineering to
get the outputs here are really interesting. They've effectively described three different
prompting techniques that you can use in order to improve the reliability of the outputs.
So some of them we already know about one of them is I'm going to keep these fairly
simple because they're slightly more technical but one of them is chain of thought prompting
which is the technique of asking the model to explain his reasoning before it kind of
gives you the answer. So talk through its steps equivalent to asking a student to show
it's working. That's something that's been around and been known by researchers for a
while. Another one of the techniques was to where we had few shot prompting so the idea
of in your initial input prompt you give it a couple of examples or a few examples of
the sort of response that you're after or what what good looks like. So you might put
here's you might be a question and answer and you give it several examples of questions
and answers in the format and it will stick to that. Well they've taken a step further
and created something called dynamic few shot prompting which is slightly more technical
and requires the use of an embeddings model in order to be able to identify the best examples
for your few shot prompts but that was a great technique for improving the quality of the
outputs. Yeah and then there was a third technique which this one's this feels like a slight
cheat right but it's basically where you repeat the prompt several times and then you get
the answers from the outputs look across those for the ones where the answer is the most
consistent and that's that's your kind of winning response as it were. So it's not like
you're just asking it for one output you're you're basically running the the prompt several
times and where you notice that there is a consistency in the output that is the output
that you should be going with. I did read a research paper that I am not qualified to
understand it will probably do a terrible job of explaining but one of the things that
I learned was when a large language model has the ability to theorize what is the likely
best answer so in other words if behind the scenes it can come up with five or ten answers
to your query and it has some model or mechanism for ranking the quality of those answers and
then choosing the one that's the most appropriate the quality of the answers that you get is
going to approve in general so it sounds to me like a way of achieving that but without
actually having to have the model do it in the background because you're basically forcing
it to do it and so definitely there was something in that paper about outlining the logic about
why that would work so well. I think it's interesting this story as well because they
used this med prompt strategy with GPT-4 didn't they and its score on the MM-LU went up to
like 90 plus percent which was better than Google's Gemini Ultra having previously had
it reported that Ultra had beaten GPT-4 and it does make me reflect when I look at a lot
of the scores and a lot of the benchmarks it's like GPT-4 79.6% Gemini Ultra 80.1 and
it's like how statistically relevant is that difference and when we see stories like this
where the quality of the prompting increases the score you get on the test more potentially
than the model are we in a law of diminishing returns in terms of once we're at 88% and
we get to 90% and then we get to 91% how much better and how much more usable is that? I
don't know because I'm not an expert in this but I think it's a good question particularly
at the front end for the user because these techniques that it's talking about here you
can't really do through the likes of ChatGPT you can do them through the API in clever
stringing together there's some engineering and pipeline workflow elements and kind of
sequencing of the prompting that you need to do in order to get these outputs I don't
think you can just run it in ChatGPT and get this out so for most users this isn't really
super relevant yeah it's I think the thing I take away from this is GPT-4 was good Gemini
Ultra is probably good but until they figure out how to better understand user intent the
quality of the prompt is still going to be quite an important part in terms of getting
a good output and I think there will continue to be clever prompting techniques that can
enhance your ability to get a good result some of which are completely unexpected and
nobody knows why they work like our next story actually Martin on how to make Claw 2.1 a
bit better not a bit better actually a lot better a significant amount better again simple
prompting technique so this was a paper that was published by Anthropic following the announcement
of Claw 2.1 and that massive 200k context window which is about 500 pages of text so
with all of that context available to you you want to be able to have reliable recall
of the information from within that context and that's something that has proven to be
quite inconsistent Gregory Cameron who was a large language model enthusiast has created
this test it's a needle in a haystack test where he inserts a random sentence somewhere
into the middle of a long document and then asks the model to give him an answer that
relates to that that random sentence the team at Anthropic used that test to try to see
the recall capabilities of this 200k context window with Claw 2.1 and they found and this
is just bonkers right so in the in the base test so with no clever engineering just trying
to run this test the model's performance was 27 percent accurate so really not very good
at all like barely better than one in four times it was getting it right however if they
did a simple prompt adjustment they basically added to the start of the Claude assistants
response so they made the Claude assistants response start with here is the most relevant
sentence in the context colon and then got it to retrieve the answer performance improved
to 98 percent accuracy it's an astonishing turnaround so even at the basically performance
drops off as you put that sentence that the kind of mystery sentence further into the
document and you put it into the middle it starts to get really really bad at being able
to effectively recall that but just putting that at the start of the prompt was a absolute
game changer in terms of performance yeah it's it's fascinating because so to like put
this in some context for practical usage one of the awesome things about Claude that's
better than any model in my hand certainly even better than gpt4 is summarizing documents
but that can include company earnings reports it can be long call transcripts for internal
calls or customer research interviews or you know whatever you can think of and if you
can't trust the outputs of those summaries well then you've got to go read the thing
yourself and definitely i've learned i've seen Claude 2.1 be significantly better when
i give it a random call transcript and ask it to summarize the key items from the call
and the actions it does way better but as you said martin it's showing that for very
long bits of information it's good at remembering the beginning and the end but it doesn't it
can't find things in the middle and remember to output those so if a simple one sentence
prompt can have that type of improvement there's practical utility right there in terms of
being able to trust it with those types of applications if anybody wants to play around
with this the Claude chat bot is not the way to go i think try using it in the in the API
console because within the API console you can actually edit the assistance response
so you can put at the start of that here is the most relevant sentence in the context
and make sure that it always starts this response with that and give it all of that context
beforehand yeah talking about the API and the console
Claude have recently released their kind of cool Claude for Google Sheets well i think
haven't they so you can a little bit like some of i think we've spoken about it on the
podcast before there are some Google Chrome plugins floating around that allow you to
basically prompt different models GPT-4 and Claude is included as cells in a spreadsheet
so why is that useful well in back in the day we don't do this anymore but for an example
Martin and I used to copy paste the URLs of interesting stories that we found through
the week into a Google Sheet in column A column B would automatically format a prompt about
those stories and then column C would actually send that prompt to Claude and have Claude
write a summary of the story for us to use as inspiration for creating the script for
the podcast and of course you can then automate and do clever things off the back of that
but now Claude's basically anthropic of made a tool dedicated to doing that with Claude
is that right have I understood that right Martin yeah that's exactly right and they've
given some examples of Claude for Sheets prompting so you can use it for long-form document Q&A
information extraction you can use it for removing personally identifiable information
so if you've got that in a spreadsheet we'll take that out and and edit it out a customer
support chatbot using FAQs an academic tutor prompt chaining they even have function calling
capabilities now as well which they announced as part of Claude 2.1 and there's a whole
resource available online they've got a Claude for Sheets prompting examples workbench where
you can actually go and give it a give it a whirl so if you are looking to get a bit
more gangster in your AI usage and you're trying to think about clever multi-step sequences
or processing lots of information in one go but using the power of AI it's probably worth
going and having a little play and understanding what this new Sheets capability can do right
next story we're going to be talking about Meta's AI image generation tool so Meta's
getting in on the image generation with their imagine with Meta AI tool which utilizes an
image synthesis model trained on over 1.1 billion images from Facebook and Instagram
which is not an insignificant amount of data and of course why we look at a lot of the
companies outside of open AI who have huge treasure troves of data to fuel training their
models on and this is a great example of Meta stroke Facebook stroke Instagram doing exactly
that what the tool will do is it will allow you to generate unique images from written
prompts which was a feature that was previously embedded only in social apps like Instagram
it's kind of interesting in that the models been trained on a vast array of sort of public
social media images and that's a bit of a little bit of an ethical minefield what are
your thoughts on that?
Those terms and conditions sheets that we've all signed up to when we signed up for Instagram
and Facebook all those many years ago and the idea that you know Facebook can use our
data if it wants to well we're now seeing the the fruits of that terms and conditions
authorization I guess yeah so have you used it it's available only in the US but is available
via VPN have you given it a try?
I haven't played with it directly but I've seen a number of side-by-side comparisons
where people feed the same prompt into Dali 3 mid-journey Meta why have you played?
I have and yeah I'm impressed I think it is one of the better models out there certainly
for a first release from a company you know this isn't V3 this is their kind of opening
gambit on the market they've done a really solid job I prompted it with the northern
lights in the skies above the city of Derby and I chose the city of Derby because no other
model can get you know there isn't loads of photos of the AI blast skyline right yeah
you know like you know parochial provincial towns right they're not well known for their
vast quantity of training data in these things and actually did a really good job to the
point where when it generated the images all it generates four at a time and all four of
them were very very similar but I immediately went oh that nearly tricked me into thinking
that skyline was was Derby but I've used it for other things you know the kind of classic
panda riding a bicycle kind of thing and yeah it does a really good job I think they've
absolutely nailed it I think it's supposed to be particularly good for you know selfie
style images as well which wouldn't be a massive surprise given that it was trained on Instagram
right is it what's this photo realism like is there any good yeah I haven't gone I haven't
gone deep into the the different styles um I think there's a way that there is there
is I haven't seen it be truly photorealistic um which I think almost has gone from Dali
3 that capability now there's there's very much a style I think with with Dali 3 it has
its own aesthetic which has been impressed upon it it almost seems like no matter what
I ask it for I'm getting a very similar look and feel but with this model I think they've
done a really good job I can't get photorealism out of Dali 3 for love nor money now it's
um I wonder if a little bit of it's the deep fake aspect right like it almost avoiding
being photorealistic so it's just five percent off looking real so that anybody who looks
at it is going to go well that's AI generated yeah I can only I can only really get good
stuff out of mid-journey still yeah and for photorealism I still think that the images
that you get from Dali are very good I've created some fantastic images on there recently
that I've used in various applications um but if you want photorealism there's that
not even uncanny valley it's just there is a there is something that looks like it's
computer generated it's something that's slightly softer yeah I think it's softer and there's
a there's a feeling of lack of imperfections I can't really describe it like the people's
faces don't look like photos they look like what I would imagine a computer game in 2050
looks like right like the computer generated images on it are so good as to almost look
real but without the imperfections that real humans have not on this podcast obviously
but the other humans yeah the rest of the 8 billion out there slim oh crumbs funny that
would trick um there's some other cool stuff on the AI image meta Instagram Facebook Mary
yeah and um what we're seeing here is that they're starting to incorporate some of the
models that we've spoken about in previous episodes and they're launching these into
consumer applications so they've just announced a new feature on Instagram called backdrop
it's available only in the US at the moment again those of us in the UK and Europe will
just have to rest our souls and patience this model takes the segment anything model which
we talked about a while ago this is basically a computer vision model which can basically
it does like clip paths around object identification but it can do that at a really impressive
scale and it uses that to remove the background and then you can put in any background that
you want so it was similar to what we've seen on things like clip drop and you know back
any kind of AI tool that uses background remover but they've incorporated the segment anything
model and a model called emu so yeah this is the actual rollout and commercialization
of some of their research tools into the Instagram platform yeah I quite like that because we
did talk about that segment anything model I don't know six months ago maybe I think
yeah yeah so what that would imply is that it could take six months to see some of these
research papers make it into products which is not that long really when you think about
it as an ex-scientist I get particularly excited about the segment and anything although whether
it can really segment stuff in biological images like images taken of cells or you know
biopsy samples is another question but a major part of biological research and pathology when
you're doing analysis of a you know like a cancer biopsy sample is image analysis and
segmentation where you have to highlight the interesting part of the image which I you
know there have been AI tools that have emerged and they are really rather good but when I
did my PhD many years ago you had to pretty much do that manually which took hours I think
I've definitely spoken on LinkedIn about taking pictures of lots of fruit fly eggs with fused
dorsal appendages and having to count them by hand and spend a good part of about 10
to 14 days 10 hours a day at the microscope so you could probably automate in about two
hours but it's kind of cool because we talk a lot about marketing and sales and I know
most of the people who are listening they love marketing sales and that's why they're
listening to the podcast but well I do get excited about some of the image analysis capabilities
that some of these tools are going to enable.
Next story is about looking at which is more is capable of running more persuasive sort
of advertising campaigns humans or chat GPT so in a recent study by MIT where they partnered
with a top consulting firm they were trying to figure out who could create the most persuasive
content and there was also a test where it was like AI only humans only or AI and human
working together and the research focused on creative content for five retail products
and five campaign topics using both professional content creators and churchypt. Over 1200
online participants evaluated the content and they were assessing it for things like
satisfaction willingness to pay how interested in it they were how persuasive they found
it and they were categorized into groups with varying levels of awareness about the content's
origin so whether or not they knew it come from a human or an AI or from the two working
together and the key findings were that when content was either created solely or ultimately
determined by chat GPT it actually scored higher in quality across satisfaction willingness
to pay and persuasion metrics compared to purely human generated content so that's interesting
finding number one. When participants knew the content was made only by humans they perceived
it as being of higher quality very what's Elon Musk statement very very no not Elon
Musk Elon Musk was the species wasn't he was very pro-human I'm pro-human as well but it
shows the rest of us are quite pro-human when we know humans have made it we think it has
higher quality although it's fair it's worth highlighting that when people knew it was
AI it didn't diminish the quality of of what people felt they saw it was just if they knew
it was humans they thought it was better the quality gap itself was actually narrower between
AI and humans for product advertising than and campaign messages which I think probably
makes sense and despite a general favoritism towards human generated content for both products
and campaigns there was no consistent aversion to AI included bits and pieces so that's pretty
interesting I think it fits in with this whole is creativity the last bastion of human endeavor
and this is another potential piece of data that says working together we do the best
stuff what do you think Martin? Well it certainly goes to show that AI naysayers the people
who say oh you can't write good copy shows they're wrong but I think we've already seen
this with other research studies that came out recently about the creativity of GPT-4
that is actually it's as creative or more creative than most people and I think that's
it's probably not as good as the in like in this example for instance I would love to
see the the best copywriters in the world pitched against GPT-4 I would well imagine
that the best copywriters are better still but against the average kind of copywriting
team or marketing team GPT-4 can perform as well or better than those but again doing
it in conjunction with humans is where we see strong results so I think there's nobody's
going to be completely out of a job just yet in fact there was a something I read this
week where some big industry AI heads were were saying that 80% of 80% of jobs will be
impacted by AI but no jobs will be replaced entirely and it's just going to change the
way that we all work and it augment the way that we work right? Yeah that definitely fits
with my sense I mean when you and I are out and about speaking at conferences we get an
asked a version of this question pretty much all the time yeah and lo and behold I was
asked several times when I was at SAHPS last week and I think what you touched on is very
much in line with a model that Ethan Molek talks a lot about we talk about Ethan a lot
on the podcast I think it was related to the paper they published where Boston Consulting
Group had two groups of consultants one used GPT-4 and one didn't and in general the team
that used GPT-4 on this fictional project they were working on were faster more efficient
and there was a higher quality of their outputs but when you dive a bit into the data it's
almost like it squeezed the normal distribution curve of capability and what I mean by that
is if you're a super expert you probably didn't see your quality increase that much when you
had access to GPT-4 but if you're in the middle of the normal distribution curve or perhaps
let's say even a slightly weaker member of the team it helped accelerate and improve
the quality of your output significantly so I do get the sense to your point about how
would the master copywriters get on against GPT-4 I think they would absolutely smash
it out of the war to be honest but I think you're right I think it can help people who
don't have 30 years of experience as ninja copywriters you know 10,000 hours or whatever
it is they say you've got to have of just refining your craft maybe with only 500 hours
you can do the output of someone who's had 8,000 hours worth of training which to a certain
extent certainly in the short to medium term is going to be really really valuable I think
if you model that out further I think the super duper uber experts become even more
valuable because now we've got this nice fat middle where everything's much better than
it used to be but then now the bar the average is higher and so to stand out above that average
you need the ninjas and I think it's those super duper copywriters that will continue
to get paid the big bucks because they'll be able to produce the quality of work that
even all the rest of us with GPT-4 in hand just won't be able to do so if you're listening
to this and you're thinking what should I do in the realms of the AI world firstly get
to grips with the tools learn about them test them figure out how they can help you be better
at what you do and then continue to invest in developing your domain level expertise
to an absolute uber area where it will take well hopefully a long time for any AI model
to be really really good at the thing that you've spent ages getting good at because
those models are still somewhat generalists but also don't be surprised if the model catches
up and completely crushes your dreams within six months because that's the brutal reality
that we all now live in so I don't know if you saw but I was looking at you Martin to
say don't say that Martin don't say it yeah okay I do think it could be true but it is
Christmas time and we're trying back in my optimist optimistic approach because it could
be three months yeah it could be our Gemini Ultra came out crumbs we saw a rumor this
week we were going to talk about this but I think we should there was a rumor about
GPT 4.5 floating around the Twittersphere wasn't there Martin and all the different
things that it could do video 3d the 3d I didn't even say 3d what one assumed some sort
of 3d imagery or can it create maistime continuums like who knows what's going on the cost per
token was astronomical I don't know how anyone could run a business off the back of it and
but we think it's fake don't we mine but if GPT 4.5 came out and was like even better
than Gemini where would we be and when I was like when you send that to me and I was looking
at I was trying to think there's a big challenge at the moment about defining AGI right advanced
general intelligence a computer that can do most things better than a human and I read
a really interesting take on this that if GPT 4 if chat GPT as we will use it today
came out 10 years ago we'd be calling it AGI which I thought was an interesting and provocative
point right it's like the more sophisticated our tools get the higher we keep raising the
bar of AGI to a bar where I feel like the new bar is nobody has to work ever again that's
why I feel like the AGI bar is well which actually chimes somewhat with Sam Altman's
interview with Time magazine this week and actually in that he talks about the next the
next steps for AI in terms of productivity and well people having more time and more
freedom to be able to do what they want because he's you can see that they're clearly thinking
at OpenAI where this is going is it is going to well agents when AI can really effectively
execute tasks and everybody goes from having chat GPT which in this interview he describes
as I can't remember his exact phrase but it was something along the lines of it's a bit
rubbish to and he's like you know fast forward a year or two's time everybody's got 10 AIs
doing tasks for them doing work for them in their pocket 24 hours a day 7 days a week
to 5 years time when everyone's got hundreds of AIs doing hundreds of tasks for them 24
hours a day 7 days a week and what does that allow you to do from a personal productivity
perspective and what does that do to the economy?
Yeah that is I think that's the thing that keeps I think that's probably what keeps people
up who are in these circles more than AI accidentally turns the universe into paper clips because
it's been told to make the best paper clip but wasn't given any common sense briefing
instructions and so just keeps churning them out until the universe is all paper clips
or terminator right I think it's how does it affect the economy when the main force
of the economy is still human labour we're still paid to work what happens when you unshackle
output from human input.
Ilya Sutsgiver in an interview a few months ago talked about economic destruction that
was that was the phrase he used with AGI that's all he's used sometimes though you know like
if there's a if there's a fire in a forest sometimes that's needed because some of the
dead trees you know they're destroyed it gives a chance for the saplings to grow up and the
forest is refreshed and it probably takes some time but ultimately it's a can be a productive
and natural process so I think it's whether or not he means doom and gloom destruction
or creative destruction yeah but it'll be definitely like the ways that it wipes out
the dinosaurs right and allow the conditions for mammals to flourish without which we wouldn't
be here today so maybe that's the exciting future we've all got to look forward to no
more dinosaurs are you a dinosaur Paul?
Well that dinosaur turn of phrase could be used in another way couldn't it in the in
the rise of AI but I'm sure when robot Paul and robot Martin are talking about how sadly
the humans made way for the more evolved efficient effective robotic life forms that they gave
birth to I wonder if they'll if they'll feel bad about the humans like do you ever feel
sorry for the dinosaurs like oh bless them they got wiped out by an asteroid but we did
all right out of it yeah well I mean they'll just turn into chickens
well not all of them but yeah the yeah look ultimately progress is progressing as it is
and without we get too deep into the weeds of where this all might go because I don't
think anybody really knows either that's the other thing all we can do is marketing professionals
and sales professionals is keep doing the best we can to have an impact enjoy our work
if these tools can help with that then you know better to be in the gang figuring out
how to use these tools I think than sticking your head in the sand
should we move on to our last couple of stories?
Yeah one that is very much rooted in the world of people who know a thing or two about marketing
that is McDonald's and Google put our press release basically announcing that McDonald's
is going to be using Google Cloud as its main cloud provider and whilst the press release
in and of itself isn't particularly interesting in the headline it did talk about generative
AI so that's what turned me on to it and in the press release they say we're this new
partnership a dedicated Google Cloud team in Chicago will work in close proximity to
McDonald's global innovation center known as Speedy Labs together they'll focus on applying
generative AI across a number of key business priorities to power exciting new experiences
for crew and customers with McDonald's unmatched convenience and value.
Now we know McDonald's likes data and digital innovation when they introduced their self-service
kiosks they were reported to have increased order size by 30% and order value by 20% and
I heard something the other day that said they found that men when using kiosks about
20% more or something order two burgers as opposed to one because they don't have to
say to someone at the desk or gym because they're thinking that person's thinking I'm
a proper fat they're judging me they're judging me yep and so that increases average spend
so we know that they're they're into their their data and how digital can transform the
customer experience it got me thinking when you've got those kiosks and those digital
displays and people are using the my McDonald's reward to connect their data in real time
to this digital display so the display knows exactly who you are knows everything about
you is hooked up to the cloud the potential for generative AI to come up with personalized
interactions there is really quite remarkable you can imagine it from personalized menu
options which in and of itself isn't particularly exciting to avatars that you know almost like
a hey Jen style avatars that will take your order in real time whether or not people need
that and actually just tapping on the burger that you want and you know hammering the plus
sign till you get to the relevant number of burgers that you desire in your in your takeout
whether that's the way to go about it or not I don't know but yeah just made me think about
the potential here this is obviously a big deal right McDonald's has huge amounts of
customer data on their my rewards at plug that into Google cloud and bring that down
into a digital interface where people are actually spending their cash you can you can
see there could be quite the uplift in average customer order yeah it's interesting the first
thing is thank you for the strategy to go and order more burgers without feeling like
I'm being judged because that was an application I needed in my life and I'm definitely taking
away from this conversation the other thing is I think on the podcast we try to go into
as many first order impacts of things as we can and occasionally we get a bit deeper into
second order impacts right but the smart people are playing these things out right they're
like well if this enables this and this enables this and this enables this how do they all
come together to impact my business right if you are McDonald's how does the ability
to leverage all the data you have on customers combine with synthetic humans combine with
predictive modeling to know exactly what type of thing people are going to order combined
with tests in using generative AI tools that make the outlay of the screen and everything
ever more persuasive to get you to buy more like how do all of those things come together
to influence your business and like I say there'll be people smarter certainly than
me modeling those out and I think they're they will potentially end up being the unexpected
success stories of where AI gets used in a way that we probably as marketers we're thinking
yeah I can write blog posts and social posts and now I can turn Martin's voice in from
his voice into a strict older woman but what how do these things does it call back to a
previous episode as well so if you think what the heck is he talking about that you'll have
to listen to episode 35 I think it is but I think it's going to be the ones that figure
out how to connect that stuff to do unexpected things that drive a lot commercial value and
perhaps that's what McDonald's play is here.
Multivariate testing right something where you can run lots of different examples of
an interface or a scenario or what have you and you see that they do market testing of
different burgers across the different country right to see what's going to land well well
and you can have as we discussed earlier Google Gemini is able to in real time dynamically
update the user interface element that applies to that kiosk where it is doing multivariate
testing on names of products or combos of meals dynamically creating different mix and
match menu items or what have you and it's doing this all autonomously doing you know
A B testing market testing real time on steroids like never seen before that's yeah that's
going to be more impactful on business than me speaking like a strict older woman.
Depends what in what context that is you but I agree I guess yeah I do agree with you and
honestly if I got if I chopped off to McDonald's moving on if I got to McDonald's kiosk if
I had facial recognition and it knew I was and I didn't have to press 18 button presses
to get the meal that I order 95% of the time I know some people would find that a bit like
invasive but personally I'd be like brilliant just saved me eight minutes of pressing the
wrong buttons just pop up synthetic person says hi Paul how you doing do you want yeah
do you want eight burgers three chips and 16 tubs of mayonnaise and I'm like yeah just
like last time the family they're all yeah the extended family they're all they're just
yeah so yeah I'd be up for that but yeah I think you're right that multivariate testing
would be be interesting anyway we better move on so at the hour mark give or take and we've
got one more story it is about an open source model called Mistral so Mistral is a French
company that received another round of funding this week I think they're only about six months
old but the latest round of funding puts their valuation at the two billion dollar mark so
they're another example of money rushing into the AI space into certain areas but why are
Mistral interesting well they're interesting for a number of reasons but this week they
became even more interesting when they released their new model which is mixed raw eight times
seven B which is a mixture of experts model so it's pretty cool because it's open source
which will come on to why that's important in a moment but just as importantly it has
beaten the likes of GPT 3.5 and met as Lama Lama 2 on a number of benchmarks so now you've
got a free open source model that doesn't come with all of the limitations that comes
with all of the reinforcement aligned models all of the weights of the models are all locked
down you can't change them that you get with say GPT 3.5 but you can now do that with mixed
raw eight times seven B it's also interesting that it is a mixture of experts model which
is what people think is the underlying architecture of GPT 4 so rather than just like making a
bigger and bigger and bigger and bigger and bigger model that the mixed raw eight times
seven B has this mixture of experts model it's also got far fewer parameters than GPT
3.5 I think it's about a third isn't it Martin so yeah 46 billion parameters compared to
about 175 billion parameters for GPT 3.5 so you get that level of GPT 3.5 performance
but about 25 30 percent of the model size this is important right because model running
large models is expensive storing them is expensive if you ever want to be able to run
a half decent model on your phone those model sizes are going to need to come down so this
is also an example of what happens when you come up with clever ways to train really good
models but you compress the information even more so the models are smaller and therefore
a bit cheaper to run the I think the critical aspect of this is it's open source so you
can run it on your own machine there are still lots of concerns about what happens when you
share data with open AI Microsoft Google like everybody's happy sharing their data with
their company until something bad happens and they wish they hadn't so for those that
are super paranoid about that the ability to get ever better model performance running
on say your own laptop even if it's somewhat slowly unless you've got a super dupe powerful
PC with cool GPUs in it or a nice powerful Mac it's going to give people access and open
up use cases that you just can't get with the other models so kind of cool.
It is very cool and I'm always interested to see where the open source community is
going they had another bonus recently and so much as the EU's AI Act regulation isn't
going to cover open source models in the same way that it's covering the closed models so
yeah open source is having a bit of a good time at the moment they're seeing some really
impressive results there's money being invested into the space and obviously with big names
like Meta contributing to the field it's definitely a place that businesses that want to deploy
their own models will want to watch that there is the critical factor there companies that
want to deploy their own models the use cases for these at the moment for many organisations
it just isn't there so these feel like emergent there'll be some interesting developments
in the coming months and years I'm sure but I think lots of organisations will first and
foremost go through the APIs of models like Claw like OpenAI models because they've got
the power and the heft and they can do what they want out of the box without having to
do more technical jiggery-pokery yeah this is yeah I mean this is kind of yeah this is
mostly still developer interest than business interest isn't it but A will start to see
capabilities emerge that are easier to achieve with the open source models that are hard
to achieve probably with the likes of Gemini and GPT-4 and it's just further evidence of
the amount of innovation that's happening in many areas of models with now Google in
the game Meta's got its new image model that have come out the video models are improving
quickly we've got open source models chasing all of the commercial models and keeping them
honest and keeping them developing their technologies so everything is continuing to race ahead
at speed and the irony was we were going to make a joke about those of you that are waiting
to sign up for chat GPT plus and still waiting for Gemini Ultra to come out that maybe the
best model you could get your hands on was actually a free open source model for Mistral
but for those of you who've been trying to get chat GPT plus and didn't see the news
this week they opened up subscriptions again as of the 13th of December so if you've been
waiting to get one of those now is the time to go get it and with that Christmas present
for those of you that wanted that model and just haven't been able to get hold of it I
think we shall say have a wonderful Christmas break we're probably going to do a special
edition episode just before Christmas but we'll get back to reporting the news and having
all of the interviews that you've known come to know and love starting at the beginning
of the year 2024 next year have a fantastic Christmas break Martin.
Same to you.
And I will look forward to speaking to you soon.
Merry Christmas everyone.
Thank you for listening to artificially intelligent marketing to stay on top of the latest trends
tips and tools in the world of marketing AI.
Be sure to subscribe.
We look forward to seeing you again next week.