Welcome to Artificially Intelligent Marketing, a weekly podcast where we stay on top of the
latest trends, tips and tools in the world of marketing AI, helping you get the best
results from your marketing efforts.
Now let's join our hosts, Paul Avery and Martin Broadhurst.
Welcome to episode 41 of Artificially Intelligent Marketing with me, Paul Avery and my lovely
co-host Martin Broadhurst.
How are you today, Martin?
I am on top form and I'm just trying to stay on top of it.
By it, I mean just this deluge of AI updates, which is great for us because we get to record
yet another episode of the podcast, but it can be a little bit much.
We broke our phones last night bashing WhatsApp and for listeners who have been with us for
a while, you'll know that we tend to record about every two weeks these days, but we had
an episode go out on Monday, it's Friday today, and we had to do another episode even though
our last episode only came out like four days ago because this week has been crazy.
Yesterday was super crazy and there's just loads of really important stuff to get through
and so a little bit of a bonus episode in some ways.
What does that mean?
It means that this is coming to your raw listeners.
Martin and I have not had our usual debrief and sort of bashing back and forth of figuring
out what different stuff means.
We haven't written many scripting elements like we normally would.
It's just going to be a good old dynamic conversation as we try and make sense of the massive news
this week.
What do we mean by massive news?
We mean that Google released Gemini 1.5 Pro, so the next version of their Gemini model,
not the Ultra version, the Pro version, but now the Pro version is better than the Ultra
version.
We'll get into details on that later, but this is pretty impressive considering they
only started telling us about Gemini in December.
Then not to be outdone, OpenAI announced Sora, which is this crazy video generation model
that somehow understands world physics and we'll get into what that means.
Those are the two sort of big massive bits of information, but mixed in with all of that,
which probably would have been big news in any other week, Stability AI released a new
way of chaining different models together to produce images, which has a load of benefits
and potentially new applications, which is really interesting.
Amazon released the largest text-to-speech model yet that shows some emerging capabilities
and we'll look into those.
And just because that's clearly not enough news, there's rumours circulating that OpenAI
are moving into the world of search and trying to come up with a product that would rival
Google search, perhaps a little bit like what Perplexity does, but we just don't know.
So that's what we've got to try and get into today, Martin.
Crazy.
Just a couple of stories there.
Nothing of the major, right?
Right.
Let's go first and let's talk about Gemini 1.5 Pro.
So yesterday, this is Friday in the UK, it's 1.25pm.
Last night, Thursday night in the UK, well, Thursday afternoon-ish, Google announced Gemini
1.5 Pro.
So in essence, it's the next generation of their Gemini models, but in benchmarks, it
compares really well with Gemini Ultra, which was their super duper model that they just
released like, I don't know, a week ago.
It uses the new mixture of experts model architecture that we're seeing.
This is what Mistral's models use as well to improve training and make it easier to
offer the surface because you don't run the whole model for every query.
You use this mixture of extras model.
Perhaps most importantly, it has a breakthrough context window of 1 million tokens, which
for context, the biggest up until now was Claude, which had 200,000 tokens.
And we'll go into in a bit more detail what token and contents length means in this context,
but that's a massive jump of five times against the current industry best.
And in research applications, it had a 10 million token context, which is mind blowing.
And we'll talk about what that means because in essence, you know, now it can process fast
amounts of text, video, audio, images, code all in one go.
It is obviously outperforming the original Gemini Pro.
And we should expect to see this starting to be released soon because they have already
allowed developers and enterprise customers into preview mode to be able to actually play
with Gemini 1.5 Pro, mostly using 128,000 token version, but the 1 million token version
will come later.
So there's all like this sort of bits and pieces you need to know about this.
Martin, tell us what does this mean?
Well, let's start with that massive context window.
What does that enable us to do?
Well, we can think of tokens as bits of words, or they are actually, they're not bits of
words, that's only when it's the text based modality.
But a token is the mathematical reference for any bit of data that the language model
understands.
If you get a word, that word will be chopped up into tokens.
Typically, like 100 tokens will be about 75 words.
But it's not just, as we're seeing with this multimodal version, it's not just words that
get turned into tokens, it's images, audio, video, and the rest of it.
So having a larger context window basically says that the model can handle that much more
information for you to interrogate, converse, provide, go back and forth with, etc.
Yeah, it's 700,000 words of text, 30,000 lines of code, 11 hours of audio or one hour of
video, and that's for the 1 million context.
So you can times all those numbers by 10 for the 10 million context window they were able
to achieve in sort of research applications.
So a lot of stuff.
Yeah, and to, you know, that many words can sound like, well, what does that equate to?
When Anthropic announced the 100,000 token context window, which was about a year ago,
and I find this mad because we went from, I think it was about 8,000 tokens, it suddenly
jumped up to 100,000.
Everybody kind of lost their mind going, oh my God, there's so much that you can do with
that.
But when they announced that, that was described as like a novel.
And now we're talking about 750,000 words, 10 novels.
And that's, yeah, it's crazy.
The more interesting thing is what they've announced it can do with that though.
There are some people that have been doing benchmark tests on ability to recall.
So if you, a big context window means you can throw in a book and then can it actually
extract information from that?
Otherwise, if it can't accurately tell you what was in the book and it's hallucinating
things then, you know, really what's the point if you can't rely on it?
What they found is that it's recallability, it's needle in a haystack test, which is basically
where you plant an obscure bit of text or ask it to find one bit of detail about the
information that you've provided it.
You know, like what colour was somebody's shoes or whatever.
It's something kind of a bit, not kind of integral to the body of the work itself.
It has 99.7% recall on a million tokens, which is far and away better than any other model
that we're seeing on the market at the moment.
Yeah, it's really interesting because having crawled through the research paper and tried
to get my head around, you know, what this means for marketers, that is a seriously impressive
recall rate to the tune of you could basically give it a whole corpus of information about
your business, say for a junior employee to come in and actually have a chat bot that
they can speak to about all your systems, documents, processes, sales decks, whatever.
If that was somehow encoded in that and Gemini Pro is obviously multimodal.
So it could take recordings of meetings or screen share videos of meetings, pictures
of whiteboards and basically answer questions by keeping all of that information in its
context window versus using something like retrieval augmented generation, which at least
to untrade folks like Martin and I sounds like it doesn't work that well.
So this being able to import all this information in the context window is really powerful.
I will say, Martin, I went digging the word hallucination as mentioned once in the entire
research paper in reference to Claude, not in reference to itself.
Not because it's saying that Claude hallucinates a lot, but actually it's saying Claude doesn't
because Claude will just refuse to answer rather than hallucinate, which is one of my
favorite things about Claude and still my model of choice for summarizing transcripts
of calls and what have you.
And there's a separate quote, I think it might be from Sundar Pichai, but it might be Demis
Saras, not sure, someone at Google saying the model still hallucinates.
And so it's interesting.
It has a great retrieval score, but it's still going to make stuff up every now and again,
which I wonder Martin, for all of this power, is that going to end up still making it somewhat
limited for some applications?
We've talked about the issues of hallucination in things like chat bots, customer facing
chat bots, for example.
Well, I think what you end up doing when it goes to be deployed in production in any environment
is actually you're not just deploying a raw interface with the model.
You're having to have layers where there is maybe a generation, you have your initial
inference where you have your prompt, but then that's run through a series of checks
before it presents the final output to the end user.
And these checks are basically saying, is any of this right or wrong?
The perplexity API at the moment has a kind of fact checking element built into it.
So if you were doing, let's say you wanted a chat bot on your website that was also connected
to the web, you could do that with the perplexity API.
And I suspect that's the kind of thing that we're going to start seeing is you don't just
necessarily interact with the raw model.
There's a layer of steps before the output is presented.
And Gemini already, the chat bot version Gemini already has that little Google bot that you
can click on that will fact checks and things that have been output.
So I think that's going to be interesting to see how that plays out in real world applications
as to whether hallucinating about something random you've asked the model like chat GPT
and you say, what are the 10 most important foods that dolphins eat and it makes one of
them up, right?
Versus asking it something specific about a corpus of information that you've given
it and whether that retrieval of 99.7% basically to all intents and purposes means it doesn't
really make any meaningful mistakes on information in the context window.
Because then you end up getting two slightly different use cases, right?
Like if you feed it the context and it can be extremely accurate about the context information
it's been given when if and if indeed this 10 million token window is becomes a commercial
reality which given the rate of development of all these tools, it almost certainly will
be that's a really interesting way of solving the hallucination problem.
I'd be in fact, I hope someone who listens to this, maybe we've got some machine learning
experts will message us on the LinkedIn's or the Twitter's and let us know what they
think about that because I'd be interested to see if that opens up or solidifies some
business applications that until now hallucinations have got in the way of.
Hallucinations are such a funny one because they are inherent in what the tool does, right?
All the tool is just in fact a great way for people to kind of experiment with these models
and see what's going on underneath the hoodies go to something like playground.openai.com
and start actually interacting with the raw model, not the filtered version through chat
GBT.
Because when you start interacting with the raw model and you start playing around with
things like the temperature setting, you'll see like how hallucinations come about.
So if you put the temperature setting really low, hallucinations very limited, but also
the range of things that it will talk to you about is creativity is limited.
It's just going to be really literal and not.
It's going to be very literal and very precise, should I say?
Right.
When you put the temperature up to, in fact, absolutely it's completely absurd and I don't
know why they allow you to do this.
If you change the temperature setting to two, which is the highest setting it can be, and
then run the prompt, it's gibberish.
It's pure gibberish, not even like a couple of words.
It is just like random characters.
It's bits of code.
It's absolutely wild.
So you say to it, write me a strapline for an ice cream shop, temperature two, it's just
gobbled at you.
Well, that's your problem.
Your ice cream melted.
I think that's cool because ultimately I think they're trying to show the range of influence
that the temperature setting can have, which is if it's high, model performance basically
falls apart completely.
Absolutely.
Right.
Yeah.
It is interesting.
Let's talk a little bit about some of the examples, Martin, that they listed out on
the Gemini 1.5 Pro launch blog post because we've talked about business applications and
will hallucinations still be a problem?
But crumbs, some of those examples are pretty exciting, aren't they?
Yeah.
So the one that caught both of our attention was the Buster Keaton video, 44 minute silent
movie uploaded and then interrogated with the model.
So asked to describe the plot, asked to identify certain bits of information from the plot
or from the video.
For example, there's a particular scene where somebody takes like a receipt or a ticket
out of their pocket and then that's shown to camera and it's got some detail on it about
like the business that was working and the product that was offered kind of thing.
And they asked the model to explain and answer what was on that.
And it does that accurately and it gives them a timestamp of the point where it happens in
the video, which is kind of mind blowing.
Well, you can just give it a video.
Bear in mind, it's not trained to do this at all.
It isn't trained on how to search and interrogate it.
It's just next token prediction.
I find that absolutely bonkers.
The other thing that it can do was the multimodality element of it allows them to do a very loose
pencil sketch, just a line drawing, put that in as a prompt and say, give me the timestamp
for where this happens.
And it accurately works out what the rough sketches and it is a very rough sketch.
It's the roughest of rough sketches.
And it says, yeah, this is the bit where you want to go.
So it understands what's going on in a silent movie that's 44 minutes long.
So, you know, this is no like five second animated gif.
This is a lot of frames and it understands the context, the plot.
It can read the text in it.
It can, well, I would love to see and push the limits of what that can do because again,
applications for that are quite vast in the sense that, well, you've suddenly opened up
new ways to interrogate company data.
So training videos, what could you do with that?
How can you get people if you've got manuals?
So I'm just thinking about if you've got a product and you've got some online videos
with maintenance, the idea that then you can connect your chat bot to a front end customer
service tool where customers can say, how do I do this thing?
And it can locate the exact point in a library of videos and say, this is the scene that
you're looking for.
That's amazing.
If you're a washing machine or a company that manufactures consumer goods and you can just
pinpoint people to the exact spot based on people's natural language questions or people
taking a photo of the product going, how do I fix this bit?
Or what that opens up all sorts of avenues for customer support and engagement.
Yes, I think I love the way all these convergent technologies start to influence each other
because when Meta released the Oculus Quest 2 and Quest and VR headsets started to become
at least a little bit more mainstream and maybe they will or won't with the Vision
Pro, there was sort of, futurists would speculate on, oh, where's this all going?
And people would imagine that we're all walking around in glasses that have a overlay.
We see the world, but they've got a digital overlay on it and the things in our field
of view would like pop up and tell us stuff about the world.
Like, oh, you're running out of breakfast cereal or this is the thing that's broken
in your washing machine, right?
To your point.
But a lot of the speculation at that time was about how do we miniaturize down the technology
of a big chunky headset, which of course we haven't achieved yet, although Meta's Ray
bang glasses are kind of interesting and proven quite popular because they're just a set of
glasses with cameras and mic in.
So the AI can see what you see and then speak to you, right?
It can't overlay anything in your field of view, but maybe that would still be valuable,
right?
And I think this is the point, which is at the time nobody seemed to realize that, or
I'm sure lots of people realize, but no one seems from what I could see to be talking
about the issues that come with a computer actually understanding what you're looking
at and what these capabilities now open up.
When you look at the converging technologies of VR, cameras and sensors everywhere, but
actually AI's ability to understand what it's seeing, what it's hearing, what it's reading,
that's what just opens up a huge plethora of capabilities, the likes of which I don't
even think we can all imagine yet, if I'm really honest, but it will be driven by convergent
technology evolution.
You wouldn't be able to do it without improvements in VR.
You wouldn't be able to do it without improvements in AI.
You wouldn't be able to do it without reductions in cost for manufacturing of all of these
things, but all those things together, you won't be able to do it without the internet.
The fact that we even have large language models and that we all as a human species
went diving down this rabbit hole is because we threw a corpus of information into a single
place that even made it possible to train the models.
We were talking offline about YouTube and how much data there is in YouTube and how
we as the human race have just spent the last 10, 15 years just producing all the data that's
even going to enable all of this stuff.
And I'm sure smart people at the time who were collecting it all probably predicted
it, but certainly I had no idea that this is where it would lead us.
And it's just mind blowing.
It's all very exciting.
Until we get our hands on it though, let's reserve judgment as is always the way with
these things.
I have joined the wait list.
So that's quite exciting.
If you are in the UK and you want to join the wait list, you do have to sign up with
a VPN to the AI studio on Google.
If you try and do it with a UK IP, it says, no, sorry.
I think the earliest I'm going to get to play with it is in Magi, which is a tool we've
talked about on the podcast before.
I'm doing it a disservice to say it's a skin of models of chat GPT that basically allows
you to interact with different models.
It's got a hell of a lot more going on inside it these days than that.
But what I really love about what the developer Dustin and the team are doing is they add
new models all the time.
So I can access Gemini Pro 1.0 through that system.
And he's already promised on the Facebook group that the minute that he can roll that
system out and Magi with 1.5, we'll get to play with it.
So I suspect that would be my first chance.
Talking about things that we want to play with, let's talk about Sora and OpenAI, the
trolls.
Gemini 1.5 Pro drops.
Three hours later, OpenAI goes, oh, you think that's good?
Check out this.
Tell the listeners about Sora.
Talk about raining on someone's parade, hey?
Big announcement from Google and then they come out with, hey, look at all of these one
minute long HD quality videos produced using only text inputs.
Aren't they amazing?
And the internet went wild.
We cared for Gemini Pro 1.5 within five minutes of this announcement.
It was crazy.
My timeline was absolutely full of it.
So yeah, this is OpenAI's text to video model, Sora, that creates realistic videos from text
based instructions.
As I mentioned, it's up to one minute in length.
They are really high quality.
They're HD videos and one of the fascinating things about them is that they have cut scenes
in them.
So it's not just one shot.
It will interchange like you would see in an advert or in a film or whatever.
When I saw that, that was the thing that barred the quality and the quality is very good.
Seeing that blew my mind slightly.
That was really impressive.
A couple of other things to note about this model.
It understands language incredibly well, enabling very accurate interpretations of the written
prompts.
It understands physics very well and the physical world.
So in one of the videos, there is a stylish Japanese woman walking down a street and there's
lots of puddles on the floor with various kind of city lighting and urban lighting in the
background and you can see reflections in the puddles and in the windows and you'll see
the reflections of other pedestrians within those puddles.
So it kind of understands all of the light and the environment as well.
It does struggle with some more complex physics.
In fact, I think in the blog post announcing it, they talk about this and complex physics
such as you might see the video where somebody takes a bite out of a biscuit and then when
they remove the biscuit from their mouth, the biscuit is still a whole biscuit.
Not got any of it.
And I think we can all agree that biscuit physics is the next frontier of science to
crack.
If they can solve that, it'd be pretty sweet is all I'm going to say.
But there is a video on the...
So if you want to read more about this and to be honest, we're going to do our best to
explain how awesome these videos are.
But if you really want to know what we're talking about, you need to go read the blog
post on the OpenAI site, which also has a link to a longer article that goes into a
bit more detail about how the model works and what it's capable of.
And in the second one, the more detailed one, there is a video where someone bites a burger
and the burger comes out of the mouth, bitten.
So there are clearly weaknesses in the model, but there are some incredible things that
it can definitely do.
It's been built using the diffusion models and the transformer architecture.
So diffusion model is what the DALI 3 model is built on and transformers is what's really
underpinning everything.
NPT, the T stands for transformers, right?
So this is the underlying technology that they've gone all in on.
And there were some interesting notes in there about what this means for where AI is heading
in terms of understanding the physical world.
Yeah.
I mean, there's two quotes that we're going to read out verbatim.
One of them is from the launch blog post and one is from the sort of more technical deep
dive of how this all works.
They mean the same thing in slightly different ways, but I just think it's interesting to
think about them, especially because we have to work under the assumption that whatever
we see today, the labs, the people in the labs doing the work are six to 12 months ahead
of us, right?
So they are on the cutting edge.
Sam and I were talking off air as it relates to Sora about how a couple of months ago,
maybe back end of 2024, Sam Altman said he'd seen something that he felt was the next step
change forward that kind of blew his mind.
And I'd have to guess it's this.
If it's not this and there's something even more powerful than this, then holy crumbs
were in for a pretty interesting three or four months.
But here's the quotes, right?
So the first one, which is at the bottom of the research preview is, we believe the capabilities
Sora has today demonstrate that continued scaling of video models is a promising path
towards the development of capable simulators of the physical and digital world and the
objects, animals and people that live within them.
Right?
Quote one, hold your thoughts, Martin, because I know you've been thinking about this a lot,
right?
Quote number two, which is at the bottom of the launch post, Sora serves as a foundation
for models that can understand and simulate the real world, a capability we believe will
be an important milestone for achieving AGI.
This model, as Martin just alluded to, is a GPT.
It predicts the next thing based on the things that came before it.
How on earth do we see this emerging capability to sort of understand the physics of the world
and output them based on a text based prompt?
I mean, Martin, thoughts?
The researchers have really been confident about this for some time.
I know there's some dispute.
If you were to listen to the likes of Jan LeCun from Meta, he would argue that there
are limitations to this, but clearly OpenAI see this as the direction of travel and then
it's a question of scale and compute.
If you keep throwing more and more of that at it, these transformer models will get them
a long way towards AGI.
This is a really interesting step in helping these models to develop what some might call
a world model, that idea of understanding the world around them.
GPT-4 had some elements of, from a text based perspective, having something of a world model
within it.
They think in the research paper when they announced it, they gave the example of with
GPT-3, if you have described a house and said you walk through the front door, on the left
there's a kitchen, a living room, on the right, and you kind of give a visual, or sorry, a
verbal description or written description of the layout of a house.
And then you said things like, what was the example I gave?
I think if you walk into the bedroom, you put something on the bed, you pick it up,
you walk into the bathroom, where is the thing?
And GPT-3 would be like, it's on the bed.
Whereas GPT-4 had this understanding, it would know that you picked it up and you've now
carried it into the bathroom.
And it was also able to give you visual reasoning with maps.
So there was evidence of this happening already.
Now seeing this manifested in video is, yeah, very, very impressive.
Yeah, like an emerging understanding of cause and effect in the real world, which is interesting.
I mean, Microsoft released that 100 page paper, Sparks of AGI, about GPT-4 and its emerging
capabilities.
I mean, when Yan Le Kun doesn't think this is the route, certainly someone who knows
what he's talking about, right?
So the fact that there is divergent opinion in the world's leading experts in terms of
what these models are going to be capable of and what they're even capable of now, in
terms of really capable of is interesting.
But I was walking the dog today, Martin, and I was trying to think about what does all
this mean?
And I was just trying to reflect upon how a two-year-old learns about the world, which
is through its eyes, its ears, its sense of touch.
So a lot of the things that a child is using as raw inputs to learn about the world is
what we've given these large language models.
If the GPT neural net framework is able to learn in inverted commas in a similar way
to the human brain, that wouldn't be a surprise.
We kind of came up with the idea based on the fact that's how the brain works.
We've just simulated it in a different way using silicon than biological components.
So it doesn't surprise me perhaps then that you feed it enough information.
I should say, I think the silicon-based version we've created is like kind of crappy compared
to biology's ability to draw general principles from very little data because our system we've
built is crappy.
We have to give it petabytes of data for it to be able to pull out those patterns, whereas
somehow the human brain is able to learn them much faster and much better.
And I'm sure as we improve our models and architecture, maybe that's where we'll see
the biggest improvements.
But feeding it that information and its ability to just learn like a child almost.
And then we were talking about this off air and I was mentioning that wouldn't it be interesting
if you took one of these models and then trained it based on two cameras worth of data, not
one.
Like you gave it human eyes.
So instead of it having to infer the world from one static 2D picture, it had two pictures
of the same thing, stereoscopic vision, so we can understand depth a bit more like we
were and how cool that would be.
And then we were like, oh, Meta have been recording it for at least four or five years,
probably longer, not just from two cameras, but from like 12.
So imagine the data repository that they've got for this type of model training of the
real world.
And now people are out walking about the streets in their vision pros collecting multi-camera
data of the real world.
So and the fact that we've been feeding all this data to YouTube for 10, 15, 20 years,
like the amount of data out there to train these models.
These tech companies have been so much more clever than I gave them credit for in terms
of collecting all this data, because if they're able to now use it to give computers this
real sense of how the world works, again, imagine the applications that's going to open
up.
Yeah, feed more into it, get more out, right?
And it was interesting you mentioned about how do toddlers learn about the world and
how do animals learn about the world?
Because that's actually Jan Lukens point about the limitations of transformers.
Because only so far you can go, all of this extra sensory input is what helps us build
our understanding of the world.
How do you get a large language model to understand things like proprioception or temperature
and gravity and all of these external forces that we encounter and what have you, it's
very difficult to get there.
And the human brain can do it incredibly efficiently.
That's the advantage that biology has, right?
Imagine we end up doing all this work and over like the course of like a thousand years
and then we're like, at some point we switched to biological components because they're much
better than silicon and we basically just create you and stuff.
That's what it's worth wanting.
A couple of last few bits on this and then we'll get into like why this is important
for marketing and business folk.
Read the research paper because some of the things this tool can do are much more mind
blowing for me than the launch blog post mentions and therefore a lot of the stuff you'll see
being shared online for me is missing the point of some critical stuff, especially when
it comes to things like marketing applications.
So I'll give you an example.
It's not just text to video, it's image to video and you can give it a video and it will
imagine what the first 10 seconds or adding an extra 10 seconds to that video might look
like.
So there's lots of creative capabilities for marketers here that we'll only just start
scratching the surface of when we get access to this.
In some examples on the research paper, they upload like a flat 2D image of some monsters.
It's just like a vector image of some characters, but then it animates them in a realistic way.
Like it waves their hands and it moves their legs.
So think of all those static graphics that you've got in all your presentations and on
your website because hey, it would be cool if these animated, they'd be more interesting,
but it's just not worth paying an animator to do the work because it just doesn't add
enough value.
Expect an explosion in animated everything just because you can.
No ghost.
Some of the examples are really cool.
This extends to photorealistic video from an image.
So there's an example of a dog wearing a beret, which I suspect was generated by, in fact,
that was generated by Dawley3.
And then the video is created of that dog and it just looks like a video of a dog, like
it was a real dog.
Like it's absolutely mind blowing.
Then there's an example where they're able to create an infinitely looping video that
you have to watch for about four or five cycles to get your head around where it loops, which
is extremely clever.
I wonder what we can do with that.
There's a video to video editing.
So there's a video shot of a car driving along a road in a forest.
And then the prompt is to change the setting to be a lush jungle.
And it looks amazing.
Like it looks very similar to the original input video.
Imagine some of those videos we saw in the early days of runway where people were filming
themselves on their phone and then creating like stylized animations based on their own
movement, but now photoreal.
How does this change the gaming industry?
How does this change the CGI video effects industry?
How does this open up CGI level capabilities to brands in the B2B and B2C space that would
have normally have gone, of course I can't afford to spend two years and millions of
dollars on Pixar level engineers to create an animators to create this level of stuff.
Oh, I did it when I went out in my garden with my phone and then put it through a text
prompt.
And then however long this model takes to render, we don't know that yet.
Four minutes later, I get this thing.
It's absolutely mind blown.
The videos are a minute long.
Most things lose can fluss coherence at 10, 15 seconds.
So for that's mind blowing.
And then the last thing that just like, again, I don't even know how we're going to use this.
I just know that it's cool.
They've got this interpolation effect.
Have you seen this button?
No, I missed this bit.
Go check it out.
So basically they've got two videos, which as far as I understand it, were both synthesized
by Sora and then they interpolate the videos.
And I'm not even sure I can really explain what that means.
In one video, they're shot.
They've either shot or I suspect synthesized a drone flying around a ruin.
So it's a drone flying around a ruin as if it's been shot by a drone above it.
Right.
And then the other video is, and again, who knows how they created this, must be synthetic
because it's a butterfly swimming, flying around a coral reef.
Then they interpolate the videos.
So the video starts as a drone flying around the ruins and then the drone morphs into the
butterfly and the ruins morph to make it look like they've been created out of coral.
Like it's crazy.
All 1080p, all photorealistic.
I don't even think we've started to begin to think about how you could leverage those
types of interesting effects to do stuff.
And we're thinking big picture creative here.
I think if you're a B2B strategist, brand manager, creative director, you are going
to get to be able to do cool things in your business or for your clients that were previously
so cost prohibitive as to be ridiculous to have even thought about them.
Prepare to unleash your imaginations because that is what this technology I believe is
going to allow.
And I'm sure it will have loads of problems and artifacts and you'll never, it might take
50 prompts to get the thing that you want.
It's still going to be faster and less expensive than hiring 420 people, film crew and like
42 animators of Pixar level quality.
I just think it's cool.
Shall we move on to our next story?
Oh, sorry, Martin.
Just in conclusion there, I mean, if you are a videographer or you've been making a bit
of income from licensing stock video footage, maybe start diversifying those revenue streams.
Indeed.
I remember when runways capabilities were really starting to be a bit more controllable
and people were shooting those 90 second videos where they obviously had to stitch all the
videos they created together.
And as people who've played with runway and got poor results, we could only imagine how
long that would have taken people, right?
Right.
A lot of it.
Yeah, absolutely.
I cannot wait to see the first short three, four, five minutes created with this technology.
I bet you could do it in a day.
Easy.
I think the limitation then will be consistent characters and locales.
How easy is that to do?
And then of course, at some point we're going to want human characters that speak and interact.
We're going to have the synthetic voices as we'll see a bit later.
But how easy is it going to be to get a human to speak and have their mouth move in time
with the speech and all that stuff?
But it sounds stupid that we would even think about that as a possibility.
But we may be months from that or maybe a year or two, but not decades if you look at
this leap.
I mean, it's mind blowing.
It's mind blowing.
Oh, right.
Everybody take a deep breath.
Shall we move on to the next story, mine?
Yeah, let's go ahead and do that.
Right.
Let's talk stable cascade.
Any other week.
I think this would be really, really cool and interesting.
But compared to the things we've talked about already today, I guess it's almost kind of
a little bit less interesting than it might have been.
But it's still important, I think, for us marketers to know about because Stability
AI have released a new text to image generation model, which has some slightly different things
about it compared to the models we've seen so far.
So unlike stable diffusion, the sort of normal model, the standard model that most tools
are using, which is just one single large model, stable cascade is basically a three
stage architecture of smaller models that work together creatively called stage A, stage
B and stage C. At least it does what it says on the tin.
Right.
Basically, the way that it works is it makes it extremely efficient to train and fine tune
the model because instead of working with one massive model, you're working with three
smaller models that interact.
And some of the data around this is that there's 16 times cost reduction over fine tuning and
equivalently sized stable diffusion model.
Why is this important?
Because if you want to fine tune a model as a developer or for a custom use case and you
want to feed it images and information to help shape what it outputs, obviously you
don't want that to be too expensive.
The image quality that produces is extremely high quality.
So it's not like they've lost any quality in terms of introducing this new way of doing
things and it's got image masking.
So for example, you can like blank out a bit of an image and then it will regenerate the
image in that space.
So in the example, there's a cat, they remove the cat's head and upper body and then they
output it as a dog in the same locale.
Works really well.
They generate images based on line drawings.
So we've obviously got in clip drops some stability AI driven sketching tools where you
do a sketch and then the tool is able to produce an image based on the sketch.
This is very, very similar to that.
It can upscale images as well, but all of this is being done with this smaller, easier
to manage model architecture, which is just finding new and more interesting ways to be
able to make it possible to train models, build models and on much simpler hardware.
I'm not sure if they mention anywhere that you can run this locally on like a computer
rather than having to use it in the cloud.
I suspect, I mean, you can run stable diffusion on your laptop.
It's just slow.
So I suspect this will allow local image generation much better than using current level of stable
diffusion models.
So, Hey, if you've just got a laptop with a decent GPU in it, you can probably run this
model pretty well, but we've got all this stuff happening in video.
We've got Gemini Pro 1.5 and all this cool stuff, but we're racing ahead, not just in
video, but image stuff is improving.
It's just so much going on in this area.
So again, if you're a marketer, really thinking about the creative tool set that you've got
available to you to do your work.
Let's talk text to speech.
We've done video, we've done image, we've done retrieval of information and text documents.
Let's talk a little bit about text image.
Martin, tell us about the news from Amazon this week.
Well, I nearly missed this, but you brought it to my attention and I'm glad you did because
it's a fascinating update in an area that we normally just talk about with 11 Labs and
well, OpenAI announced their text to speech model at the developer day and that was very
good, but it's not an area that we focus on greatly.
So Amazon have announced a new large scale text to speech AR model called Base TTS, so
Base Text to Speech and it exhibits some interesting emergent abilities.
And the researchers train this model thinking that these might occur, that they might achieve
some of these emergent abilities.
So just to be a background on it, it's the largest text to speech model to date, 980
million parameters trained on a hundred thousand hours of public domain speech data, 90% of
which is English, 10% Dutch, German and Spanish.
They found, the researchers found that training a TTS model on just 10,000 to a hundred thousand
hours of speech can result in some emergent abilities similar to the emergent abilities
that we see when you scale up the data and the parameters in large language models.
So the kinds of things that it's emerged, the capabilities that have emerged from it
include a significantly improved ability to naturally speak certain elements of complex
sentences such as compound nouns, emotions, foreign words, the kinds of things that normally
trip up a text to speech engine.
It also had punctuation, so if you added a piece of script and it had like hashtag just
saying, that phrase would trip up TTS models historically, but now they can do that.
It has uncanny naturalness, can mimic speaker characteristics with just a few seconds of
reference audio, all the kind of stuff that is kind of almost there with the existing
models.
And I'm really excited to get my hands on this because I've actually just upgraded my
11 Labs subscription to get the fine tuned voice clone, the professional voice cloning.
And for that they require about three hours of audio is what they recommend.
So I'm really interested to see how closely this new model can match somebody's voice
with the existing models, things like 11 Labs and other text to speech providers.
If you try and clone your voice, rather than do a pure clone from scratch of your voice,
what actually happens is the model tries to fit your voice to one of the existing voices,
which if you're British, one of the things that you'll often find is that it overfits
to a British voice, which can make you sound more RP in your pronunciation, more old school
BBC.
Yeah, BBC in this, hush.
Yeah.
I believe pronunciation is a problem, right?
Because that's not how actually most people in Britain would talk in 2024.
So I'd be really curious to see, can it capture our actual dialect tone from just a couple
of minutes of recording, or actually you still need quite a bit of fine tuning.
This area for me is one of the most exciting and interesting areas of AI development for
content marketing in particular.
If you're a content marketer, I think there's so much that you can do with this because
of the just vast amount of video content being consumed.
If you can scale up your content production for social snippets and things like that,
without you having to sit and record and re-record and all of the editing process, you can just
get the voice that you like or get your own voice and type in some text and have it read
aloud for you on a social post or something.
I think there's massive potential there.
But at the moment, I think the tools aren't quite up to the job.
So this sounds very promising.
Yeah, it's interesting.
We've got an agenda item for one of our podcast episodes, but there's always, or recently
has been too much exciting news for us to get to it where we want to deep dive into
what does marketing and content consumption look like in a post.
It was going to be in a post GPT-5 world, but I think you can add the base TTS model
to that and Sora and Gemini 1.5 Pro.
But yeah, the ability to create high quality content at scale, whether that's video, audio,
text is basically already there, but I don't think we've quite seen the ramifications of
yet as content marketers.
How does that change how people consume information?
How does that change how brands create information?
Because I feel like I'm hooked to my devices already just trying to keep up with the information
flow.
Now, this just all lowering barriers to create more information that I can also engage with
in other domains.
One of the benefits of text to speech is listening to podcasts I love because when I'm walking
the dog, my ears aren't really doing anything, but the rest of me is my eyes and my body.
It's not the same as when trying to read a book or sit at a computer.
So I do think we should get into that on one episode, and I probably don't have time today
because we're near the end of the episode, but all the barriers dropping means more deluge
of stuff.
Only 24 hours in a day though.
Well, this is it.
So then it becomes about prioritization, the quality of the content that you're consuming.
None of this stuff is doing any of the thought behind what would be useful for people to
hear about what unique expertise and viewpoints do you have that you can share as a company,
as a person.
That puts obviously still the critical bit that's far from automatable at this point.
But I've listened and maybe we'll in the edit, we'll add a few of the audio clips from this
paper in because they're kind of really interesting in the way that...
When I think about this, I'm always thinking because I've got a lot of books in my Kindle,
I buy a lot of audio books, but I've got a lot of Kindle books where there isn't an audio
book version and I'm like, that sucks.
I want to listen to this when I'm walking the dock.
I don't want to read it because I'm finding that my time to sit down and read long form
content is ever more reduced.
So I'm like, cool.
When do I click a button in Kindle and listen to the audio book version of a book that doesn't
have an audio book recording?
One of the criticisms of that has been performance, right?
But a couple of these audio examples, they must be trained on those types of performances
because a lot of them are like stories talking about characters speaking to each other.
And then the synthetic voice reading will be like, Paul entered the room and he looked
to Martin and he whispered, Martin, we got to be careful.
The dragon's over there.
Martin looked frightened and like these examples that have been given, the model knows to...
I can't even describe what that is.
Like put on a different voice to imply that a character is speaking to another character.
But also, and I chose the whispering example deliberately because one of the examples is
whispering and that would be one of the main barriers, I think, because nobody wants to
listen to an audio book that's monotone basically, right?
And these text to speech engines, they're way better than monotone, but the ability
to actually perform in them.
Wow.
I think that's where they... I mean, when base even stands for... The E in base stands
for emergent abilities.
So they're hanging their hat on this as we trained it on a load of stuff and it can do
things that it shouldn't really be able to do and that we could perhaps have possibly
predicted but it's much better than we thought it would be.
So, what a week.
What a week.
And let's talk the last story, mine, because you spotted some interesting stuff online
about this.
And again, in the context of everything else that's happened this week, this is probably
a bit like, well, it's just a rumor, but it probably would have had top billing last week,
mine.
So, let's imagine it's last week, dear listener.
Martin, tell us about the rumor about OpenAI and Search.
So there's a report in The Information, which is fast becoming my number one source of interesting
content when it comes to the AI and technology field, that OpenAI is developing a web search
product that will leverage Microsoft Bing's infrastructure in a move to basically position
themselves up against Google in the search domain.
The timing is interesting.
So Satya Nadella had declared a year ago that Microsoft was making Google dance by infusing
Bing with OpenAI's AI prowess.
And despite that, Bing's market share of the search market has barely moved really.
But we know that there is demand for AI-powered search.
And how do we know that?
Well, there is already an AI-driven search engine on the market, Aplexity.
And Aplexity has been grabbing people's attention.
Recently, it had a half a billion dollar valuation.
Big investors going in, including Jeff Bezos.
It's been, I think it was a hundred million dollars it raised in its recent round.
It was reported that they have eight million dollars in annually recurring revenue.
So that's people paying a subscription to access a search engine.
So there is clearly an appetite and a shift in the way that people are searching.
And when I see people talking about the way that they use Perplexity and when I reflect
on my own use of Perplexity, I recognize that I go to it for answers immediately.
I don't want to be presented with a link to find the answers.
I want the answer given to me.
And it looks like OpenAI have realized that that's what other people want and they are
well positioned to give it to people.
Yeah, it was obviously a rumor that was circulating, but then you spotted a post by AI guru, Ethan
Molik, who we've mentioned before on the podcast.
And one of the things that we're seeing as a consistent pattern with Ethan is that some
OpenAI will drop some news and then he'll have a nice long post written about it for
his Substack email newsletter because he's had it for a month already.
And he's very, very good at keeping quiet about what he's got access to until everybody
else in the world knows.
But you spotted a post on LinkedIn related to this that we thought was interesting and
maybe implied that Ethan has access already.
I'm going to summarize the post and basically it's about sort of using iron bridges as a
metaphor because he talks about how the first iron bridge was made using the same approaches
that woodworkers would have used because no one really knew how to use and work with iron.
So they applied what they knew to this new material.
But of course, we know today that woodworking strategies and approaches are not appropriate
for iron.
You have to think about the problems that you're trying to solve with the material that
you've got in hand in a different way and take different approaches.
And then he goes on to say, for example, AI powered search is adding a new gloss to an
old paradigm.
Instead, we can focus on the reasons we search and try new ways to solve the problem.
So why replicate search engines when even a GPT-4 class AI is capable of silently observing
what you do and jumping in at the right moment to help provide informational suggestions
if you seem to be confused or faltering?
Well, dear listener.
If past behaviour is a reasonable predictor of future behaviour, what is it that Ethan's
playing with that we don't have access to yet?
It does open some interesting possibilities.
You've got to think that that is, you know, that's like kind of subtweet-esque.
He's onto something.
He knows something.
That would be the guess.
And I look at Gemini Chat's Google button that allows you to basically fact search the
output that you've just generated to see if it's, you know, either wrong or can be substantiated
by information available on the internet.
And one has to assume that that sort of button at the very least is coming to ChatGPT.
Right, you get an output and you'll be able to check its validity.
And then maybe if you've got these tools baked into a tool like Word, you're writing about
something, you guess maybe, or you get it to generate a bit of an output for a paragraph
that you're working on.
And then it's silently searching other information sources in the background, making suggestions
to you like, oh, if you thought about whether this point might be important for your blog
post as well, here's some information about it.
We can write you your next paragraph if you want us to type of stuff, I guess.
I mean, I'm thinking about it very much like a marketer there, I guess.
But yeah, all of which comes at the same time as we know, it's been reported this week that
OpenAI is also investing quite heavily in the development of AI agents.
So talk about convergence of different technologies.
Where will this lead us, eh?
So I think that's probably a great place to close out the episode because it's been a
crazy week.
I never thought we would see this acceleration with Gemini that we've seen.
And that context window could really open up some interesting new applications.
We've said it before on the podcast, if you're not recording and archiving a lot of your
meetings, client calls, customer service calls, record them.
Even if the technologies are not quite there today to do what you might want to do with
them, they will be tomorrow.
So get the data now.
So that Gemini is cool.
Sora, I mean, who saw that coming?
That is absolutely mind blowing.
But your point about agents, supported search or whatever this thing is that Ethan's referencing
to, I don't even think, well, it's 16th of February, 2024.
We are not half done in terms of what we're going to see by the end of the year.
And I think this week has taught me that as much as I feel like we're very close to it
and can predict some of the things that are going to happen, I would not have seen that
Sora video thing coming.
No, no, that I would have expected 2025.
That's when we'll see that kind of quality.
Yeah.
So we can't promise you such an earth shaking episode next week, but we can promise you
we'll be here.
There'll be a little bit of Derby County.
There'll be a little bit of Banta and there'll be a little bit of AI news.
So until then, if you find this valuable, subscribe, share it with people who you think
might also enjoy it.
Martin, have a fab weekend and I look forward to speaking to you soon.
See you.
Thank you for listening to Artificially Intelligent Marketing.
To stay on top of the latest trends, tips and tools in the world of marketing AI, be
sure to subscribe.
We look forward to seeing you again next week.