Okay, so get this.
AI that thinks in ideas, not just words, but actual ideas.
Yeah, it's a pretty big shift.
Right, like totally different from how we usually think
about language AI.
Definitely.
So this paper you sent in, large concept models,
language modeling in a sentence representation space.
It basically says, forget the old way of doing things,
we're going concept driven.
It's exciting, trying to like get closer
to how we think, you know?
Oh yeah, for sure.
Like our brains don't think word by word, right?
It's more like-
Oncepts, exactly.
Think about when you're writing something,
planning a speech.
Totally, you're not just like stringing words together,
there's a bigger picture.
You've got these ideas you want to get across.
Okay, I'm with you.
But how do they actually teach an AI to do that?
What are these, what did you call them, large concept models?
It's LCMs.
Right, LCMs, what's the deal with those?
Think of it like this, the language AI we have now,
like the stuff behind chat JPD.
They're good at guessing the next word.
Like statistically what word comes next.
Right, right.
But LCMs, they're trying to predict the next concept.
And the way they do that is with something
called sentence embeddings.
Sentence embeddings, so like taking a whole sentence
and squishing it down into a code.
Yeah, basically like a fingerprint of the idea
behind the whole sentence.
And the really cool part is this paper uses a system
called Sonar.
And Sonar can make these embeddings for like
over 200 languages, even speech.
Hold on, 200 languages.
So you're saying this AI could like listen to me
talk in English and then I don't know,
translate the ideas into French.
But without actually translating word for word.
Yeah, that's the goal, it's still early, but.
That's wild.
Okay, but back up a sec.
How do these LCMs actually work?
Like step by step.
All right, so imagine you feed some text into the AI.
First thing it does, splits that text into sentences.
Gotcha, like breaking a book into chapters.
Yeah, good analogy.
Then each sentence, it gets turned into this concept vector.
That's where the sentence embeddings come in.
Okay, so each sentence is like a little package
of meaning, this concept vector.
Right, and then the LCM comes in,
it takes those concept vectors
and it tries to predict what the next one should be.
So it's not just the next word,
it's figuring out the next idea.
Exactly, it's about the flow of ideas, not just words.
So you're getting the logic behind a story
and argument, whatever.
It's just like moving past just grammar
and getting into, ah.
Semantics.
Yeah, semantics.
Yeah.
And once it's predicted the next concept,
it uses sonar again to turn that vector back
into a sentence we can read.
So it's like text to concept, predict next concept,
concept back to text.
Pretty neat, but I'm guessing it's not that easy, right?
You got it.
Language is messy.
Oh yeah.
Like there could be tons of ways to continue a sentence,
lots of different meanings.
Right, right.
So how does the LCM deal with that?
Does it just pick like the most statistically likely
next concept?
They do something more interesting than just statistics.
They have these things called diffusion-based LCMs.
And these diffusion-based LCMs,
they use this process of noising and denoising.
Imagine you take a picture, like a photograph,
and you add a bunch of static to it.
Like on an old TV.
Yeah, exactly.
And then they train the LCM to like remove that static
and get back the original picture.
And by doing that over and over,
the LCM learns to see through the noise,
to get the structure, even with all that mess.
So like training the AI to see the core meaning,
even if it's hidden.
Exactly.
And there are two main types of these diffusion-based things.
One tower and two tower.
One tower, two tower.
Why the different names?
It's all about their architecture,
like how they're built inside.
One tower, everything happens in one structure,
like a one-stop shop.
OK.
But two-tower models, they split it up.
One tower for encoding the input into concepts,
and another tower for predicting and decoding the next one.
Huh.
So it's like dividing the labor?
Yeah.
Each tower is a specialist.
Yeah.
And what they found is this two-tower setup,
it seems to work really well for language,
especially for generating text that makes sense, you know,
is coherent.
Like it consistently outperforms the others.
Makes sense, right?
Like a team of exploits.
Exactly.
But there's another type of LCM they looked at too.
It's called a quantized LCM.
Quantized.
That sounds complicated.
Well, it's about dealing with how huge the concept space is.
Imagine all the possible concepts, right?
Like every idea ever.
OK.
That's a lot.
So quantized LCMs, they try to simplify that
by like breaking it down into chunks.
So instead of infinite concepts, it's more like a grid.
Yeah, exactly.
It makes it easier to learn.
And they have two kinds of these,
quant LCMD and quant LCMC, which do this chunking
a little differently.
OK, so we've got the basic LCM, the diffusion-based ones
with their towers, and these quantized ones.
That's a lot of options.
How do they know which ones best?
That's where the experiments come in.
They do what's called ablation studies.
Basically, they test them all out head to head.
So like an AI bake off.
Yeah.
And what they found is those diffusion-based LCMs,
especially the two-tower ones, they do really well.
So two towers the winner so far.
What else did they learn from these tests?
One interesting thing was about tuning the noise schedule.
Noise schedule.
Remind me what that was again.
Remember that whole noising into noising thing?
The noise schedule is like, how much noise you add and when?
It's like how much static you put on the TV picture.
Yep.
And by adjusting that, they can actually
control how well the LCM predicts the next concept
and also how diverse the output is.
Oh, that's cool.
So like finding the balance between sticking to the script
and being creator.
Right, exactly.
And another thing they found, this was interesting,
this idea of fragility in the sonar space.
Fragility.
So it turns out that some sentence embeddings,
they're really sensitive to changes.
Like you change one tiny thing and the meaning's
totally different.
Oh, like that telephone game.
Exactly.
And that's a big deal for training these LCMs.
Because if the foundation's shaky,
the whole thing could fall apart.
So it's not just about the LCM itself.
It's got to have good, solid concepts to work with.
Right.
If the sentence embeddings are messed up,
the whole thing's messed up.
OK.
So we've been talking about these kind of smaller models.
Yeah.
But I have a feeling they didn't stop there, right?
You know it.
They went big.
They built a 7 billion parameter two tower LCM.
Whoa, 7 billion.
That's massive.
What could you even do with a model that big?
Well, they tried out some tougher tasks,
like text summarization.
You know, taking a long piece of text
and boiling it down to the main points.
And they even did this new thing called summary expansion.
So you take a short summary and you
generate a longer text from it, like fleshing out an outline.
Wow.
So not just thinking in concepts,
but also doing these complex tasks.
Yeah.
Did it work?
Like was this mega LCM any good?
They used a bunch of different metrics to check it out.
And yeah, it's impressive.
Even at that scale, LCMs were holding their own
against the usual large language models,
sometimes even beating them.
That's incredible.
But wait, there's more, right?
You mentioned something about multilingual magic.
Yeah, this is where it gets really cool.
Because these LCMs are working with concepts,
they can potentially handle any language that's
so in our supports and get this, without extra training.
Hold up.
You mean I could train an LCM on English text.
And it could understand, I don't know, Japanese
without ever seeing Japanese before.
That's the idea.
It's learning the concepts, not just the words.
My mind is blown.
Did they actually test this out?
Oh, yeah.
They did a test on this multilingual data set called
XL Sum.
And the LCM, it beat a really powerful language model,
Lama 3.18BIT, on 42 different languages, many of which
it had never seen before.
No way.
It's pretty amazing.
This Zeryshot multilingual thing,
it's a total game changer.
And this is huge.
It sounds almost too good to be true.
I know.
It's still early.
But think about it.
Communication, collaboration, understanding.
No more language barriers.
Exactly.
And they even go one step further in the taper.
They start talking about giving LCMs a plan to follow,
like an outline.
So instead of just predicting whatever concept comes next,
it's got a roadmap.
Right.
They call them break concepts and plan concepts
to help structure things.
So break concepts are like chapter breaks in a book.
And plan concepts tell you what should be in each section.
They even tried it out with what they call a large planning
concept model, or LPCM.
And it looks really good.
The text it generated was way more coherent.
So these LCMs, they can think in concepts,
they can do multiple languages, and now they
can follow a plan.
This is revolutionary.
It's pushing the boundaries, for sure.
We're getting closer to AI that can actually think like us.
This is mind blowing.
But OK, before we get too carried away,
we've got to talk about the limitations too.
I mean, no AI is perfect, right?
Definitely not.
And the paper is honest about that.
Good.
So that's what we'll do in the next part of this deep dive.
Stick around.
We've got more to uncover.
Back again, ready to talk limitations.
Let's do it.
It all sounds amazing, but no AI is perfect.
Right.
The paper's upfront about that.
There's still a lot to figure out.
Yeah.
I mean, concepts themselves, they're
kind of fuzzy, right?
Mm-hmm.
How do you even define a concept for an AI?
That's the million dollar question.
How do you really capture what a concept is?
And that ties into, remember that whole fragility thing
we were talking about?
Oh, yeah, with the sentence embeddings.
Like, a tiny change can totally mess up the meaning.
Exactly.
So when you're talking about concepts,
that fragility is a big deal.
Imagine the AI trying to understand something
complex, democracy, justice, these big ideas.
If the sentence embedding for those is off,
even by a little bit.
The whole thing's messed up.
Yeah.
It's like you're building on a shaky foundation.
Doesn't matter how fancy the rest of it is.
Right, right.
So you've got to make sure those concept vectors are, like,
really solid, accurate.
Absolutely.
And that kind of leads into another limitation,
the computing power, encoding all these sentences
into concept vectors, and then decoding them back.
Oh, yeah.
I bet that takes a lot of juice.
It's a lot, especially with these huge data sets.
It's like a whole extra layer on top
of regular language processing.
So it's a trade-off, right?
You want the AI to think big, but it's
got to be able to handle it computationally.
Totally.
And then there's the data itself.
Like, LCMs still need to learn from tons of text, right?
For sure.
And if that data is bad, biased, or just limited.
Garbage in, garbage out.
Exactly.
So finding good data, that's a huge challenge,
especially for all those languages.
200 plus, that's sonar can handle.
It's got to be high quality, diverse,
and represent all those different ways people use language.
That's a tall order.
Yeah.
OK, so we've got the abstractness of concepts,
the computing power issue, and the data problem.
Anything else?
Well, there's also the question of how
you evaluate these things.
Like, how do we know if an LCM is doing a good job?
Right.
It's not just about predicting the next word anymore.
Exactly.
We need ways to measure how well it understands ideas,
how well it can think conceptually.
That's tricky.
Yeah.
So we need new metrics, metrics that
make sense to humans that capture what we think of
as conceptual understanding.
Like, you can't judge a painting just
by looking at the breast drugs.
Perfect analogy.
There's more to it than that.
We need to see the whole picture, you know?
OK, so limitations.check.
But we can't forget about the potential here.
I mean, this is AI that could change everything.
Communication, understanding, the way we learn, create,
even solve problems.
I know, right.
It's mind blowing.
This paper, it just opens up so many doors.
It does.
Even with the challenges, this feels like a huge step forward.
Absolutely.
Like, we're getting closer to AI that's truly intelligent.
So in the last part of this deep dive,
we'll explore some of those possibilities.
Get ready for a glimpse into the future.
All right, final part.
The future.
Let's get into it.
The big picture, right?
Yeah.
I want to hear about all those possibilities.
We've talked about how it works, the challenges.
Yeah.
Now let's, like, unleash our imaginations.
Imagine a world where language isn't a barrier anymore.
Where AI can translate, understand, but not just words,
the ideas across, like, all these different languages.
That's powerful.
No more struggling with translations
or trying to figure out what someone really means.
Just clear communication.
Global collaboration.
And it's not just language, right?
Because it's all about concepts.
These LCMs could work with other stuff, too.
Images, music, even, like, scientific data.
Wait, hold on.
You're saying an AI could get the meaning behind a photo
or, like, the emotions in a song.
That's the idea, bridging those gaps between how we express
ourselves and how we understand things.
So, like, an AI could read a scientific paper,
get the key points, and then, I don't know,
make a chart of the data.
Yeah.
And explain it all in simple language.
Exactly.
Think about how useful that would be for research,
for learning.
So many possibilities.
It's like having this super smart assistant
that can process any kind of information
and tailor it to you.
Personalized learning.
Imagine that.
An AI tutor that knows exactly how you learn best.
OK, that's amazing.
But what about, like, creative stuff?
Could these LCMs help you write a story or compose music,
design new products even?
I think so.
Because you're working with concepts,
it's, like, breaking free from how we create now.
You could explore new ideas, combine them in different ways.
Push those boundaries.
Exactly.
It's like having a brainstorming partner, one that never
runs out of ideas.
That would be awesome.
But OK, zooming out even more, what about the impact on,
like, society as a whole?
Could these LCMs help us solve big problems?
That's the big question, right?
What if they could analyze all that data
we have on climate change, poverty, whatever,
and come up with solutions that we haven't even thought of?
It's like a global brain working on all these problems.
It's exciting, but we've got to be careful, too.
Like, with any powerful technology,
there are ethical things to think about.
We need to make sure these LCMs are used for good.
Right, for good.
But still, this is pretty amazing stuff.
This whole concept of conceptual AI, it's
It's a game changer.
It is.
This paper, it's really opened my eyes.
Thanks for taking this deep dive with me.
It's been a pleasure.
This research, it's mind-blowing.
And we're just at the beginning.
That's what's so cool about it, right?
Who knows what the future holds?
All right, that wraps up our deep dive
into large concept models.
Until next time, everyone, keep those brains buzzing.
And if you find any cool AI research out there.
Send it our way.
We'll be here ready for the next deep dive.