Hey everyone and welcome back, ready for another deep dive.
Always.
Awesome.
So today we're looking into something pretty amazing
in the world of AI.
You know how we always hear about bigger and better AI models.
Right.
Well, what if I told you there's this new law emerging
that's all about making those models smarter and smaller.
Interesting.
At the same time.
Yeah.
It's called the Densing Law.
And it's gonna like change how we think about AI,
especially when we're talking about using it
on everyday devices like our phones.
Right.
So pretty interesting stuff.
We're gonna be checking out the paper,
Densing Law of LLMs today.
Sounds good.
So this paper really tackles that tension
between like wanting really powerful AI
but needing it to run efficiently.
Yeah.
On something like a smartphone.
Exactly.
We're talking about a potential future
where your phone has the brain power
of some of today's most advanced AI systems
without needing like a supercomputer to back it up.
Exactly.
It's crazy.
So okay, so I think we need to break down
this whole Densing Law thing.
I read the paper and it's a little dense even for me.
Where do we even begin?
Well, I think the best place to start
is with capacity density.
Imagine you have like this giant toolbox
overflowing with tools,
but you only really need like a handful
to build something amazing.
Okay.
That's kind of what's happening with these AI models.
Yeah.
They have tons of parameters
which are like the tools in the toolbox,
but they might not be using all of them effectively.
Oh, so it's not just about having a huge AI model.
It's how efficiently that model uses its resources.
Yes, exactly.
Interesting.
So does that mean that a smaller model
with a high capacity density
could actually be more powerful than
potentially a much larger model
that's less efficient?
You got it.
And that's what this Densing Law is pointing to.
Okay.
The researchers found that the maximum capacity density
of open source LLMs,
those are the AI that power things
like chatbots and text generators,
is doubling about every three months.
Every three months.
Hold on, that's insanely fast.
It is.
What does it even mean for like someone like me
who uses AI every day?
Think of it this way.
Okay.
Remember Moore's Law.
That's so many transistors.
Yes.
It's a chip.
Exactly.
It's about squeezing more and more transistors
onto a chip, making computers more powerful.
Right.
And the Densing Law,
it's doing something similar for AI.
Wow.
We're squeezing more intelligence
into a smaller package.
So you're saying in a few months,
we could have the same performance
as a large AI today.
Yeah.
But with a model half its size.
Yeah, potentially.
Wait, so that means my phone could be running
something as good as chatGPT.
Uh-huh.
In your future,
Yeah.
without, you know, killing my battery.
That's the idea.
That's a game changer.
Yeah.
This is the kind of future the research is hinting at.
Yeah.
And they have some pretty good evidence.
Like this model called Mini CPM1 2.4B
released back in February.
And it managed to do about the same
as the Mistral 7B model.
Which came out.
Which came out in September of last year,
but using only 35% of the parameters.
Wow.
That's a huge difference in size.
So the size of the model still matters.
Of course.
But this dinsing law is like suggesting
that we could be entering a world
where those massive models.
Yeah.
The power hungry ones.
Right.
Become a thing of the past.
Exactly.
It's not about raw size anymore.
Yeah.
It's about using your resources well.
So smart AI.
Yes.
And you know what?
This actually has some pretty big implications
for the cost of running these AI models too.
Oh really?
Yeah.
So we're talking more powerful, smaller,
more efficient.
And cheaper to run.
Yeah.
It's all thanks to this dinsing law.
That almost sounds too good to be true.
I know, right?
What's the catch?
Well, there are definitely some challenges
that researchers are still working through.
Like what?
One of the big ones is figuring out
how to measure AI model performance
across different tasks.
In a way that's fair.
Okay.
And comprehensive.
So we gotta make sure we're comparing apples to apples.
Exactly.
When we talk about capacity density.
Yes.
That makes sense.
Uh huh.
So how is this dinsing law actually like playing out
in the real world?
Are there any examples besides many CPM?
Oh yeah.
There are actually quite a few.
And researchers are finding that even in things like
image recognition and natural language processing.
Okay.
Smaller models with high capacity density
are starting to do as well
as their larger counterparts.
Oh wow.
It's becoming pretty clear that this dinsing law
isn't just some fluke.
Yeah.
It's a trend that could like fundamentally change
how we design and develop AI.
That's huge.
But how are researchers actually getting these improvements
in capacity density?
It sounds like magic.
It's not magic.
It's really clever engineering.
Okay.
One of the things that is helping
is the increasing scale and quality of the data
used to train the models.
So like if you wanna teach a kid a new language,
you give them lots of books and conversations
and experiences, right?
Exactly.
And it's the same with AI.
The more high quality data you feed it,
the better it learns.
So it's like we're giving these AI models
a bigger and better library to learn from.
That's a great way to put it.
Cool.
And along with better data,
we're also seeing some incredible innovations
in the algorithms and model architectures.
And those are...
Like the brains of the AI system.
Okay.
Researchers are finding new ways
to streamline these models.
Okay.
You know, pruning away unnecessary connections
and optimizing how they think.
So they can do more with less?
Exactly.
So it's smarter learning.
Thanks to better data.
Yes.
And then more efficient thinking.
Uh-huh.
Because of better algorithms and architectures.
Yeah, yeah.
So they're like optimizing every aspect of the AI engine.
Yes.
And this is leading to a big shift
in how we think about developing AI.
In what way?
The paper argues that we should move
from a performance-centric approach.
Okay.
To a density-centric one.
Okay, so instead of just chasing higher scores
on benchmarks.
Right.
Which usually means bigger models.
Exactly.
To prioritize building models that hit those scores.
Yeah.
But use less resources.
That's the idea.
Makes sense.
Because what's the point of super powerful AI?
If it's too expensive or uses too much energy to run.
Exactly.
It's like having a car that's super fast.
Right.
But it only gets like one mile per gallon.
Yeah.
It's about finding the right balance
between performance and efficiency.
Right.
And it's interesting.
The paper actually makes a connection
between this trend in AI and Moore's Law.
Moore's Law.
Remind me, that's how many transistors
you can fit on a chip, right?
Yes.
I mean, you can fit on a limited area.
But how does that relate to AI?
Well, think of it this way.
Just as chip makers are focused on squeezing
more transistors onto a chip.
To make computers more powerful.
AI developers are now trying to get more capacity density
out of their models.
So it's all about getting more bang for your buck
in terms of computing resources.
Yeah.
So it's not just about making AI models bigger.
Right.
It's about making them smarter and more efficient.
Exactly.
Just like with computer chips.
Wow.
And this trend could make AI much more accessible.
Yes.
Absolutely.
You mean like democratize it?
That's exactly what I mean.
If we can build smaller, more efficient models that
still perform well, then AI tools
could be available to way more people.
Oh, wow.
Think about students in developing countries
using advanced AI tutors right on their phones.
Right.
Or small businesses using AI analytics
to compete with bigger companies.
That's a powerful vision.
It's like leveling the playing field.
Yeah.
For anyone who wants to use AI, no matter what resources
they have.
Exactly.
But are there any like limitations?
That's a good question.
So we really make it that accessible
without sacrificing how well it works.
Well, that's the challenge.
OK.
One concern is making sure that these smaller models don't
lose accuracy or the ability to generalize.
Right, we don't want AI that's fast and cheap.
Right.
But can't handle difficult tasks.
Yeah, or adapt to new situations.
It's all about balance.
It is.
The paper mentioned some interesting future directions.
Oh, yeah.
Like this multimodal dancing law.
What is that?
That's where it gets interesting.
OK.
Right now, this dancing law is mainly about LLMs.
Which are all about text.
Yes.
OK.
But what if we could do the same thing for other AI models?
OK.
The ones that deal with images and sound.
Oh, wow.
Imagine AI that can see and hear and interact with the world.
Almost like a human.
Yeah.
Think about AI assistants that can understand you,
but can also read your expressions.
Oh.
Or robots that can navigate using vision, touch, and sound.
That's incredible.
The possibilities are endless.
Wow, that's mind-blowing.
What about this other concept, the inference dancing law?
Oh, yeah.
What's that about?
That's another cool research area.
It suggests that as AI gets better at reasoning
and problem solving, it might need fewer thinking steps.
So more efficient thinking.
Exactly.
Not just smaller size and energy usage,
but actually thinking better.
That's the idea.
Imagine AI that can solve complex problems in an instant.
Using minimal energy.
Yes.
Wow, that'd be a game changer.
Like in scientific discovery or medicine.
Yeah.
Or even everyday decisions.
We could have AI that's powerful.
But also fast and efficient.
Exactly.
And while this is still pretty new,
it could change how we think about AI completely.
It does sound like we're on the edge of something big.
Definitely.
But with all this focus on efficiency and smaller models,
does that mean the end for those huge AI systems?
That's a good question.
While smaller, more efficient models
are definitely becoming more popular,
there will always be a need for those large, powerful models
for some things.
So it's about picking the right tool for the job.
Yes, exactly.
Not just assuming smaller is always better.
Right.
And you know the world of AI development
is getting more diverse.
OK.
With more models being made for specific purposes.
So not a one size fits all anymore.
Exactly.
That makes sense.
Yeah.
So as we move towards this future of AI
that's more accessible and efficient,
are there things we should be thinking about as a society?
Oh, that's a great question.
Are there downsides to this trend?
Well, one concern is that these smaller models
could be misused.
In what way?
What if someone with bad intentions
got a hold of a powerful AI that could run on a phone or laptop?
Yeah, that's scary.
That could be really bad.
So how do we stop that?
Can we even control this technology?
It's tough, but it's something that researchers, policymakers,
and industry leaders are all working on.
OK.
One way is to develop strong ethical guidelines
and regulations for how AI is developed and used.
Right.
Another is to educate people about the potential risks
and benefits.
If you can make good choices.
It sounds like this needs to be an ongoing conversation.
Oh, definitely.
AI is always changing.
And we have to adapt as new challenges come up.
Well, this has given me a lot to think about.
Me too.
And hopefully our listeners as well.
It's clear that AI is moving towards efficiency
and accessibility.
Yes.
But like any powerful technology,
we need to be careful and make sure it's used for good.
I agree.
And its benefits are shared by all.
Definitely.
Definitely to keep an eye on this.
And to our listeners, we encourage
you to check out the paper Densing Law of LLMs.
Yes.
Good read.
And learn more.
It is.
Thanks for joining us for another deep dive.
See you.