Welcome back everyone. Today we're going to take a deep dive into the world of cutting edge AI.
Sounds exciting.
It is. It is. Specifically, we're going to be focusing on two Chinese companies, DeepSeek
and Moonshot AI. They're really making waves in the AI scene. And, you know, our main source for
this deep dive is a document called Team of 20 Interns. It's a fascinating really, gives you
real inside look at these companies. Oh wow. Team of 20 interns. That's interesting. It is. And what
really struck me is how these relatively small teams, you know, mostly recent grads, even interns,
are actually going head to head with giants like Google and Microsoft in this AI race.
Yeah. That's wild. Like, how are they doing that? Well, that's what we're going to find out. That's
the whole mission of this deep dive. Okay. I'm all ears. So to kick things off, DeepSeek, right,
they've kind of come out of nowhere and just disrupted the entire AI landscape. I mean,
they start to price war in China with their V2 model. Oh yeah. I remember hearing about that.
Like, big players like Tencent and Alibaba were forced to lower their AI prices drastically.
Exactly. There's a huge shock to the industry. It's almost like what happened with Pinduoduo,
you know, in e-commerce with their whole discount model. You know what? I was thinking the exact
same thing. I read somewhere that DeepSeek even got the nickname the AI Pinduoduo. It's a pretty
apt comparison, I think. Yeah. I mean, the question is how are they able to offer such low prices and
still, you know, actually make a profit? That's the million dollar question. It is. And it all
comes down to their approach to AI model architecture. They've got this system called
memory efficient local attention or MLA for short. Okay. MLA. Sounds pretty technical.
It is, but it's the key to their success. So most AI models use a mechanism called multi-head
attention, MHA, to process information. Think of it like the AI's way of connecting the dots
between different parts of data. The problem is MHA takes a lot of memory to run.
Okay. So DeepSeek's MLA is like a more efficient way to connect those dots.
Exactly. Instead of trying to process all the data at once like MHA does,
MLA focuses on smaller localized chunks. Hmm. Okay. So it's not trying to do everything at once.
Right. It's more like reading a book one paragraph at a time, absorbing it fully,
and then moving on. Whereas MHA is trying to take in the entire book all at once.
Ah, okay. So you get the depth of understanding without overloading this.
Precisely. You got it.
But does that ever like backfire? Are there situations where this
hyper focused approach, you know, might not be as effective as the traditional way?
That's a great question. While MLA is like really good at processing sequential data, like text,
there might be instances where that broader context provided by MHA is beneficial. So
for things like complex image recognition or understanding subtle relationships across
huge data sets, MHA might still have an edge.
So it's about choosing the right tool for the job, basically.
Exactly. It's a trade off.
It seems like DeepSeek is really betting on MLA being the right tool for a lot of jobs, though.
I mean, they've even implemented this other innovation called a DeepSeek-MoE sparse design
to make their models even more streamlined.
Oh yeah, DeepSeek-MoE, which stands for mixture of experts.
It's a pretty clever system. Basically, different parts of the model specialize in different tasks.
So like division of labor.
Exactly. Specialization. And because of that, not every part of the model needs to be active
for every task. It's much more efficient. The sparse part of the name means that only a small
subset of these experts is used at any given time, which, you know, further reduces the
computational load.
Okay, so they're not just optimizing how the AI processes information.
They're actually optimizing the structure of the AI itself.
Exactly. And that's how they can offer such competitive pricing.
Amazing. But it's not just about cheap chatbots.
Right.
They're founder, Liang Wenfeng. He's got his sights set on something much bigger.
Oh, absolutely. He's talking about artificial general intelligence, AGI.
AGI. That's the holy grail of AI, isn't it?
It is. And Liang Wenfeng really believes it's the ultimate goal. But his approach is different.
He's not just chasing trends or, you know, trying to copy what others are doing.
So what's he doing differently?
He's focused on building a solid foundation through architectural innovation.
He believes that true progress comes from fundamental changes in how these AI systems
are designed. And that's what DeepSeek is all about.
So they're not just building a better chatbot. They're trying to build a better brain.
It's a great way to put it.
Okay, so that's DeepSeek. Now let's shift gears a bit and talk about Moonshot AI,
the other player in this AI revolution. They've developed something called Kimi K1.5,
a powerful multimodal AI.
Yes, Kimi K1.5. It's making a lot of noise in the AI community.
Multimodal AI. What exactly does that mean?
Well, it means it can understand and process different types of data, like text, images,
code, all at the same time.
This is like an AI with multiple senses.
Exactly. And to get this, Kimi K1.5 has actually
been outperforming some of the biggest names in AI, like GPT-4, and certain benchmarks,
especially math and coding.
Wow, really? That's impressive. So it can juggle different types of data,
and it's a math and coding whiz. What's the secret to its success?
Well, one key factor is its massive context window.
Imagine the AI's memory like a container. Kimi K1.5 has a huge container, so it can
hold and process a lot more information compared to other models.
It's like giving it an elephant's memory.
Aha, yeah, something like that. We're talking about 128,000 token context window.
That's massive.
No wonder it can handle such complex problems. And what's even more remarkable is that they've
made this powerful tool accessible to everyone through a free chat interface.
That's right. Anyone can experience its capabilities firsthand.
That's incredible. And speaking of access, both DeepSeek and Moonshot AI have embraced
open source methodologies, which is quite different from how a lot of AI companies operate.
Oh, absolutely. It's a major shift from the traditional approach of keeping everything
under lock and key.
It seems like they're not just competing with the big players.
They're actually challenging the entire way the AI industry works.
Exactly. And the impact is already being felt globally.
DeepSeek's success, especially their low inference costs, has sent ripples throughout the industry.
Yeah, even seasoned investors are trying to figure out how these smaller companies are
changing the game.
It's a real shakeup. And it raises a lot of questions about the future of AI.
Will these smaller, more agile companies, with their open source approach,
ultimately reshape the entire landscape?
That's a fascinating question and one that I think we'll be exploring further as we delve
deeper into the specifics of these companies and their AI models.
But before we move on, I'm curious to hear your initial thoughts.
What stands out to you most about DeepSeek and Moonshot AI so far?
Honestly, what intrigues me most is their shared belief in the power of open source.
They're not afraid to share their knowledge, and they believe that the true value lies in
collaboration and collective advancement.
It's a refreshing perspective in a field that's often so secretive and competitive.
It's definitely a bold approach, and it's clearly making waves.
So let's delve a little deeper.
Let's start by taking a closer look at DeepSeek's revolutionary model architecture
and how their MLA and DeepSeek Moa designs are achieving such remarkable efficiency.
So let's unpack this DeepSeek's model architecture a little bit more.
We talked about memory efficient local attention MLA earlier, but it's worth really digging
into how it actually works.
Traditional multi-head attention, MHA processes all the data at once, and that requires a ton
of memory.
MLA, on the other hand, breaks the data down into smaller, more localized chunks.
So instead of trying to grasp everything all at once, it's more like focusing on
smaller details and then putting them together.
Exactly, and that approach has some big benefits when it comes to efficiency.
MLA only needs to focus on a small portion of the data at any given time.
Right?
So it uses way less memory than MHA.
Like, it can achieve the same level of accuracy using, get this, only 5% to 13% of the memory
footprint.
Wow, that's a huge difference.
It is.
It's huge in terms of both cost and speed.
Okay, so they're making the AI, I think, smarter, not harder.
Yeah.
But doesn't focusing so intently on those smaller chunks risk missing the bigger picture?
Are there any downsides to this approach?
Yeah, that's a valid point.
While MLA is great, incredibly efficient, actually, for tasks involving sequential data
like text, it might not be the best solution for every situation.
There are times when understanding the broader context is crucial.
Okay, I see.
So for analyzing an image with a lot of details, or maybe understanding connections across a
huge data set, the traditional MHA approach might still be better.
Yeah, exactly.
So it's about knowing the limitations, right?
Right.
And it seems like DeepSeq is making a calculated bet that the benefits of MLA outweigh those
potential downsides.
And they've actually doubled down on this efficiency drive with their DeepSeqMo sparse
design.
Right, right.
The DeepSeqMo, we talked about that earlier.
Yes.
DeepSeqMo, or mixture of experts, adds another layer of specialization to the model.
It's like, imagine you have a team of experts, and each one specializes in a particular area.
Instead of having everyone work on every problem, you just assign tasks based on their expertise.
Delegate, delegate.
Exactly.
And that's essentially what's happening in the DeepSeqMo system.
Different parts of the model are trained to handle specific types of tasks,
so only the relevant experts are activated for a given input.
So it's like they built an AI with a specialized workforce.
Yes.
Ha-ha.
That's a good way to put it.
And the sparse part means that only a small subset of these experts is used at any given time.
So that reduces the computational load even further.
OK, so they're not just optimizing how the AI processes information, but also the entire
structure of the AI itself.
Precisely.
And this focus on efficiency, it's not just about saving money.
It's about opening up new possibilities.
By reducing that computational burden, DeepSeq can train larger, more powerful models that
can handle even more complex tasks.
So it's a long-term strategy.
It is.
And it aligns with their ultimate goal, which is pursuing artificial general intelligence, AGI.
Right, AGI.
I'm glad you brought that up again.
It's easy to get caught up in all the technical stuff and lose sight of the bigger picture.
Yeah, it's important to keep that in mind.
And DeepSeq's founder, Liang Wen-Fun, he seems to be approaching AGI in a very,
I don't know, almost philosophical way.
Oh, absolutely.
He's not interested in quick wins or building a chatbot that can trick people into thinking
it's human.
He wants to create systems that can truly understand and reason about the world,
just like humans do.
And he believes that the key is building that strong foundation through architectural innovation.
So they're not just trying to create a better version of what we already have.
They're aiming for something fundamentally different.
Exactly.
And it'll be fascinating to see how that plays out.
Now, let's switch gears for a bit and talk about Moonshot AI and their Kimi K1.5 model.
We mentioned earlier that it's a multimodal AI capable of processing different types of data
at the same time.
Right, right.
But what does that actually mean in practice?
Well, imagine an AI that can read a research paper, analyze a graph, and write code all at the same time.
That's pretty mind-blowing.
Is that even possible?
It is.
And Kimi K1.5 is proof of that.
This model has shown some amazing capabilities and tasks that require, you know,
combining information from different sources.
For example, it can accurately identify and extract numerical data from images,
something that most language models just can't do.
Wow.
So it's not just understanding different types of data separately.
It's about connecting the dots between them.
Exactly.
And that ability to process information holistically is a big step towards creating AI systems
that can understand the world the way we do.
Because we don't experience reality through just one sense.
We use all of our senses to build a complete picture.
Right.
And Kimi K1.5 is, you know, a step in that direction for AI.
So even though it's not specifically designed for AGI,
it's still contributing to that larger goal by pushing the boundaries of multimodal understanding.
Absolutely.
And Moonshot AI's commitment to open source development makes that contribution even bigger.
By making Kimi K1.5 freely accessible, they're empowering researchers and developers all over the world
to experiment with it and explore new possibilities.
Speaking of open source, I read that Kimi K1.5 actually outperformed GPV4 and Claude 3.5
in some pretty tough math and coding tests.
How's that even possible?
Those are like the big names in AI.
It is remarkable, isn't it?
Part of Kimi K1.5's success is because of its huge context window.
Remember, it can hold and process incredibly long sequences of information.
And that's really helpful for tasks like code generation,
where you need to understand how different parts of the code are connected.
So it's like giving the AI a much bigger working memory.
Exactly.
But that's not the whole story.
Moonshot AI has also used some clever techniques during training to optimize its performance.
They've used methods like rejection sampling and partial rollout.
Those sound pretty technical.
Can you break those down for us non-AI folks?
Sure.
Imagine you're teaching someone to play, I don't know, a musical instrument.
You wouldn't just let them play random notes and hope for the best.
You'd guide them, correct their mistakes, and encourage them to try different approaches.
You even feedback?
Right.
And that's kind of what rejection sampling and partial rollouts do.
They help the AI explore different solutions during training and learn to pick the best ones.
So it's like having a virtual music teacher for the AI.
Yeah, that's a great analogy.
And by refining that training process, Moonshot AI has created a model that can handle some really complex tasks.
It's fascinating to see how both DeepSeek and Moonshot AI are tackling AI development with such different yet effective strategies.
It really highlights the diversity of thought in this field and the incredible pace of innovation.
Now, let's put these two impressive AI models head to head and see how they stack up in some real-world tests.
Okay, so they designed a bunch of tasks to see how these models performed in different situations.
One test involved analyzing images like tables or charts that had numerical data in them.
Kemi K 1.5, it aced this.
It picked out the text and numbers without any problems.
DeepSeek's R1 model, though, had a little trouble.
It mixed up some of the values.
So Kemi K 1.5 is pretty good at understanding visual information, then.
What about language-focused tasks?
How do they handle those?
Well, they had this web search challenge.
The goal was to find red gowns that cost under $200.
Okay, sounds pretty straightforward.
It does.
But DeepSeek R1, it returned a bunch of links, but some of them weren't quite right.
They weren't red bounds or they were over $200.
So it didn't quite get the search parameters.
Right.
But Kemi K 1.5, it delivered two perfect links.
And it even gave additional options in a separate panel.
It really understood the color and price limits.
Impressive.
Did they test how the models handle multiple documents?
Yeah, they gave each model a set of files and asked for a summary of the information.
Kind of like a research assistant.
Exactly.
And Kemi K 1.5, it managed to summarize at least two out of the three files they gave it.
DeepSeek R1, however, struggled with this unless the files were given one at a time.
So it seems like handling a lot of information from multiple sources at once is a bit of a
weak spot for DeepSeek R1.
What about coding?
How do they do with that?
Ah, the coding challenge.
They tasked them with creating a Snakes and Ladders game.
And this is where DeepSeek R1 really stood out.
It made a more advanced game with clearer features and it was actually playable.
Oh, wow.
So it actually made a game.
It did.
Kemi K 1.5's version, it worked, but it was simpler.
And it had some bugs that let the pieces move off the board.
So DeepSeek R1 might be better at those complex coding tasks.
Did either of them actually get the Snakes and Ladders logic right though?
Well, neither one fully implemented it.
The gameplay was basically just moving pieces around randomly, which just goes to show
that even with all these advancements, AI still has a long way to go.
That's a good point.
It's easy to get carried away with all the hype and forget that we're still on the
Earth's stages.
But even with their limitations, these models are pretty incredible.
Oh, absolutely.
And these tests, they weren't about finding a winner or loser.
It's more about understanding the strengths and weaknesses of each model.
DeepSeek R1 is a coding whiz, while Kemi K 1.5 shines in web searching, image analysis,
and summarizing multiple documents.
So different tools for different jobs, basically.
Right.
And what's really cool is that both of these companies are all in on open source.
They're all about transparency and collaboration, which is pretty refreshing in an industry that's
usually, you know, all about secrecy and competition.
Like they're building a community, not just tools.
Exactly.
DeepSeek believes that real progress happens when you share knowledge, not hide it.
And Moonshot AI seems to agree, considering their open access to Kemi K 1.5.
Yeah, they're fostering this whole ecosystem of developers and researchers who are contributing
to its development, so it's constantly evolving.
It's a really exciting time for AI, that's for sure.
It is.
And it'll be interesting to see how these new open source models change the game even more.
I think DeepSeek and Moonshot AI have definitely shaken things up.
Their success, especially DeepSeek's low costs, has got everyone's attention.
Yeah, even the big companies like Microsoft, Google, and Meta are taking notice.
They're having to rethink their strategies to stay competitive.
And the investors are probably watching this all very closely too.
Oh, for sure.
This is a big deal.
And not just for the corporate world.
We're seeing more and more support for open source AI all around.
It's like a movement.
It is.
And the core of it is this fundamental question, who will control AI?
Will it be a few powerful entities or will it benefit everyone?
That's a big question and a really important one.
I think the work that DeepSeek and Moonshot AI are doing gives us some hope though.
They're proving that innovation can happen outside the big corporations and that collaboration
and openness can lead to amazing breakthroughs.
It's a powerful message.
Yeah.
And a really inspiring one for anyone who believes in making technology accessible to everyone.
Well, folks, that brings us to the end of our deep dive into the world of cutting edge AI.
We've explored the rise of these two incredible companies, DeepSeek and Moonshot AI,
and we pondered the implications of their open source approach for the future.
We'd love to hear your thoughts on all of this, head over to our website, and share your reactions.
Does this new wave of OP source AI inspire you?
Does it make you nervous?
Let us know.
And until next time, keep exploring, keep questioning, and keep diving deep.