Lutz (00:00):
Hello and welcome yet to another episode of Lutz and Jasper. We are here. It's a beautiful morning in San Francisco and beautiful afternoon in Berlin.
(00:12):
Today we are talking about RAGs. It's a relatively technical topic. Essentially a RAG is a combination of vector search plus large language model. We go into the technical details of what is vector search compared to simple search, we go into the details of why large language model is just an interface, we show that this might be a solution for models hallucination, and we show that this is absolutely needed for enterprise search to be useful. And we discuss the challenges which are still out there and, because of those challenges, we have space for a lot of niche applications as well as we will discuss the investment in Contextual AI.
(00:54):
Join us for another episode.
Jasper (00:55):
Now let's kick it.
Lutz (01:12):
Yes, RAGs. Essentially, we come always back to the same topic. Large language models, as much hype as there are, they are 'just' nice interface and just is in quotes because or cursive or italic or however you want to say it because it's huge that we have now an interface which any human can use.
(01:33):
If we start talking to text, documents, databases, we have the problem that the answer might not be 100% correct. We have this encoded. The listeners who saw my LinkedIn post and reacted to it, if you actually can easily ask a database now, and you can use OpenAI or ChatGPT with a interpreter, and that code gets actually generated, tested before it is then executed, because it might not be right in the beginning.
(02:09):
Now, for a code interpreter it's easy because there is an error if it's not right. But what is happening if we are retrieving some language human code text and that is wrong? That's what we call hallucination. What are we doing then?
Jasper (02:25):
Yes, and I think one other example is it's also not up to date. If you would ask the model who won Wimbledon women's championship, for today's data, you could get an answer because we know the women's winner, for the men's you don't know because they play tomorrow, after our recording. But the model would still give you an answer there. You need to make sure that the data is accurate, correct, up to date and this is not easy to do.
(02:51):
I wanted to tap on that challenge because we spoke about it before. You could probably solve this with an LLM if you retrain it every day with new data. But what happens then?
Lutz (03:03):
Very often in machine learning, and this is part of machine learning, you are right, is you have many ways to do one thing. It's similar to working with wood. You can use a chainsaw or you can use a pickaxe, all of those devices somehow cut but they are different. And if you want to look who has won some tennis championship, how about you search for it? And then LLM is an interface, as I said, a retrieval interface and it's not a search engine.
(03:34):
There are other tools which we can use for search. And the easiest tool is you take text, you tokenize text, meaning each word becomes a token and now you count the tokens, and then you make the map and say you wanted Wimbledon. How often does the word 'Wimbledon' come up in the text? Let's find a document which has a high count looking exactly for the word 'match.' And then, the next evolution was like, "Yes. But what if I didn't say 'Wimbledon' but I wanted to say 'Tennis'?" And then, if you don't have the exact bird matches, you actually start to do vector representations.
Jasper (04:11):
And I like that you want to tap into this a bit deeper. We spoke about vectorization, we spoke about Pinecone, Weaviate, vector databases receiving a lot of funding. It's probably interesting just to do a little bit of deep dive.
Lutz (04:24):
Totally.
(04:24):
Now, I could say, "You picture a tree, now we have a tree." If I write the word 'tree,' you're saying, "This is a tree." But what if I say the word of a tree, a birch, a pine tree. And I don't use 'pine'? In your mind, you know that this is a tree. Now, let's sketch out for a moment a two-dimensional space where you say, "Plants on the bottom and [inaudible 00:04:51] on the top." I need to change my arm movements here. And on the right is comic and on the left is real images, for example. And I'm doing this for images. Text is the same thing. You can imagine it for images better. Now, if I give you a tree, okay, a tree is a plant, it goes to the right basket. And depending on whether it's a picture or a comic, it goes to the right or to the left.
(05:17):
Now, you start being able to sort it. If I tell you I'm looking for a tree in comic style, you go to the right corner of your two-dimensional space, in this case, and start retrieving everything which is out there, which might be a pine tree, a birch, whatever. That is called the vectorization. You actually created a space, in the case a two-dimensional, the real space are way more dimensional so we can make as many dimensions as we can count, and you put everything in there which belongs there. When you take a Word document and in this Word document you talk about a tree or you wanted to talk about Wimbledon, then we say, "Okay, Wimbledon. What does Wimbledon stand for? Wimbledon stands for... Let me place this in the corner of tennis, as well stands for a city in the UK.
Jasper (06:15):
Yes, put it into different corners. But at the end of the day, if I get it right, this vectorization also means I make it readable, understandable, comprehensible for the model by putting this into numbers.
Lutz (06:27):
And it helps us to not have to work directly with search terms. I built my company Fisheye Analytics a long time ago and one of the big things which we did is we used a vector approach, Elasticsearch in this case, by now it's old news. But we married Elasticsearch together with SQL. RAGs are very similar because you marry now... SQL is a retrieval language, large language models is a retrieval effort and you marry it with a vectorial search, which is super exciting.
(07:03):
Let me explain why. If you look for 'Wimbledon' but you didn't type 'Wimbledon', but you type 'Tennis', then the vector search goes into the right corner and finds there, "Tennis. Close to tennis is Wimbledon. Let me read back Wimbledon as well." That is the way you do vector search.
(07:24):
Note that vector search is what we call a nonparametric search, meaning there is no parameters involved, you just define places in a dimensional space where you keep your information, and then you go to that space and you look around and you take everything you can find and bring it back.
Jasper (07:40):
Yes.
(07:40):
And I guess, for the listeners, if you use a search function somewhere, just put in something you're searching for and you didn't type it just exactly as it should be written and you don't get any results, then this is not in place. Although, it should be because everybody could do it nowadays. You should probably write an email to the company and say, "Honestly guys, what about vector search, Elasticsearch?" You can't change the LLM all the time. That's what we learned. It's way too expensive. You have to attach something to the model, let's call it like that, to make the prompt that I'm ingesting or asking, I'm actually asking the model something to make it better.
Lutz (08:17):
Life is like a box of...?
Jasper (08:19):
Pralines.
Lutz (08:20):
Wow, very good.
(08:21):
Okay, not 'chocolate' in this case because you trained your model with something else. Now, how do I convince you just that to use a different word than 'chocolate'? Because you say, "It's very traditional, people talk about 'life is like a box of chocolate.' This is what all my language model tells me." But I have new information and that new information has precedent or is more important and therefore I should use that piece in the new structure. And that's essentially what we are trying to do. We are using the large language model to create the language understanding to create a latent image, to put it somewhere in a space where we are saying, "This is the question."
Jasper (09:07):
I think that's a bit too complicated, right?
Lutz (09:09):
The word 'latent image', the latent representation is actually what deep learning is all about unfortunately. We have texts which we humans somehow understand. The large language model creates from that text a latent representation and, so far, large language models will then take this and do from that an answer to the question, no other additional information.
(09:33):
What they do now is they take the latent representation, go into the vectorial space which I described, grab information they see which is additional information and that information says, "Pralines are now the new thing, it is not chocolate anymore." And therefore they're saying, "Okay. I think normally life is like a box of chocolate, but in this case we are talking so much about the new marketing slogan from Jasper's new company about Pralines, therefore I grab the word and the answer is pralines." The language model takes information from a vectorial space and adds it to the answer and then does the same thing what language models are good at, converts this back into a complete sentence and tells you.
Jasper (10:20):
And the good thing is, if I understand that right, I'm a corporate or a smaller company, again, I can just attach this to a large language model of my choice and I make this better so I can use my own data and I can make sure that large language model, whatever it does in search, whatever it does in my prompting, it would focus on my data. Is that right?
Lutz (10:47):
No, and this is where we need startups.
(10:52):
As the initial idea came all about to saying we use large language model just as an interface and then we are using the vectorial search, nonparametric search, to look at the right information, and then we use a large language model again to distill that piece of right information back. That in theory would tell us, "There is no hallucination because it always uses the right information."
(11:18):
I did the following thing. I asked the uploaded data about healthcare data, and then I had them ask the question to the data. Now, the model would answer me with the right code by tapping into that data source, running it, write code, but as well would add on information. I ask, "What is the most statistical important variable?" And they would say, "Okay. This is how you calculate statistical importance. You have a T test, you have an F test, you have a P value." It explained all of this. That information did not come from the database. That information came from the corpus the model got trained on. The model in itself uses both sides, the trained side as well as the new side.
(12:00):
The question is how much weight does it give to each side, and how does it control that something is right or wrong?
Jasper (12:09):
Yes, because the big question is, just to remind everyone, there is a lot of input data in the context of LLMs, the whole internet. And then, I ask a question that maybe might be answered by, I just give one number, 0.001% of the data found on the internet, it's probably even much less. How does the model know it's exactly that and not something else?
(12:33):
But this is where the RAGs are supposed to help. They should even be possible to give me the source where they found this information. I think I read about, "Who is the President of the United States?", and then it says, "Yes. My data is from September 21," you remember the OpenAI issue there, "... But I found this new information on the website of the White House and the White House might be more right than the general internet."
Lutz (12:59):
True.
(13:00):
But here is we had a lot of discussion in our society about what is right and what's wrong, and meaning now the RAG needs to take a decision about what's right, what's wrong. I tried yesterday the following thing, I took OpenAI, ChatGPT, I made a document describing myself as Captain America.
Jasper (13:21):
Yes, I love that.
Lutz (13:22):
I actually used ChatGPT to write a document about Captain America and then I just replaced Steve Rogers with Lutz Finger. That's what I did. I said, "Lutz Finger is Captain America." It's a very nice document. It's out in the open, it's public. I published it and then I pointed ChatGPT to it and said, "Read this." And then I tried to get ChatGPT to actually say that Lutz Finger is a superhero. But ChatGPT says, "No, there is one document which you told me to read and I think that's fake." Then, ChatGPT had understood that there is more information about Lutz Finger being a professor or a President at Marpai and so on and so forth than this document. But guys, honestly, readers, look at me, I'm a typical superhero in a different metaverse, definitely.
(14:14):
Anyhow. But ChatGPT didn't want to go down that route.
Jasper (14:17):
Yes. And we discussed in the last episode the human factor, so there's a lot of quality control by humans. But in this case this wouldn't have been possible because no human would've had the time to read the internet quickly or anything. It must have been OpenAI in this context.
Lutz (14:32):
But there's a problem. What if I'm really Steve Rogers in the next Marvel movie? What if this just got announced? Let's come down to business because we have a solution now to use the LLM as interface, as we always said. We have a solution to look up data and like data in a vectorial space, which makes search pretty powerful. We do not yet have a very good solution to actually manage that difference. At what time do you actually want to Lutz be real superhero? Because that's the right information. Who is taking that decision? Is the LLM taking that decision or the RAG taking the decision that I'm not a superhero? Or am I as enterprise want really to say, "Lutz should be a superhero"?
(15:24):
There needs to be some control and observability and we do not yet have that capability. Amazing startups to come will help to create this observability, we will solve as well access control, different [story then 00:15:40] , and we will actually see how to manage the difference between what the corpus was trained on and what our corporate data says.
Jasper (15:49):
To your point, we also saw this in cloud adoption.
(15:53):
Okay. Now startups. As I was just alluding to, we had for example Datadog observability, you just mentioned it. We have the same issue here. There was cloud, people didn't know what was going on. Same with security with ORCA or these startups when cloud came up, and probably something will happen here as well.
(16:10):
We mentioned that we had around, there are many startups funded right now. Just one example, just quickly Contextual AI is a company, maybe the listeners want to check Contextual AI. They came out of Hugging Face and other companies. They want to address the issue of, "I want to control my model," and then actually also on the cost side using smaller models. Now, we discussed this a little bit before, Lutz, and you weren't so convinced yet, but I think it's also because there's not just a lot of information. But still, it sounds again we are in this discussion, "Will this be a commodity or solved by the larger platforms? Or is there really one company who can offer product for it?"
(16:56):
And I also thought Cohere and all the other enterprise large language model providers, they should also go into that space.
Lutz (17:03):
In my view, it has to be solved by the larger ones. If you are Microsoft and you have seen Google Docs and Google Drive taking on more and more over, what would be the first thing you would do? You would say, "All your documents which are in the Microsoft cloud are now becoming the core knowledge for your enterprise." I give you a way to ensure that you have now a super smart person, super smart tool, which is an LLM, which has read all the documents and know about which document they're supposed to talk to whom. And now you can access all enterprise knowledge at the right moment. You don't even have to ask the question because, when you ask something which is close to it without using the right words, in the vectorial space, that's the reason why we do vector search, it figures out the right document and gives you the feedback you need to have to take the right decision at the right moment with the right information from your company.
Jasper (18:13):
Yes. And that's actually pretty nice, because it's a faster search but I'm still in power and can read the document. It would might even highlight the part in the document that might be relevant for my question, and then I see the model answer but the model said, "Maybe you read it. And by the way, if the answer is wrong that I gave to you, please let me know so I could even be retrained. Thank you very much."
Lutz (18:37):
And therefore, for me, I think the use case makes perfect sense. The use case, absolutely, it's needed. What RAGs are aiming to do is highly needed. There are questions about, "How do I steer?", as we discussed, "How do I find the balance between pre-trained and the actuality of a document?" But this is all technical solvable. But who is best positioned to enter that space? These are the companies who already have today all your enterprise documents, meaning it's the Googles and the Microsofts of this world.
Jasper (19:13):
Yes. But what you can still do, probably if you think about it, now everybody starts talking about, "We need Vertical AI," and it's always the same. First comes the general application, then comes the infrastructure, then comes the tools, and now we're back to vertical applications. Still, there are Microsoft offers products that have competitors because they don't have very specific solutions, so it's always a bit more general. You could still build this probably in a case of blue-collar worker more on the production side, but that's then a product challenge rather than an AI challenge. Or would you disagree?
Lutz (19:51):
No, I actually agree because now you could... We said RAGs are awesome, but they don't completely stop hallucination. RAGs are awesome, but you have the difference in, "Is Lutz a superhero or not?" That is still an issue. And now, what we could see or what we will see is that we are overcoming this issue by more industry-specific solutions. For example, we had this amazing interview with Torben from LegalOS. That's an industry-specific solution. But he's doing exactly that. He's using large language model as an interface and he talked about Pinecone, which is a vector database which is exactly to do vector search. He uses vector search on top of Pinecone using the LLM as an interface, and then spitting out the answer with LLMs in the interface. That's a RAG. That's RAG 100%.
(20:51):
However, he knows that he has to give high weighting to the document more than to the pre-training. He knows that he has to have certain guardrails and access controls. As long as we have not solved them in a generalist approach, which would work for Microsoft and Google, there will be a lot of companies which can, and think about the typical applications, this could be a chatbot, it could be a translation effort, it could be content creation, "Write me a document," this could be question answering, and this could be even the develop business development tool. Those specific niche applications will exist and will thrive as long as we haven't solved the general problem.
Jasper (21:38):
Yes. And I also love, and maybe then we move more into questions you should ask yourself when you build something around it, I also love the documentation case. Yes, there are already some companies around there. You can imagine when it comes to code documentation, to your example of Microsoft, obviously someone is active there. But just think of your corporation, even maybe smaller companies and what documents you have, knowledge you have and who's doing the documentation for that. I hate documentation, personally. I would love to have some tool that just tells me, "Ask me a question. I will at least guide you to a certain space and then you can read again and dive deeper. But you don't have to search and read everything."
(22:20):
Now, when I want to get an answer to the question, LLMs and now RAGs, these guys are telling me I should use RAGs, probably the first thing that jumps into my mind is, "Why should I try it out?" And we discuss it various times. If you don't think about AI applications, your company, somebody else, some competitor might do, so you might end up with a problem. Still, this is a challenge. It's not easy, I would say.
Lutz (22:49):
I think the question is what is your business all about? If your business has some form of knowledge retrieval, some form of access to knowledge, then you probably are using already today Elasticsearch, you probably already are looking up data. Now, that looking up data, that Elasticsearch user experience might be that you have an analyst team, types in SQL code in order to generate then a new Elasticsearch, and there is a translation layer built between the person who has the question, the business person, then the person who transforms it into an Elasticsearch, searches the database, comes back. That translation layer can be replaced by an LLM. But if you have a very specific enterprise need on retrieving information from text, then RAGs are your thing.
Jasper (23:46):
And I also would say, even if your whole business is around clicking buttons, moving sliders, that's a way of inputting data, asking questions, it's just in a different form, and even that could be done by AI as an interface and then a RAG, because maybe you can ask even deeper questions. Now you just have a few buttons. But if you just ask the personas working on SAP, on Oracle interfaces, maybe they want to ask deeper questions.
(24:15):
We spoke about using MySQL through the AI as an interface for nontechnical users. All these things probably get around you have people in your team that could ask deeper questions that want to use the knowledge in the company more, that are frustrated by that. And every previous attempt around it was either too costly or too complex or would take too much time. The only question some people might ask themselves is, "Do I have to give my proprietary data to some LLM that then uses it and trains for it, and some servers somewhere outside of Europe?" That's at least our issue here. But you can already do local solutions. I heard about many companies. Just spoke to one last week that trained their own LLMs locally.
Lutz (25:05):
It might be worthwhile for listeners as a summary, if you think about an LLM writing code, SQL code, that's not a RAG. If you use an LLM to access your text document, then most likely it's a RAG. How does it work? LLM for SQL code, the LLM is trained in languages as well as one language like French, German, Italian, as well as one is SQL. It understands it should write some code in SQL, it writes a code and it might hallucinate and do bad code. But that code wouldn't execute.
(25:43):
What you do is you use the code, you execute it, you figure out there's an error, you feed it back and let it write again. Nothing visible to you, at the end the code is executable and nice, as an end user. For a RAG is, "Who is the President of the United States?" The RAG doesn't know, but it has a document where it can look something up and say, "Actually, let me look in the vector corner of presidents and grab the documents which I find there and summarize them in a way so that you understand it," then it's a RAG.
Jasper (26:15):
Yes. But I would say still that one question might arise, "Do any technical people in my company if I really want to build around this?" Yes, you probably need much less technical people, much less people in total than six, seven years ago. But it's still nothing that you could just do easily unless you probably work with a consultant, and there are very good ones out there, but yes, I guess my recommendation at least would be have one persona in your company that understands what this thing is doing.
Lutz (26:46):
That's an interesting comment. I would say it depends on who you are. If you're a startup and you're looking for funding and you have just one persona, I would like say, "Hm."
Jasper (26:54):
Yes, at least one. I said, "At least one." Come on. Yes.
(26:58):
All right. That's the end. Thank you very much everyone for listening up to this point. I would love to summarize a little bit.
(27:05):
What I enjoyed today discussing with Lutz is we discussed the next step of applying large language models in practice, so real applications. And it's all around how, not just to tame them, to make them more efficient, create better outputs, build an application that is for your specific company, for your specific use case. We explained a little bit how that works technically, didn't get into much more detail. We can provide you much more links, if you want to read this up. Just ask us that question.
(27:35):
Ask us more questions if you want to dive deeper into other topics, we are very happy to cover them. If you know anyone we should speak to as well, please intro us and then obviously we're always happy to speak about your startup idea.
(27:49):
Until the next episode. Thank you very much, Lutz, for this. And have a pleasant weekend. Goodbye.
Lutz (27:55):
Bye-bye.