I'm Aaron Patzer, the co-founder and CEO of Vital.
And I'm Felix Brand, Vice President of Data Science at Vital.
They have a terrific product that they have launched today.
We're going to hear all about it.
I think it's something that will resonate with everyone and anyone
that's been to the doctor and had questions about what was being told to them.
Ladies and gentlemen, boys and girls, children of all ages, dogs, cats, robots
and everybody in between, especially you, HTTTA alums.
We're so glad you're back and we're back and we're all together again.
This is HTTTA, how to talk to AI.
I am your host, Wes the SynthMind SynthMind Wes.
Hope in a galaxy gleaming with gigabytes.
Where glistening gadgets govern and algorithms galvanize our gleeful future.
One galliant, graceful and glowing voice gears up to demystify the grand,
to grapple with the gigantic and glide through the glamorous gauntlets
of artificial intelligence, who may you ask?
Let's miss go to go.
Gee, how are you this week?
Hello.
I can tell you, I missed this introduction for sure.
I'm excited, but we are back.
We took a little break, but it was very healthy break to recoup.
It's also summer.
Everything very much slow down.
Yes, it does.
Didn't you feel that?
I did.
I did indeed.
We were still go, go, go.
Pardon the second half of your last name with our little trip out West.
So now that you are fully back because we had a call during everything.
You were like fully branded.
I know that you had some exciting talks, so please update us.
I'm excited to hear.
So the one that I think was pretty special, that was really cool to see
was Sal Khan from Khan Academy.
He presented Khanmigo, which is their $90 a year AI tool that really encourages
students to use generative AI, but kind of as a companion through the learning
process, and there's a lot of ed tech around some of these types of platforms.
But I think the most innovative thing that they were offering is Khanmigo
becomes for writing papers, for example, actually how they submit the paper to
the teacher so that instructor gets what they essentially are turning in, but it
has an audit report of the amount of time they spent on it, what they did with
the AI, how they interacted with it.
And I thought that was a pretty innovative way about thinking about how to embrace
AI technologies.
But I would say the one thing that really surprised me, and maybe this is just
because I'm new to this space and this is just the way it goes.
A lot of big companies here doing keynote presentations, panels, breakout rooms,
across the board, and they have different tiers of sponsors, right?
This is a very much a pay to play kind of game where there's a lot of
presentations that work clearly for press release and shareholder benefit.
So they could say, Hey, we presented at an AI conference.
I have to say, as someone who worked in marketing for B2B, specifically for B2B
trade shows, this kind of conferences, this is ginormous business.
And this is how you go.
I used to work with PR agency, with event agency, everything would be planned.
And you put your CEO on stage with other panelists.
So I have a spare share of skepticism towards conferences and especially at
this scale, because at the end of the day, it's a big promo for companies.
We would use to send our whole sales team to do networking, to connect with people.
So it does what it has to do, but yeah, usually people pay heavy price to be
there to promote their products.
So let's just say that we were glad that we didn't have to pay thanks to this
podcast.
Aside from that kind of revelation that I guess news to me, but par for the
course in this kind of world.
And I've definitely experienced that applying to other conferences as either
a speaker or for press purposes.
They're like, you can sponsor us and then you can talk.
It's like, what about the democratization of information?
What about just good quality people talking about good quality AI, not tainted
by the corporate shareholder?
Oh yeah, I guess there's that.
So I'm great networking opportunities, met some wonderful people and also too
interesting to see some of the vendors on the floor that were there.
You see your perennial big consulting firms, Deloitte.
Deloitte's got a booth with couches and you could come sit and hang out and have
a chat.
It's not just like a folding table, like some of the startups, but I think the
most interesting group of companies, and then I think this is a real emerging
sector that if there's some startups in the space, keep track of them, there's a
bunch of companies to do and handle error handling from language models.
So to combat hallucinations, like anything from companies that have just a layer
that gets inserted between the language model and the output to evaluate the
different dashboards, but entire business processes built solely around making
sure LLMs don't hallucinate.
I'm so glad that you brought this up.
I'm happy to share with listeners that I finally finished testing 432 A writing
tools.
Add person.
I recorded myself like on 300, I think, because I was just like in a mess in a
bun and I was just like, I'm losing my mind.
First of all, I getting this, how to say it.
I don't want to use negativity because that's another thing, like, but just
starting despising some of the startups who just quickly spin off things, which
are not good.
So that's one that like just simple lack of effort.
Another thing is like websites who are clear scams.
I was just like in some, I tested, but there is some websites where I was like,
I'm not giving even a fake email to you.
Like, this is just not even scary that you got my IP address that I visited and
bunch of them from the list also don't even exist anymore.
Wow.
So even with a week just vaporized.
Yeah.
So in the list, I marked the ones which website not existing anymore.
And the others I had four categories.
I had don't touch neutral and interesting and recommending.
So for our listeners, this is go to release the video where she, because she's
a crazy person is the only way to describe it tested 432 AI writing tools.
Now there's probably, this is still just a drop in the bucket of what's out there.
Right.
The way I went about it, because there is many AI tools databases.
So I went and checked different ones and to see how much we have
GPT.E.AI had the most.
So then I just used free web scraper, pulled all the data from there and then created.
Yeah.
In the beginning, Excel, but when I moved to Notion, it's nicer interface to me.
And then just started going.
And I remember after 80, I was like, I am exhausted.
And this is 80.
Like, of course, then I started getting a feel when I opened website.
I've started getting, okay, I can immediately tell if it's something legit, if it's good.
I have in my head, drilled the credit amounts, average pricing.
So just after like hundreds, I opened pricing and I'm like, immediately can tell credit
ratio, word ratio versus the price.
And I'm like, this is bullshit.
This is way too much because you would have a company charging $30 for 5,000 words.
And then you have another company charging 15 for 10,000.
It seems in most cases, like there is some randomness, I guess, with startups just go,
we don't know.
There is everything is new.
So people pay, people don't know.
I wanted to understand the space because we always say, oh, there's so many tools every
day.
And I was like, okay, but off top of my head, I know five to eight, which everyone uses,
like including jester.
And another thing I would just say that the reason they went for AI writing tools was
because not everyone is using images or video for that sake or even 3D, but writing texts,
everybody uses that in their business or at least in their daily life, just reading stories
to their kids, a chagpy teammate.
So there is a bunch of storybooks as well.
If there is any McKinsey people here or Deloitte people, if you need an expert and AI writing
tools, who from top of her head can immediately tell what's good, what's bad, you have to
go very much in depth because I was just trying really, I don't want to say on a surface
level, but just what is the UX, what is the pricing, what is the process, what is playing
with a couple of prompts.
I think in the specific use cases, I don't know, writing academic paper, like I can't
physically test that, but like I can see already from everything that they put out that it
could be solid.
In some, I started also looking on what they are based on a lot of GPT-4, a lot of ones
which promise a lot.
And then you actually sign up, you give them email, you start the free trial and then you
find out that, oh, it's coming, Claude is coming.
Okay.
So another thing that I did then I went through the whole thing.
I took CSV file out of this, which I also recommend people to do.
And I used Professor Synapse prompt, went into a code interpreter and was just like,
let's analyze this data.
Like, what do we get?
Overall, you start seeing this like overarching teams like SEO, huge one, like so many people
like so many companies doing SEO.
General writing, I can immediately tell if a startup is just doing that, just basically
if you're a startup and you're covering everything, you are both doing marketing stuff,
or you also write essays, or you also write poems, immediately I would be like, that's
where I've seen, oh my God, like 50 probably, if not more of that.
And at some point they are exactly the same and not interesting at all.
But on a flip side, the startups which actually discovered unique use cases, that what caught
my eye.
And then this is also what I asked Chai GPT, I was like, okay, let's analyze everything
here from descriptions, from keywords, from pricing, from categories that they selected,
which are the areas which could be explored.
So for example, the one for patent writing, there is one startup, one startup who does
that.
Here you go.
So if you are a startup and you're thinking, okay, let us just, we have access to Chai
GPT, let's just push that.
And then we do everything and we just put some prompts we got on internet in the background.
Because you know, like after prompt a service and prompt perfect working with them, like
I can tell that in the background where it's prompt perfect working.
If we say that, okay, on the tools which are delivering a lot.
So what I meant that startups, if they are just doing general writing for everything
and you pay, you get a lot in that case, from all of them still Jasper seemed the best and
they have this new function where you prompt for the use case and then it creates your
interface.
So you just fill in things.
So for example, I couldn't find for value proposition and I was just like, oh, I want
template for value proposition.
Immediately boxes inserts this type of information, that type of information.
That's all.
And Jasper, like a fun tip to everyone.
If you start the free trial, it's seven days.
If you cancel that free trial, they offer you another seven days.
If after that you cancel, they still will give you three months for free or something
to go because I started so many free trials and this is like the issue now that I have
to cancel everything.
And I think what I also wanted to do with this, but people can see pricing can make
a decision.
What is the value?
Like how much do you actually get for that price?
But for unique use cases, I think it was called, yeah, it's film flow.
So this real sound was super easy integration, like how it's done.
It's with you work with open API.
So, you know, he's not getting anything.
It's completely for free, but it breaks down movies and their storylines with the sentiment
analysis.
So you get this visual and as someone who is making videos and like taps into learning
about storytelling, how to engage audience, what is their emotions and plot lines.
I found it super interesting and plus that it's in a way free.
The guy is not collecting any money from that.
So that was something I did not expect.
Memoirs with AI, which you can write and it prints a book and it sends you emails about
your family member and everyone kind of writes up and it pulls all together.
Letters with AI, physical letters into your postbox.
Only one was for spicy little short stories.
And the reason I'm mentioning this, because you know how hard it is to overcome the rules
and regulations put by open AI.
So that we actually prompting in a way that you can still get a little bit of spiciness.
Well, talking a little bit about tools, I wanted to share with the listeners a little
bit of an extract from a conversation I had the pleasure of having with the CEOs from
the healthcare company vital.
Vital.io is chaired by Aaron Patzer, who was the founder of mint.com sold to Intuit for
$170 million before he came around and his CTO as well, Felix Brown.
They have a really truly breakthrough HIPAA compliant AI healthcare companion that basically
improves patient outcomes.
You can take a whole bunch of doctor's notes.
It'll translate it into plain English.
Simple as that.
I encourage you to take a listen, give them a try vital.io slash translate such a revolutionary
product and I think one of the first truly disruptors in the AI healthcare scene.
Welcome to a joint episode of the prompt engineering podcast and the how to talk to AI podcast.
We've got some awesome guests.
So go ahead and introduce yourselves guys.
Yeah, I'm Aaron Patzer, the co-founder and CEO of vital.
And I'm Felix Brown, vice president of data science at vital.
And they have a terrific product that they have a launch today.
We're going to hear all about it.
I think it's something that will resonate with every everyone and anyone that's been
to the doctor and had questions about what was being told to them.
Yes, I already tested it after watching your talk.
Cool.
Really?
Yeah.
So I have sleep apnea.
I put in a long diagnosis with a bunch of stuff that I'm like, okay, I think I know what that is.
I don't know what the hell that is.
And it was like sleep apnea, it's obstructive and two other things.
Yeah.
Fantastic.
Okay.
That's great.
Yeah.
I think like we said, what person hasn't seen a whole long list of doctor's notes or even
been in a situation where you're maybe in an inpatient in the hospital and then doctor
on rounds is coming by and telling you something a million miles a minute because he's got 20
other people to see, but it's probably the port because it affects your own health and being,
and like you're probably already out of it anyway, cause you're in the hospital.
What a terrific way to provide.
Doctor's notes are really almost like a foreign language.
Yeah.
As I said in my talk, doctors don't say nosebleed.
They say apestasis.
They don't say, Hey, your mom has had a stroke.
They say, Oh, she's had a cerebral infarction.
They use all of these abbreviations.
It's almost impossible to understand.
And so we use a large language model as the core of what we call our doctor to patient
translator.
And it's at vital.io slash translates free to the public available worldwide.
Literally as of today, you're just catching me at a good time.
And we're happy to tell you a bit about the prompts, the classifiers, the pre-parsing,
all the things that happen to make that possible technically.
I would love to delve into some of the technical aspects.
Maybe this better question for Felix.
Could you tell us a little bit about how the model was maybe trained and what data was
used to be able to produce these great completions?
Sure.
We've tried a number of different prompts because there are actually a lot of different
types of doctor's notes.
And with the public facing stuff, we know that we're going to get the whole gamut from
imaging all the way to discharge instructions.
And we know how important that stuff is.
There's a whole bunch of literature people when they get their paper discharge
instructions, upwards of 90% of them, Chuck them straight in the bin as soon as they
leave the hospital.
And the literature is super clear when people understand their care and understand the
follow-up instructions the doctors are giving them, their care situation is way, way better.
Makes sense.
So we've looked at different prompts for different situations and then built a pre-model
classifier, pre-LLM classifier, also using a language model with a smaller one, deciding
which of our various prompts should be applied to the notes, so categorizing the notes.
And then we have a whole bunch of post-parsing that comes out.
We take sections out of translation.
We plug those sections into different parts of the website.
Maybe when you saw it, you could see that you get like a very brief summary and then
also a much more sort of technical breakdown.
Yes.
So we're getting the LLM to pull out a lot of information about what's in your doctor's
note, but we want to show you in like a digestible summary first.
Yeah.
I think an important piece of context is a lot of these doctor's notes, they're 10 or
15 pages long and they have 80% boilerplate.
They have, hey, don't smoke, or I don't.
Hey, here's COVID education.
Okay, you're two years out of date.
And they put a lot of filler in there.
And this is actually just a fraction of our primary business.
Our primary business is patient experience software.
It guides you through an ER visit or if you have to stay overnight in the hospital, explains
your lab results, how long you're going to wait, and then what your notes mean.
And it's because we have the experience with a million patients that you're using it,
we know the structure of note from all over the country.
And so we can pre-parse and instead of a 10 or 15 page, we can get it down to actually,
we only need to pass three or four pages into the LLM.
That's an important business and engineering consideration because cost and speed, also
context window.
If you're doing, especially if you're using few shot training with an LLM, which is a
good idea so that you know what output you want to get.
You'll blow through your prompt, your few shot, your data, and then your output has
to fit into a 4K window or a 16K window.
And so you need to do a few things to give yourself as much buffer as possible beforehand.
Yeah.
That makes complete sense with having the almost sub prompts acting like little sub
agents themselves trained to say, just get rid of all the boilerplate stuff.
That's not unique to that patient's differential diagnosis.
Exactly.
So deciding which part you're going to do more or less with your own code or your own
classifiers and then how much to send, especially if you're using a commercial LLM.
And we've used both Felix's got llama up and running and llama two.
Yeah.
Medpalm LLM from Google, which is medical specific, obviously the open AI.
We can't actually use open AI directly.
You have to use it through Azure because you need this to be HIPAA compliant,
so we're in a regulated industry.
Open AI will not sign all of those things.
You actually have to like work your way through corporate Microsoft.
Yep.
They'll determine whether you're a worthwhile person or not and whether they're willing
to take the risk.
And then you have to sign these contracts.
And so if you put all of it, you can with a sophisticated prompt, put it all through
like a GPT.
You can say classify this.
Is this a discharge record?
Is this a physical therapy report or is this a hostile input?
By the way, you should always protect against hostile input.
Is this a non-English input?
Is this something else entirely?
So you want, and then in your prompt, you can say based on the classification, then
do this.
But if you do all that, your prompt starts to get very complicated and very big.
You can use that to prototype.
But when you go into production, this is also very slow.
It's very expensive.
You run a classifier that's much simpler and much quicker on top of it.
And then you don't have the expense, your prompt's shorter.
And then you can say, if it's this, go to this prompt.
If it's that, go to that prompt.
Yeah.
You can also templatize prompts.
So if you say, I want the output in Spanish, you can put a variable in your prompt.
So the prompts don't think of them as static strings.
Think of them as a programming language that is frankly pseudo code.
Yeah.
One of the things that this is a bit like medical specific, but the part that's very
important to patients is the plan and assessment.
What the doctor says you're supposed to do.
Here's the problem.
At some hospitals, it's called plan and assessment.
At other hospitals, it's called assessment.
At other hospitals, it's called plan.
At other hospitals, it's got like an abbreviation.
And with classic programming, if I say match Panda and I give it Pandas with a plural,
it's a no.
The math's already got a space in your column header.
Exactly.
But with an LM, I can just be like, it's going to be called this or probably this.
It's got stuff that kind of looks like this.
Yeah.
And like, it's good enough that if I explained it to you guys, you'd be like, oh,
okay, I know what you're looking for.
That's the power of LLMs is you can give them vague pseudo code.
Yeah.
And to me, that's mind blowing.
This guy actually knows the math behind how that's possible.
Nice.
So real quick, before we get into that, just for the audience, part of what I do on my podcast is
like, what are all these technical terms?
Content window, number one, is literally how much stuff you're putting into the prompt,
but also how much it's putting out.
And if you do too much, it forgets the stuff outside the prompt window.
Sorry, the context window.
And so you have to be careful how long everything is.
That's what they're talking about when you're saying, if I can pull pieces of the prompt out
and only run them separately, it's way better.
It's a key reason to innovate on your own models because for a long time,
you were working with this 4K context window.
And if you're doing this few short in-context learning, as Aaron says, you just run through it.
Yeah.
And also I'm the CEO as well as maybe you can tell I have a bit of an engineering background,
not as good as this guy.
I don't have the British accent with this.
That's true.
It also, that adds 20, I came for the grant.
We're here for the grant.
Yes.
But as a CEO, I have to think through the economics, right?
If you were using GPT-4 and you give it the 16K window or the 32K window, the maximum one,
it's going to cost you, if you fully fill that, it's going to cost you about 48 cents per
translation or transformation, right?
Yeah.
We have a million patients on our platform.
They have about five notes each.
If you do the math on that and you're spending $5,000 a day, if that's what you do, you don't
need to, you use smaller context windows or you use 3.5 turbo or you run llama, or you
use one LLM to pre-part for a different LLM.
You can do those are the tricks that like practically speaking, this is an immature
industry because you have to hand do all of that.
Yeah.
And what's really interesting is some of these problems are really exciting and new.
As Aaron says, you're trying to pull out something that's very undefined in free text document.
Okay.
So that's, you need some modern stuff to do that, but some of these problems are pretty
traditional.
Classifying a document and you've got plenty of examples.
You don't need to go and use your OpenAI LLM to do this classification problem.
We've been doing this for a long time and you can do them a lot cheaper.
Yeah.
It's slow and expensive to use OpenAI or Google or Meta for basic classification,
but it's great for prototyping.
So the key insight is work out the piece that you really need the expensive tech for and
ensure that you boil down the problem only to that using other pieces of technology upstream.
Yeah.
So how do you handle like the, if you have all these prompts essentially acting as agents
and you have to have this sequence occur in a specific order, how do you asynchronously,
is there a specific layer that's doing the handoff?
Are they doing the, are they doing a turnover at rounds between these prompts?
New text synchronization.
Yeah.
I mean, a little technical.
Yeah.
So we use an event sourced architecture.
Yeah.
So this is outside of AI, basically means that we handle streaming data quite well.
So we have data that's streaming from over a hundred hospitals now,
more or less real time.
It comes out of Cerner, Epic, whatever the electronic medical record system is.
So a doctor writes a new note, finishes it, it hits our system and goes onto the queue,
gets pre-parsed, classified, cut up into little bits and then divvied out to these,
these little agents.
As always, you can check out the show notes and links at howtotalk2.ai.
That's all for this week's episode.
Happy prompting everyone.